Patent Application
20180005157
The extraction of descriptive metadata sufficient to characterize a media asset, such as a feature film or animation, for example, often requires the participation of human contributors having specialized knowledge. In addition, some of the metadata relied on to characterize a media asset may be extracted by automated processes, such as those using facial or object recognition software. Although tools for enabling collaboration among human contributors exist, those conventional tools are typically designed to passively process the inputs provided by each individual contributor. There remains a need for a solution enabling workflow management for the efficient extraction and synthesis of metadata for characterizing a media asset from a combination of automated and human sources.
There are provided systems and methods for media asset tagging, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims.
The following description contains specific information pertaining to implementations in the present disclosure. One skilled in the art will recognize that the present disclosure may be implemented in a manner different from that specifically discussed herein. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elements among the figures may be indicated by like or corresponding reference numerals. Moreover, the drawings and illustrations in the present application are generally not to scale, and are not intended to correspond to actual relative dimensions.
The present application addresses the challenges to collaboration described above, as well as analogous obstacles to successful workflow management. According to one implementation, a system and method according to the present inventive principles may be used to characterize a media asset utilizing tags based on metadata extracted from the media content by multiple human contributors and/or automated processes.
As disclosed in the present application, a media asset tagging system includes a workflow management software code including a tagging application template and a multi-contributor synthesis module. The workflow management software code, when executed by a hardware processor of the media asset tagging system, provides a workflow management interface enabling the workflow management software code to receive data identifying a media asset selected for tagging, as well as data for determining workflow rules. In addition, the workflow management software code utilizes the tagging application template to generate, based on the determined workflow rules, custom tagging applications for use by human contributors to extract metadata from the media asset.
The workflow management software code receives tagging data via one or more of the custom tagging applications, or in some instances from an automated media asset tagger or taggers as well. Based on the tagging data received, the workflow management software code can determine constraints for subsequent tagging data. In some implementations, the workflow rules may specify one or more quality assurance analyses of any of the received tagging data or the determined constraints. The workflow management software code can then utilize the multi-contributor synthesis module to tag the media asset based on the tagging data, subject to the determined constraints.
The collaboration and workflow management enabled by the systems and according to the methods disclosed in the present application can be applied across a wide variety of project types, including highly complex multidisciplinary projects. For example, as discussed in greater detail below, the present solution may be specifically applied to characterization of a media asset such as a video, feature film, or animation, using metadata based tags.
Alternatively, the present workflow management solution may be suitably adapted for application to the maintenance or upgrading of theme park assets, such as hotel accommodations, dining venue, rides, or shows, for example. Moreover, in some implementations, the present solution may be suitably adapted to provide workflow management for scheduling seasonal routing and/or relocation of cruise ships so as to substantially optimize passenger safety, comfort, and enjoyment. Coordination and management of the exemplary collaborative projects described above, as well as collaborative projects of many other types, can be enabled and enhanced through implementation of the systems and methods disclosed in the present application.
Media asset tagging system 102 includes hardware processor 104, and system memory 106 storing workflow management software code 110 including tagging application template 114 and multi-contributor synthesis module 116. In addition, system memory 106 is shown to include media asset 108 and workflow management interface 112 provided by workflow management software code 110. Also shown in
According to the implementation shown in
It is noted that although
It is further noted that although management system 122 is shown as a personal computer (PC), and client systems 140a and 140b are shown as mobile communication devices in
Media asset 108 is a media asset undergoing metadata extraction and tagging in a process guided and controlled by workflow management software code 110, executed by hardware processor 104. Media asset 108 may correspond to a variety of different types of media content. For example, media asset 108 may include media content in the form of video and/or audio content. Specific examples of media content that may be included in media asset 108 include feature films, animation, television programming, games, music, and educational content.
Referring to
As shown in
Network communication links 234 and 238, and media asset tagging system 202 including hardware processor 204 and system memory 206 correspond in general to network communication links 134 and 138, and media asset tagging system 102 including hardware processor 104 and system memory 106, in
Client systems 240a and 240b correspond in general to client systems 140a and 140b, respectively, in
Hardware processors 244a and 244b may be the central processing units (CPUs) for respective client systems 240a and 240b, for example, in which role hardware processors 244a and 244b run the respective operating systems for client systems 240a and 240b, and execute respective custom tagging applications 218a and 218b. Displays 242a and 242b may take the form of liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, or any suitable display screens that perform a physical transformation of signals to light.
In the exemplary implementation represented in
Media asset tagging system 102/202 in
Referring to flowchart 350, with further reference to
Referring to
Workflow rules 460 may be selected or modified by workflow manager 120, via workflow management interface 412, to produce workflow 470 specifying the processing events used to extract metadata from and tag media asset 108/208, as well as the sequencing in which those processing events occur. Workflow 470 will be described more completely below.
As shown in
Flowchart 350 continues with receiving media asset identification data and workflow rules data via workflow management interface 112/212/412 (action 352). Media asset identification data and workflow rules data may be received by workflow management software code 110/210, executed by hardware processor 104/204. Referring to
The media asset identification data received by workflow management software code 110/210 may populate media asset field 448 of workflow management interface 112/212/412, and may be used to identify media asset 108/208. The media asset rules data received by workflow management software code 110/210 may be used to select among or modify rules 460 for producing workflow 470.
Flowchart 350 continues with generating custom tagging applications 218a and 218b based on the workflow rules data (action 353). Generation of custom tagging applications 218a and 218b can be performed by workflow management software code 110/210, executed by hardware processor 104/204, and using tagging application template 114/214.
By way of example, human contributors 130a and 130b may each have specialized knowledge regarding different features of media asset 108/208. Consequently, custom tagging application 218a generated for use by human contributor 130a may be different from custom tagging application 218b generated for use by human contributor 130b. That is to say, for example, workflow manager 120 may utilize workflow management interface 112/212/412 to identify different questions 464 and to make available different metadata tags 466 for inclusion in respective custom tagging applications 218a and 218b.
As a specific example, where media asset 108/208 is a feature film, human contributor 130a may have specialized knowledge of locations appearing in the film, while human contributor 130b may have specialized knowledge about special objects, such as weapons or vehicles, used in the film. Under those circumstances, the questions and metadata tags included in custom tagging application 218a may be selected or composed by workflow manager 120 to elicit location information from human contributor 130a. Analogously, the questions and metadata tags included in custom tagging application 218b may be selected or composed by workflow manager 120 to elicit special object information from human contributor 130b.
Flowchart 350 continues with receiving a first tagging data for media asset 108/208 (action 354). The first tagging data may be received by workflow management software code 110/210, executed by hardware processor 104/204, via automated media asset tagger 136, or from human contributors 130a or 130b via respective custom tagging applications 218a and 218b.
Referring to
However, the particular metadata extraction and tagging process governed by workflow 470 relies on tagging data inputs from a combination of automated and human contributors, and is specific about the sequence in which those contributors participate. According to workflow 470, for example, contributors 462 include automated media tagger 136 and human tagging contributors corresponding in general to human contributors 130a and 130b. In addition, workflow 470 specifies that the first tagging data is to be tagging metadata identifying characters in media asset 108/208, and that the first tagging data be received from automated media tagger 136.
It is noted that according to the exemplary media asset tagging process described by workflow 470, automated media tagger 136 is tasked with identifying characters appearing in media asset 108/208. In that instance, automated media tagger 136 may utilize facial detection or recognition software to automatically identify characters in media asset 108/208. However, in other implementations, media asset tagging system 102/202 may utilize other types of automated media taggers to identify other attributes or characteristics of media asset 108/208. Thus, in other implementations, automated media asset tagger 136 may utilize object recognition software, computer vision, or natural language processing, for example.
Flowchart 350 continues with determining one or more constraints for tagging media asset 108/208 based on the first tagging data (action 355). Determination of the one or more constraints based on the first tagging data may be performed by workflow management software code 110/210, executed by hardware processor 104/204.
For example, and returning to the case in which media asset 108/208 includes video, and the first data provided by automated asset tagger 136 identifies characters appearing in the video, one or more constraints may be determined based on those characters. For instance, the cast of characters identified by automated media tagger 136 may be known to have appeared in video content including some locations but not others. That information may be available to workflow management software code 110/210 from a media asset knowledge base accessible over communication network 130 (knowledge base not shown in the present figures). Workflow management software code 110/210 may use such information to constrain subsequent identification of locations within media asset 108/208 by preventing a subsequent automated or human contributor from selecting a location tag that does not correspond to one of the subset of locations corresponding to the cast of characters identified by the first data.
Similarly, special object tags and/or action tags utilized by subsequent automated or human contributors may be constrained based on special objects and or actions known to correspond to the cast of characters identified by the first tagging data. Where the constraint or constraints are imposed upon human contributors, for example, custom tagging application 218a and/or 218b generated by action 353 may be updated based on the one or more constraints determined by action 355. Such updating of custom tagging application 218a and/or 218b may be performed by workflow management software code 110/210, executed by hardware processor 104/204.
Flowchart 350 continues with receiving additional tagging data for media asset 108/208 (action 356). The additional tagging data may be received by workflow management software code 110/210, executed by hardware processor 104/204. Like the first tagging data received in action 354, the additional tagging data may be received via automated media asset tagger 136, or from human contributors 130a or 130b via respective custom tagging applications 218a and 218b, and may be communicated to workflow management software code 110/210 over one of network communication links 138 and 134.
Referring to
According to workflow 470, in addition to an automated media tagger providing the first character tagging data, contributors 462 include first, second, and third human contributors providing additional location tagging data, special object tagging data, and action tagging data, respectively. In addition, workflow 470 specifies that the first tagging data received from automated media asset tagger 136 identifying characters in media asset 108/208 be used as an input to locations tagging performed by the first human contributor. That locations tagging performed by the first human contributor is used, in turn, as an input to the special objects and actions tagging performed in parallel by the second and third human contributors. In other words, the additional tagging data may be generated based on the first tagging data.
It is noted that although according to workflow 470, the first tagging data is received from an automated media asset tagger, while the additional tagging data is received from human contributors via custom tagging applications corresponding to custom tagging applications 218a and 218b, that specific workflow organization is merely exemplary. Other workflows implemented using media asset tagging system 102/202 may specify receipt of a first tagging data via a custom tagging application from a human contributor, followed by receipt of additional tagging data from a combination of one or more additional human contributors and/or one or more automated media asset taggers. Thus, the additional tagging data for media asset 108/208 may include a second tagging data received via an automated media asset tagger or a custom tagging application, as well as a third, fourth, or more tagging data each received via an automated media asset tagger or a respective custom tagging application.
Flowchart 350 continues with determining one or more additional constraints for tagging media asset 108/208 based on the additional tagging data (action 357). Determination of the one or more additional constraints based on the additional tagging data may be performed by workflow management software code 110/210, executed by hardware processor 104/204.
Returning yet again to the case in which media asset 108/208 includes video, the first data provided by automated asset tagger 136 identifies a cast of characters appearing in the video, and additional tagging data provided by a first human contributor identifies one or more locations corresponding to that cast of characters, one or more additional constraints may be determined based on those locations. For instance, some special objects may be known to appear, and/or some actions may be known to occur, in some locations but not in others. As noted above, such information may be available to workflow management software code 110/210 from a media asset knowledge base accessible over communication network 130. Workflow management software code 110/210 may use that information to constrain subsequent identification of special objects and/or actions within media asset 108/208 by preventing subsequent automated or human contributors from selecting a special object or action tag that does not correspond to one of the subset of special objects or actions corresponding to the identified locations, or to the identified cast of characters.
In workflow implementations in which the additional constraint or constraints are imposed upon human contributors, custom tagging application 218a and/or 218b generated by action 353 may be updated based on the one or more additional constraints determined by action 357. Such updating of custom tagging application 218a and/or 218b may be performed by workflow management software code 110/210, executed by hardware processor 104/204.
Moreover, and as shown by workflow 470, in some implementations, the constraints determined by action 357 may be imposed on more than one tagging contributor working in parallel. For example, in exemplary workflow 470, the second human contributor generates the additional special objects tagging data substantially concurrently with generation of the additional actions tagging data by the third human contributor.
Flowchart 350 may conclude with tagging media asset 108/208 based on the first tagging data and the additional tagging data, subject to the one or more constraints determined based on the first tagging data, and the one or more additional constraints (action 358). Tagging of media asset 108/208 may be performed by workflow management software code 110/220, executed by hardware processor 104/204, and using multi-contributor synthesis module 116/216.
In one implementation, for example, multi-contributor synthesis module 116/216 may be utilized by workflow management software code 110/210 to filter the tagging data received from all contributors, i.e., automated media asset tagger or taggers 136 and all human contributors including human contributors 130a and 130b, using the constraints determined based on that tagging data. As a result, a comprehensive and consistent set of metadata tags may be applied to media asset 108/208 that characterizes many or substantially all of its attributes.
Although not included in the outline provided by exemplary flowchart 350, as shown by workflow management interface 412, in
From the above description it is manifest that various techniques can be used for implementing the concepts described in the present application without departing from the scope of those concepts. Moreover, while the concepts have been described with specific reference to certain implementations, a person of ordinary skill in the art would recognize that changes can be made in form and detail without departing from the scope of those concepts. As such, the described implementations are to be considered in all respects as illustrative and not restrictive. It should also be understood that the present application is not limited to the particular implementations described herein, but many rearrangements, modifications, and substitutions are possible without departing from the scope of the present disclosure.
