Patent Application
20140229472
The present invention relates generally to the association of metadata to a multimedia content and, in particular, to a method and device for enriching a multimedia content defined by a timeline and by a chronological text description with, for instance, comments posted by Web users on a social network or other user-generated content repositories such as web forums.
This section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present invention that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present invention. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
The text analysis of comments on a multimedia content (e.g. movies) wrote by non-professional authors (as a TV user) has gained a strong interest in recent years with the development of social networks and platforms, such as TWITTER and YOUTUBE, and other Web forums.
In particular, several researches focus on the synchronization of user comments with a video content and, especially, with the timeline of said video content. It is then known to enhance an audiovisual content by using text micro-posts generated through social networks (as a tweet thanks to the TWITTER platform) during live events, the synchronization being done directly by using the timestamp of micro-posts.
In addition, it is also known systems allowing users to view a video content and to simultaneously post comments which are automatically and naturally associated with a time within the video.
In both cases, the synchronization of the comments with the timeline of the audiovisual content, if achieved, is straightforward since the comments have already a timecode, thanks to timestamping.
In other words, those prior art techniques allow for synchronizing the comments with the timeline of a multimedia content, only because each comment is associated with temporal information (e.g. the time a text micro-post has been sent on its corresponding social network), said multimedia content being played simultaneously.
Nevertheless, one downside of those prior art techniques relies on the fact that they require to write the comments during the playback of the multimedia content, in order to match the time of emission of each comment with a specific time point of the multimedia content. In addition, since it takes time to write a comment, its content might be related to a previous scene of the multimedia content, and not to the scene in which the comment is finally associated with. In other words, the synchronization appears to be inaccurate.
The present invention attempts to remedy at least some of the previously mentioned downsides and to align, in particular, textual metadata to a multimedia content at a specific time point in its corresponding timeline without using any timecode.
The present invention concerns a method for enriching a multimedia content defined by a timeline and by a chronological text description.
To this end, the method comprises the steps of:
In the present specification, it should be understood that:
In addition, in the following specification, it is assumed that the multimedia content and its chronological text description have already been aligned together thanks to well-known techniques, so that the chronological text description is directly and already synchronized with the timeline of the multimedia content. In a variant, such an alignment might be performed after the implementation of the present invention.
Thus, thanks to the present invention, a text to text alignment can be performed between a text document and the chronological text description of a multimedia content without, necessarily, using timecodes or time information. Such a method might align at least a piece of text document with one or several corresponding parts of a chronological text description of a multimedia content, so as to associate one or several time points or intervals of the timeline it is referring to within the chronological text description.
The method of the invention does not intend to match directly a text document, or a part of it, with the multimedia content, but only through the corresponding chronological text description.
Moreover, it should be appreciated that a feature mentioned in a single text document may refer to several distinct time points or intervals in the chronological text description.
In addition, Natural language Processing is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of humancomputer interaction.
In an aspect of the present invention, said text document can be identified, from a set of text documents, as being related to the multimedia content.
In addition, said set of text documents might be retrieved from the Internet network.
In another aspect of the present invention, during the step of aligning, an anaphora resolution technique might advantageously be implemented to perform the semantic correspondence between the said determined part of the text document and the chronological text description.
Besides, the step of identifying and the step of aligning can be applied on a plurality of text documents to automatically synchronize said text documents with the timeline of the multimedia content.
According to a preferred embodiment of the invention, the feature belongs to the following group of features comprising at least:
Preferably, the natural language processing corresponds to an entity recognition treatment or to a feature based sentiment analysis.
In an example of realization of the present invention, the multimedia content is an audiovisual content and the text document is a comment (so called post) wrote by a Web user.
In another aspect of the present invention, the multimedia content being segmented into scenes in which a corresponding time interval of the timeline is associated with, the text document can be synchronized to the time interval of the scene it is related to, as a result of its synchronization to the associated chronological text description.
Moreover, the present invention also concerns a system for enriching a multimedia content defined by a timeline and a chronological text description. According to the invention, said system comprised:
Certain aspects commensurate in scope with the disclosed embodiments are set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of certain forms the invention might take and that these aspects are not intended to limit the scope of the invention. Indeed, the invention may encompass a variety of aspects that may not be set forth below.
The invention will be better understood and illustrated by means of the following embodiment and execution examples, in no way limitative, with reference to the appended figures on which:
Wherever possible, the same reference numerals will be used throughout the figures to refer to the same or like parts.
According to an example of a preferred embodiment, the present invention is depicted with regard to a movie in which a timeline and a script are associated with. It has to be noted that a script is a particular example of a chronological text description of a movie.
Obviously, the present invention is not restricted to this example and can be applied to any multimedia content defined by a timeline and by at least one chronological text description (such as a script).
In particular, the system S is connected—directly or through a gateway (not represented on FIG. 1)—to a network N (e.g. the Internet Network). Obviously, in a variant, said system S might not be connected to any network.
A set of posts Pi (i∈[1; N], N integer) related to the movie is stored on a remote server RS (as for example a movie dedicated Web site like IMDb), connected to the network N. Said set of posts Pi related to the movie might be retrieved from the server RS by the system S, using the Internet network N, and might be stored in an adapted memory M of the system S.
Moreover, in the example, the movie and the corresponding script might be downloaded from a video server VS, through the Internet network N, and might be stored in said memory M. Naturally, in a variant, the movie may be retrieved from any other adapted ways (e.g. from a USB key or a DVD). Obviously, in a variant, the remote server RS and the video server VS might be the same server.
The system S also comprises a man-machine interface MMI (as for example a touch screen), intended to be used by an operator to enter one or several distinct features to be extracted from the set of posts Pi. Naturally, the features might be defined and/or selected automatically. Once entered in the system S, the features might be stored in the memory M.
Each feature is a particular element of the movie which might be mentioned, described or evaluated in a post Pi, such as a specific scene, event or action, a particular aspect of the movie (named entity, actor, director, light, etc.) or a particular aspect in a specific scene.
In particular, a feature might be:
As shown on
In particular, the natural language processing can correspond to an entity recognition treatment and/or to a feature based sentiment analysis, as defined, for instance, in the document “Aspect-based sentiment analysis of movie reviews on discussion boards” (Thet & al., 2010) published by the Journal of Information Science, 36(6), 823-848.
Once at least some defined features have been identified, a correspondence table might be established to associate with each predefined features the corresponding part or parts of posts Pi which have been identified by the module NLP. This correspondence table might be stored in the memory M. Moreover, as shown on the
In particular, the semantic correspondence may be obtained by computing text similarity measures between the extracted feature(s) and parts of the chronological text description: for example, using the Jaccard coefficient (as defined in “Étude comparative de la distribution florale dans une portion des Alpes et des Jura” (Jaccard, Paul (1901) published in the Bulletin de la Société Vaudoise des Sciences Naturelles 37: 547-579) or cosine measures (as in defined at the following Web address http://en.wikipedia.org/wiki/Cosine_similarity), or simply by computing the number of words in common. Then the part of the post containing the feature is aligned to the closest parts of the chronological text according to this textual similarity score.
In an alternative, the semantic correspondence may itself be conducted using Natural Language Processing.
In addition, the alignment module A can implement an anaphora resolution technique to perform the semantic correspondence between the determined part(s) of posts Pi and the script of said movie.
In this way, each identified feature of a post Pi is aligned to a specific point in the script, which might be a scene (each scene being identified by a predetermined time interval) or a more precise time information of the timeline (e.g. a minute). As a consequence, posts Pi get aligned to the script, possibly at multiple time points.
Thanks to the present invention, posts Pi or part(s) of them are directly and automatically synchronized with the script of the movie. A text-to-text synchronization of posts with the script is thus implemented. A time information—defined by reference with the timeline of the movie—is then implicitly attached with each post Pi or part(s) of it.
In the case where the script is already temporally aligned with the movie, posts Pi or part(s) of them become implicitly temporally aligned to the movie as well, through the script.
The present invention can then perform the alignment of text documents to a chronological description of a multimedia content without using the chronological information nor time-stamps.
The flow chart of
In a first preliminary step E0, posts Pi—stored on the remote server RS—are preliminary identified as related to the considered movie.
In a further step E1, identified posts Pi are retrieved from the remote server RS, so as to be stored in the memory M of the system S.
In a further step E2, the features intended to be identified and extracted from the posts Pi are defined and entered in the system S, via the man-machine interface MMI (or, in a variant, through software programming).
In a further step E3, the module NLP automatically identifies, in each post Pi, one or several defined features and establishes a correspondence table, wherein, to each predefined feature, corresponding posts Pi or part(s) of them are associated with.
In a further step E4, the alignment module A automatically aligns the posts Pi or part(s) of them with the semantically corresponding part(s) of the script. Such an alignment provides to the posts Pi (or part(s) of them) a time information with respect to the time of the movie.
Naturally, the previous steps might be implemented in a different order.
Then, once the alignment has been performed for a predetermined movie, the aligned posts Pi might be stored with the script, so that, during the playback of the movie, aligned posts Pi or part(s) of them can pop up, at a corresponding time point, on the main display device (e.g. a TV) and/or on a second screen (e.g. a tablet).
Besides, as a first illustrative but non-limiting example, the
This post has been retrieved from the movie dedicated Web site IMDb and has the following content:
The text of the post is very rich and refers to many aspects of the movie entitled “Boyfriends and Girlfriends”, as well as specific locations and/or scenes, as for instance:
After the step E3 of identification performed by the system S, the following table of correspondence might be established:
It is assumed that the movie script provides the following information of the scenes:
The alignment module A of the system S realizes, in step E4, the mapping of the script timeline with extracted posts or part(s) of them.
In particular, the first feature indicates a location. According to the script, scenes 3 and 13 both take place in an apartment. This part of the post may refer to these scenes. This could be checked or improved using, as previously mentioned, anaphora resolution techniques, which would link the word “her” (in “her empty white apartment”) to Blanche.
The second feature describes a scene with Blanche and Fabien which takes place “in the wilderness”. Characters match in both scenes 7 and 13. However, “wilderness” is much closer semantically to “forest” than to “apartment”. This could be found using external word ontologies such as WordNet. Scene 13 is thus discarded.
As a result, one might know that this post refers to scenes 3, 7 and 13, and it is able to point out in the text which portions correspond to which scene.
In a second illustrative but non-limiting example, the multimedia content is soccer game video (Chelsea versus Barcelona). Posts are forum comments referring to this soccer game, which can be crawled on sport dedicated websites. The script is a textual summary of the soccer game. It may be, for instance:
Hereinafter is represented a ticker showing the main moments during a section of the soccer game:
Since each sport has its specific glossary (e.g. goal, basket, foul, line-out, etc.), some terms of this glossary can be used to perform an alignment between the script and the retrieved posts.
It is then possible to pull out the names of players involved in the game, some specific terms such as goal, chronological information, etc.
In
Namely, they could be developed in the form of software, hardware, or be implemented in one or several integrated circuits.
References disclosed in the description, the claims and the drawings may be provided independently or in any appropriate combination. Features may, where appropriate, be implemented in hardware, software, or a combination of the two.
This invention having been described in its preferred embodiment, it is clear that it is susceptible to numerous modifications and embodiments within the ability of those skilled in the art and without the exercise of the inventive faculty. Accordingly, the scope of the invention is defined by the scope of the following claims.
All examples and conditional language recited herein are intended for pedagogical purposes to aid the read in understanding the principles of the invention and the concepts contributed by the inventor to furthering the art and are to be construed as being without limitation to such specifically recited examples and conditions.
Moreover, all statements herein reciting principles, aspects, and embodiments of the present principles, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 13305163.1 | Feb 2013 | EP | regional |
