Patent Application
20160098602
The present invention relates to video processing, and in particular, the invention is related to a method for computing a similarity measure between segments of videos based on social network analysis.
Sharing video clips or video segments has become an entertaining aspect of digital exchanges. Various websites on the Internet sometimes feature entire videos, video clips, or video segments that are newsworthy or are entertaining. Social media has also engaged in video sharing where one social media user shares a video segment with another social media user. In one sharing scenario, a video segment is first determined by one user to be worthy enough for sharing with a friend user by making a personal determination of how a new video segment compares to an older video segment that the friend user likes before sharing the new video segment with the friend user. Thus, a method of performing a comparison of one video segment to another video segment to determine a similarity is of interest.
In an article by Liang Gou, Hung-Hsuan Chen, Jung-Hyun Kim, Xiaolong (Luke) Zhang, and C. Lee Giles, entitled “SNDocRank: a social network-based video search ranking framework”, published in 2010 in the Proceedings of the International Conference on Multimedia Information Retrieval, ACM, New York, N.Y., USA, pages 367-376, one technique of similarity comparison is to first compute the similarity of a video document with a keyword search query and by using the social network of the user who entered the query. This technique uses social scores, relations between social network members, text descriptions, audio features, or image features that characterize the videos, such as comments about a sports game, or other aspect of a video segment. In addition, this technique only allows for the comparison of a video segment with a keyword search query, and not with another video segment directly.
Thus, a continuing need exists for other, simpler, and more accurate techniques for determining a similarity measure of a video segment. For example, there currently does not exist a measure of video segment similarity that takes into account a social network of viewers that digitally commented on video segments.
This summary is provided to introduce a selection of concepts in a simplified form as a prelude to the more detailed description that is presented later. The summary is not intended to identify key or essential features of the invention, nor is it intended to delineate the scope of the claimed subject matter.
In one aspect a method for generating a similarity measure of video segments includes selecting a video segment, by a user who viewed the video segment. A social network of other viewers who commented on the video segment is accessed to identify other viewers who viewed and commented on the selected video segment. A similarity score is assigned to each pair of the other viewers and a grouping of the other viewers into clusters is performed. A similarity is determined and links are established between video segments identified by the comments from the other viewers. Pairs of similar video segments and their associated comments are assembled and a similarity score to each pair of video segments is assigned. The results are a triplet of similar video segments and a similarity score which can be provided to the user.
An apparatus for generating a similarity measure of video segments includes a network interface to communicate with a social network of viewers of a video segment who commented on the video segment. A social network module assigns a similarity score to each pair of the viewers and groups the viewers into clusters. A linking module establishes links between video segments identified by the comments from the viewers. The linking module arranges pairs of similar video segments and associated comments together. A similarity module assigns a similarity score to each pair of video segments. A display device displays similar videos segments to a user of the apparatus.
Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments which proceeds with reference to the accompanying figures. It should be understood that the drawings are for purposes of illustrating the concepts of the disclosure and is not necessarily the only possible configuration for illustrating the disclosure.
The foregoing summary of the invention, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the accompanying drawings, which are included by way of example, and not by way of limitation with regard to the claimed invention. In the drawings, like numbers represent similar elements.
In the following description of various illustrative embodiments, reference is made to the accompanying drawings, which form a part thereof, and in which is shown, by way of illustration, how various embodiments in the invention may be practiced. It is to be understood that other embodiments may be utilized and structural and functional modification may be made without departing from the scope of the present invention.
The problem of computing a similarity measure between video segments is addressed with a novel technique as described herein. The video segments may belong to the same or different videos. Use is made of digital comments, being published on dedicated Web forums or on blogs, that are attached to these video segments, in order to link together some of those video segments. Also used is the social network of viewers who posted these digital comments in order to link the video segments of related viewers. The current invention takes into account a social network of viewers that digitally commented on these video segments. One aspect of the invention is that similarities between pairs of segments of video documents are compared in addition to the reliance on the social network of the viewers who commented on the video segments.
A video segment is a continuous interval of video images in a video document. A video document is herein also called a “video”. A video segment is defined with a starting timecode and an ending time code. A viewer is a user who posted one or more digital comments on one or more videos or video segments. A comment is a digital document that is attached with a particular video and that is characterized by a set of text tags or keywords. The comment may be any type of multimedia document such as video, audio or text document. Some comments are synchronized with the videos' timeline, meaning that for each of these synchronized comments, there is a known link to a time point in the video segment.
In overview, according to aspects of the invention, assume a primary user has a selected a set of video documents or video segments (“videos”). For each video, assume a set of digitized comments from other viewers that have been synchronized to their video's timeline. According to the invention, it is desirable to compute the similarity between segment si in video vi and segment sj in video vj.
Segments si and sj are each described by a set of digital comments Ci and Cj. Each comment ck in Ci (or respectively cl in Cj) has been posted by a viewer uk (or respectively viewer ul). According to aspects of the invention, using the social network of other viewers who commented on the video segment, a social network algorithmic module infers a similarity measure between viewers uk and ul and uses it to build and evaluate clusters of viewers grouped together according to their similarity measures.
An algorithmic linking module then links together video segments: Two video segments si and sj are then considered similar if there exist at least a digital comment ck on si and a digital comment cl on sj that were posted by viewers uk and ul, where uk and ul belong to the same viewer/commenter cluster, and where ck and cl share at least one tag or keyword. Please notice that video segments may belong to the same video.
According to an aspect of the invention, a similarity module assigns a similarity score based on the similarity of the viewers and on the similarity of their digital comments for each pair of similar video segments si and sj. These similar video segments may then be presented to one or more users so that the one or more users may benefit from the additional discovered similarity measures. Similar video segments as well as commenting views may then be shared via communication such as social networks.
In either instance, a user, operating the user equipment 150 can obtain a similarity measure of two video segments where that information can be utilized to provide the user recommendation for videos or the opportunity for recommendations of user social connection. For convenience, the user of user equipment 150 is referred to herein as the primary user.
In the web-server embodiment, a web site can identify or host videos and viewer digital comments associated with these videos in a YouTube™ like fashion. Comments could also be pictures or photographs, posted on an Instagram™ like web site, and associated with tags posted by users. Registered users of the server implementation have the possibility to post digital comments in the feed of videos. They may also explicitly specify the time point in the video that is being discussed. For example, they can write “LOL 3:24” which would mean something funny is happening at time 3:24” (three minutes and 24 seconds) into a specific video.
The social network module 210 analyzes the social network of viewers who have been identified as those social network members who commented on a video segment. The social network module assigns a viewer similarity score to each pair of identified video segment viewers based on known social network analysis methods that take into account the links between viewers and also some external features optionally (e.g. areas of interest, categorization of user—leader, follower, instigator, . . . , etc.) Using this viewer similarity score, the social network module 210 automatically builds and evaluates groups of viewers who have commented on one or more video segments (i.e. viewers clusters) using traditional clustering methods (hierarchical clustering, k-means, etc.). This social network member clustering may be built considering the viewers who commented on the whole set of videos, or only viewers who commented on a subset of pre-selected video segments. As a result of this step, each pair of viewers is assigned a viewer similarity score and all viewers are grouped into a certain number of clusters, each viewer appearing in one and only one cluster. Triplets of viewer uk, viewer ul, and their viewer similarity score are output 240 from the social network module 210. These triplets may be placed into a memory for subsequent use. Also, the social network module outputs a clustering of viewers 250. These user clusters may also be placed into a memory for subsequent use.
The linking module 230 accesses all video segments and their synchronized comments 260 and establishes links between the video segments. A synchronized comment can be a time synchronized comment relating to a specific time mark in a video segment or a comment that is simply associated with (synchronized) with the video segment. The synchronized comments and the related video segments are functionally obtained by accessing the comments and related video segments of members in the social network 210. Then, the synchronized comments and their related video segments are stored in memory for subsequent use by the linking module. The linking module accesses the clusters of viewers 250 from memory as well as the synchronized (associated) comments and related videos 260. Memory can be any kind of data storage, such as static or dynamic random access memory (RAM), flash memory, magnetic or optical disk memory, remote or local memory, volatile or non-volatile memory, and the like. Memory used by any of the modules of
The task of the similarity module 220 is to assign a similarity score to each pair of video segments. The similarity module 220 combines triplet of users/viewers and viewer similarity scores 240 corresponding to the video segment pairs and synchronized comments and comment similarity scores 270 using, for example, a weighted mean. The output is a triplet 280 of similar video segments (segment si and segment sj) and a final video similarity score (sim score). This result is then useful for many purposes.
One use is to provide recommendations of video segments for the primary user. In another embodiment, the triplet 280 can also be used to recommend video segments to viewers who have commented on similar video segments. This implies that viewers are not necessarily recommended video segments commented by someone from their viewer cluster, but also from other clusters. In another embodiment, the triplet 280 can be used to recommend a video creator to another video creator if the videos they created share similar segments. In another embodiment, the triplet 280 is useful for recommending similar video segments for all segments in a given video, presented as alternatives for the user or for the creator of the video. In another use of triplet 280, results can be used to connect viewers of similar areas of interest in order to get more digital comments and better later recommendations. Other advantages and uses include enriching users' experience by providing new content discovery.
Step 310 of method 300 determines a viewer similarity score for each pair of viewers. For example, the viewer similarity score between two viewers may be the percentage of common viewers that both viewers share in their neighborhood in the social network. Once a viewer similarity score is obtained, step 310 also groups viewers into clusters of viewers. Each viewer appears on only one cluster. Step 310 produces viewer similarity triplets 240 and clusters 250 of
Step 315 of method 300 establishes links between video segments based on synchronized comments and viewer clustering. Stated another way, this step establishes links between video segments identified by the video comments from people who are in the social network of the other viewers and commenters. Video segments identified for comment similarity checking are those video segments from viewers and commenters that are clustered together and have at least one feature (such as a video tag or keyword) in common. At step 317, a comment similarity score is computed between all comments associated (synchronized) with pairs of video segments that are linked. Once links have been established between similar video segments after similarity checking, and comment similarity scores have been computed, Step 320 arranges pairs of similar video segments and associated synchronized comments together with the same comments arranged in pairs and their respective comment similarity scores. This step essentially assembles similar video segments into pairs along with their respective comments and the respective comment similarity scores. Step 320 produces a list of the pairs of similar video segments together with their synchronized comments and comment similarity scores 270 of
Step 325 of method 300 calculates and assigns a final video similarity score to each pair of video segments listed by Step 320. Step 325 combines the viewer similarity scores of the viewer similarity triplets 240 with the pairs of similar video segments and associated comment similarity scores 270 to generate the triplets of similar video segments and a final video similarity scores (280) (sim score) for each pair of video segments. Step 325 produces the triplets of pairs of similar video segments and video similarity scores as a result of the action of the similarity module 220 of
where n is the total number of comments associated with the two video segments si and sj, ck and cl are comments on si and sj, uk an ul are the viewers who produced comments ck and cl, sim(uk, cl) is the comment similarity score between comments ck and cl, and sim(uk, ul) is the viewer similarity score between viewers uk and ul. The calculated video similarity score is the measure of similarity between video segments.
Step 330 begins the utilization of the previous steps by providing a listing of similar video segments and the corresponding video similarity scores. Examples of providing a list is to access the listing for display or accessing the list for transfer to another process, computer, etc. These similar video segments are similar to the video segment that was initially selected by the primary user. In addition, other similar video segments, not selected by the primary user may be provided to be displayed. In addition, the listing may optionally include, at step 335, the respective commenters of the similar video segments. In another embodiment, not shown, the listing is available to any one or more members of a cluster group that the commenting other viewers belong.
Turning now to
In the apparatus 400 shown in
In one embodiment, one or more digital busses 424 interconnect the controller processor 408 with other functional blocks. One of skill in the art recognizes that non-bus-oriented topologies are also possible implementations of the media device 400. Blocks that have a bus 424 interface with the processor 408 include bus interface 404, storage device 406, and display interface 418. Functional modules, such as social network module 412, linking module 414, and similarity module 416 may be implemented in hardware, software/firmware, or a combination of both. Modules 412, 414, and 416 may be implemented together as one processing unit or may be separate as shown in
The decoded output signal of network receiver/transmitter 402 is provided to a bus interface 404. The bus interface 404 performs the conversion of acquired network information into bus-oriented signals and data to allow the various bus modules to perform operations on the received data. The bus 424 allows a complete communication between the occupants of the bus 424 including display interface 418 which allows a user to view at least the results of the method 300 on display device 420.
A storage device 406 stores video segment information and comments that are accessed and received at the input 402 from the social networks. The storage device 406 allows later retrieval and playback of the content under the control of a controller 408 and social network module 412, linking module 414 and similarity module 416. A user control interface 422 such as a control panel, touch screen, remote control, or other use interface allows user control of the functions of apparatus 400, such as selection of a video segment. The storage device 406 can be a hard disk drive, one or more large capacity integrated electronic memories, such as static RAM (SRAM), or dynamic RAM (DRAM), or can be an interchangeable optical disk storage system such as a compact disk (CD) drive or digital video disk (DVD) drive.
The controller/processor 408 is further coupled to control memory 410 (e.g., volatile or non-volatile memory, including RAM, SRAM, DRAM, ROM, programmable ROM (PROM), flash memory, electronically programmable ROM (EPROM), electronically erasable programmable ROM (EEPROM), etc.) for storing information and instruction code for controller/processor 408. Control memory 410 can store instructions for controller/processor 408. Further, the implementation of the control memory 410 can include several possible embodiments, such as a single memory device or, alternatively, more than one memory circuit communicatively connected or coupled together to form a shared or common memory. Still further, the memory can be included with other circuitry, such as portions of bus communications circuitry, in a larger circuit.
The implementations described herein may be implemented in, for example, a method or process, an apparatus, or a combination of hardware and software. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms. For example, implementation can be accomplished via a hardware apparatus, hardware and software apparatus. An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to any processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device.
Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions may be stored on a processor or computer-readable media such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD” or “DVD”), a random access memory (“RAM”), a read-only memory (“ROM”) or any other magnetic, optical, or solid state media. The instructions may form an application program tangibly embodied on a computer-readable medium such as any of the media listed above or known to those of skill in the art.
| Number | Date | Country | Kind |
|---|---|---|---|
| 14 306 570.4 | Oct 2014 | EP | regional |
