Embodiments of the invention relate to a method, a server, a user device and a system for video retrieval. The embodiments further relate to a respective computer program product.
The popularity of online video communities is one reason for a strong growth of video databases. Browsing such big collections is often time consuming and cumbersome for the user and, therefore, it is difficult for a user to find a preferred video he would like to watch.
Thus, finding videos of interest in large video databases has become a challenging task.
It is an object of the embodiments to provide a method and a server for video retrieval.
This object is solved by a method and a server according to claims 1 and 15.
Further details will become apparent from a consideration of the drawings and ensuing description.
The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and together with the description serve to explain principles of embodiments. Other embodiments and many of the intended advantages of embodiments will be readily appreciated as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.
In the following, embodiments are described. It is important to note, that all described embodiments in the following may be combined in any way, i.e. there is no limitation that certain described embodiments may not be combined with others. Further, it should be noted that same reference signs throughout the figures denote same or similar elements.
It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.
It is to be understood that the features of the various embodiments described herein may be combined with each other, unless specifically noted otherwise.
In
The first audio video file may include encoded music data. The source of the first audio video file may be a video database. The first audio video file may also be provided via the internet, an intranet, from a local or remotely located storage, by a portable storage medium (e.g. memory stick or the like), and/or by the user via a graphical user interface.
At S102 a first identifier of a first piece of music included in the first audio video file is determined. Determining said first identifier may comprise determining a fingerprint of an audio signal of said first piece of music. Said fingerprint may be the first identifier.
In a further embodiment, it would also be possible that a piece of music is directly provided which would then be used for finding videos including the piece of music or at least similar pieces of music.
In a further embodiment, it would also be possible that a first identifier of a piece of music is directly provided which would then be used for finding videos including the piece of music or at least similar pieces of music. In this embodiment, in
At S104 for the first identifier meta data are looked up in a music database 100, in which said first meta data are associated with said first identifier. The look-up table, in which fingerprints are associated with identifiers, may as well be integrated in the music database 100. This integration makes it possible that the fingerprints may as well be used as meta data.
At least one meta data descriptive of at least one property of the first piece of music is looked up, but it may be as well possible to look up more than one meta data, thus, more than one property of the first piece of music may be looked up.
The meta data in the music database 100 are associated with an identifier of a corresponding piece of music. The meta data may have been determined independently from the originally provided audio video file including the piece of music. The meta data may be based on a music file and/or an audio video file, which is obtained independently from said originally provided audio video file. For example, a piece of music in an audio video file may comprise only several seconds of a piece of music, e.g. a song. The several seconds of the piece of music may nevertheless allow determining an identifier of the piece of music, e.g. a fingerprint. The identifier may then be used to look-up meta data for the piece of music. Therefore, in the respective data base, the meta data may be associated with one or a plurality of identifier that have been determined based on the same algorithms used for calculating the identifier of the several seconds of the piece of music. Of course, the meta data will then be descriptive of the whole (entire length) of the piece of music. Thus, for example, the meta data “average beats per minute” will be related to the entire piece of music and not only on the several seconds of the piece of music. The meta data are therefore more reliable (“clean” meta data).
Thus, there is an advantage since only a few seconds of a piece of music can be used to determine meta data for the piece of music. These meta data will then be descriptive of the entire piece of music. Moreover, since these “clean” meta data will be used for the search of videos (see below, and reference sign S112 in
The above shall be further illustrated at hand of an example: The music data base 100 in
Regarding the term “fingerprint”, this may refer to an acoustic fingerprint that may be a condensed digital summary, deterministically generated from an audio signal that can be used to identify an audio sample or quickly locate similar items in an audio database.
Properties have here the meaning of a generic term of meta data which are descriptive for the same property. For example artist is the property while “U2” may be the meta data of the first piece of music and “Prince” may be the meta data of a second piece of music. As well other descriptive information of the pieces of music may be used to retrieve similar videos. Thus, properties relate to the meta data as they are discussed in more detail in conjunction with
At S106 second meta data of a second piece of music, which is included in a second audio video file are provided. The source for the second meta data may be the same music database 100, where the first meta data have been looked up, or alternatively another music database. The second meta data may as well be provided via the internet, an intranet and/or from a local or remotely located storage. The meta data provided for the second piece of music may be descriptive for at least one property that has been looked up for the first piece of music. For example when the artist for the first identifier of the first piece of music is looked up as well the artist of the second piece of music is provided. The meta data for the second piece of music may as well be descriptive for more than one property of the second piece of music and in case that the meta data descriptive for the same properties have been looked up at S104, a refinement of the video retrieval functionality may be achieved.
At S108, a similarity measure is determined. The similarity measure is descriptive of the similarity of the first piece of music and the second piece of music. This similarity measure is generated by comparing the first meta data associated with the first identifier of the first piece of music and the second meta data of the second piece of music.
Depending on the type of meta data, the comparison may be conducted in different ways. If the meta data comprises numbers, e.g. in case of beats per minute, these numbers may directly be compared by e.g. determining the ratio between the two numbers and thus the similarity measure may be a number. In other cases where the meta data is based on categories, e.g. for meta data being descriptive for the artist name, the similarity measure may be based on a more complex comparison algorithms. For example the full string of the artist name may be compared, or parts of the name may be compared, or the number of identical letters may be compared. The similarity measure may be binary information with the meaning of “yes” or “no”.
As will be discussed below, at S112, the second audio video file or an identifier thereof is provided, for example to a user and/or an application, depending on the similarity measure. The dependence may result from a comparison of the similarity measure with a threshold.
Thus, at S110, the similarity measure may be compared with a threshold. The type of threshold depends on the similarity measure. Therefore, the threshold may be set by setting a number or a content such as “yes” or “no”, “1” or “0”. It is as well obvious that in dependence of the determination of the similarity value, the threshold needs to be defined. For example if the ratio for the similarity measure is beats per minute for the first identifier of the first piece of music divided by beats per minute of the second piece of music, the threshold would be set as a value deviating from the value one. In this case the similarity measure may have values larger or smaller than the value one. If however the similarity measure is defined to be always the ratio of the smaller number to the larger number, the threshold may be set as numbers being between larger than a number smaller than 1. Further, it is correspondingly possible to set a threshold value to be an upper boundary condition, so that the similarity value has to be below the threshold. Correspondingly the threshold may as well be a conditional threshold with the meaning of “being equal to”. This may e.g. applicable if the similarity measure is the comparison of strings.
It may as well be possible to set the threshold for the retrieval of a video that is as different as possible from the first audio video file. For example if a user has watched a movie he did not like at all, he may be interested to retrieve videos which are different from the video he did not like, i.e. the first and second meta data have a predefined distance from each other. Thus, the definition of the similarity measure may be set, so that the condition of the similarity measure to be above the threshold is only fulfilled if the first meta data and the second meta data differ.
Such a broad flexibility in setting and selecting the threshold value allows defining a search adapted to the need or interest of the user and avoids misleading video retrieval. Thus, the property described by the meta data to determine the similarity measure and a user may select the definition of the similarity threshold. However, the property described by the meta data to determine the similarity measure and the definition of the similarity threshold may as well be pre-set resulting in a lower effort for the user and the administrator, an easier usage of the method and a higher automation level of the video retrieval functionality.
At S112 the second audio video file or an identifier thereof (second identifier) is provided for example to the user and/or a further application or any other requestor for the functionality of video retrieval depending on the similarity measure. For example, if the similarity measure fulfills a threshold condition, for example being larger than a threshold, at S112 the second audio video file or an identifier thereof (second identifier) is provided.
As described above, as well more than one meta data descriptive of the first piece of music may be looked up. Correspondingly as well more than one meta data may be used for the similarity measure determination at S108. The similarity measure may be condensed to one similarity measure at S108. However, as well a plurality of similarity measures may be determined and compared to a plurality of thresholds at S110. The result of this comparison of a plurality of similarity measures with a plurality of thresholds may then be condensed into one similarity measure that is compared to one similarity threshold and may be described by a similarity distance measure.
If the similarity measure does not fulfill the threshold condition, the process may be terminated at S114. However, the method may be applied repetitively in that a music database of meta data of various pieces of music included in audio video files are scanned for similarities to a first piece of music. This may be achieved by repetitively conducting S106 to S110. The option to repeat the video retrieval is as well applicable in combination with any subsequent embodiment. The benefit of such a scanning function is the automation of the video retrieval functionality for video databases and allows to obtain a list of similar audio video files. The repetitive video retrieval may as well be used for an efficient management of a video database in that the videos may be grouped and non-relevant videos may be erased to save storage capacity or the video database may be rearranged to allow for a less time consuming and/or less computational power requiring usage of the video database.
As described above a music file and/or an audio video file used to determine the meta data may be available independently from the originally provided first audio video file. Therefore the steps of looking up the first and second meta data and the step of determining the similarity measures may be conducted prior to and/or independently from the provision of the first audio video file. The result of this determination of the similarity measure may be stored in a similarity database comprising a similarity matrix of pieces of music and therefore the similarity measures associated with corresponding identifiers of the first and second piece of music may be directly looked up and used for the provision step of the second audio video file. This (independent) pre-calculation of the similarity can safe computing time when searching the second audio video file based on the first audio video file.
Beyond the provision of the second audio video file or the second identifier the similarity measure itself may be provided in combination with the second audio video file or the identifier thereof at S112. This would have the beneficial effect that a receiver of the information may have a further decision criterion how to proceed with the corresponding information. For example, in case that he has experience with the method, he may know that even though there is a similarity threshold, he only wants to continue with a video which fulfills his own similarity measure condition.
Applying a method according to the embodiment show in
Another benefit of using a similarity of pieces of music for video retrieval may be that the descriptive information may be obtained with less effort in comparison to descriptive information of the video data. This may e.g. be the case for feature extraction methods for music in that they require lower computational power as they are computationally efficient and the algorithms are of limited complexity. Further, it may be beneficial using a similarity of pieces of music for video retrieval, since the number of parameters to describe a piece of music may be kept small without limiting the capability to distinguish between videos including pieces of music to be similar or not. Therefore, a music database comprising the meta data of the descriptive information may be limited in the storage demand and therefore databases with a very large number of pieces of music may be realized. It is understood that adding further descriptive elements to the music database allows for a refinement of the retrieval of similar videos and the music database owner may design the music database according to his needs.
The confidence measure is described in more detail below.
The similarity measure at S108 in
Considering the confidence measure when determining the similarity measure may have the advantage that similar audio video files are retrieved with a higher accuracy. This may improve the video retrieval method selectivity, if the differences in the meta data are generally not very large. It may be as well of a major advantage if the second piece of music is included in many audio video files, as the additional criterion of the confidence measure may lead to a gradation of the similarity measure for the individual audio video files. Thus, not all the respective audio video files identified in the meta data to comprise the second piece of music may be provided to a user. Thus, an overwhelming of the user with a large number of audio video files may be avoided and the data transfer volume may be reduced. This confidence measure consideration further has a time and resources saving effect, if a large music database is compared to the meta data of first piece of music, since again a refinement of the comparison may be achieved.
The confidence measure may be defined as an exclusive criterion in that prior to comparing the other provided meta data at S106 the confidence measure must be above a threshold value. Only if this criterion is fulfilled the comparison in S108 may be continued. This may have the effect of an acceleration of the video retrieval process and may reduce the computational power requirements. As well other meta data may be identified to be such an exclusive criterion. For example if the first piece of music is identified to be performed by the artist “Prince” only pieces of music as well performed by the artist “Prince” may be selected and another meta data such as mood is compared at S108.
Another refinement option is shown in
At S118 the user may provide his user profile. Such a profile may include age, watching preferences, interests, music interests, home address, gender etc. The profile may be provided a priori before carrying out the method or during the carrying out the method. It may as well be provided upon request. The latter would have the advantage that it would not be necessary to have the profile data stored without carrying out the method, thus, it would allow limiting the storage space and the data management effort. Saving the profile once it has been entered would cause a lower effort for the user, using the method periodically.
At S120 collaborative filtering information may be provided. The collaborative filtering information may be descriptive of the listening behavior regarding the second piece of music or watching behavior regarding an audio video file comprising the second piece of music of different user groups and/or different users.
The similarity measure at S108 may then be determined in dependence on the user profile and the collaborative filtering information. For example, if the user is of an age below 35, he may want to have the second audio video file or an identifier thereof only provided if the respective second audio video file comprising the second piece of music has been watched very often by a user group of persons having as well an age below 35. By taking the user profile and the collaborative filtering information into account, when determining at S108 the similarity measure, only pieces of music or audio video files that are appreciated by a specific user group may result in high similarity measures.
Considering this collaborative filtering information in the similarity measure determination has the advantage that it may improve the video retrieval method selectivity. It may as well be of an advantage if the second piece of music was included in many audio video files, as the additional criterion collaborative filtering information leads to a gradation of the similarity measure for the individual audio video files. Thus, not all the respective audio video files may be provided to a user. Thus, an overwhelming of the user with a large number of audio video files may be avoided and the data transfer volume may be further reduced. This collaborative filtering information refinement further has a time and resources saving effect, if a large music database is compared to the meta data of first piece of music, since again a refinement of the comparison is achieved.
The collaborative filtering information refinement may be defined as an exclusive criterion in that prior to comparing a meta data provided at S106 the comparison of the user profile to the collaborative filtering information must yield a positive result. Only if this criterion is fulfilled the comparison in S108 is continued. This has the effect of an acceleration of the video retrieval process and reduces the computational power requirements.
It is noted that the collaborative filtering information meta data may be used in two different ways. At S120 the collaborative filtering information meta data of the second piece of music is compared to the individual user profile. However, the data may as well be used independent from a specific user. Therefore, the collaborative filtering information meta data may be used in the same way as any other meta data at S104 to S114 in
The arrangement of the meta data in the music database may as well be different from the example in
For example the collaborative filtering data may be clustered in the music database or the similarity database as a user group/music matrix wherein the clusters contain the music a user group or listener group listened or likes. Therefore by finding the clusters similar music may be identified.
The meta data may be restricted to one dimension or may, alternatively, comprise a multidimensional representation. This flexibility allows for an optimized design of the music database according to the requirements and available hardware.
The different meta data may be grouped by different characteristics. A first group may be described as high level descriptors. Examples would be the title, the artist 408, the genre 404 or the mood 406. The meta data of this first group are normally not directly derivable from an audio signal of music, but are properties, which need to be identified separately from the audio signal of the piece of music. These meta data may allow a comparison of a first and second meta data descriptive of a title and an artist and/or a mood and/or a genre.
A second group of meta data may be described as low level descriptors. These meta data may be deduced from the audio signal music. Examples for these meta data are beats per minute, zero crossing rate, intensity fluctuation or relative position of intensity peaks or any other non-frequency spectrum based meta data for example time domain based descriptors. These non-frequency spectrum dependent meta data have the beneficial effect that they require low computational power. Frequency spectrum based descriptor such as Mel Frequency Cepstral Coefficients may as well be grouped under the term low level descriptors. These low level descriptors may be suited to generate a characteristic fingerprint of a piece of music and allow for a high refinement for the video retrieval method. All these meta data of the second group have the beneficial effect that they may be extracted from the audio signal of a piece of music automatically. These meta data may allow a comparison of first and second meta data descriptive of beats per minute BPM and/or of a further low level descriptor. If low level descriptors are calculated in the time domain only, this may save computational power.
A third group of meta data may be described as time dependent and video specific descriptors. Examples are the temporal position 410 or the order position of pieces of music 412, a length of a piece of music 414 or a temporal distance between pieces of music. In
A further group of meta data may be described as environment dependent descriptors. Examples are the confidence measure or the collaborative filtering information. These features are supporting the video retrieval functionality by describing further characteristics of pieces of music. This information is preferably comprised as well in the music database 100, if the meta data provision at S106 is conducted via this music database 100. For example the collaborative filtering information may be based purely on playback statistics e.g. of all users.
Other information, which may be used for the video retrieval, may comprise web page information that includes audio video files or is associated with audio video files. One example is shown in
The web page may as well comprise a design. This design may be used for determining the similarity measure of a first and second piece of music by comparing the design of a first web page including the first audio video file comprising the first piece of music with a design of a second web page including a second audio video file comprising the second piece of music. Such a design may be any visual element characteristic for the web page. Examples of such visual elements may be logos, emblems, background color, patterns, text elements, fonts, specific style elements or alike.
As described above the method for video retrieval according embodiments may be used for audio video file retrieval in a video database. Such a video database 700 is shown in
The video retrieval may as well be based upon extracting low level descriptors or high level descriptors from audio data comprised in audio video files. This information may be stored as meta data in a descriptor-video database, in which every audio video file is linked to its meta data.
In this embodiment a first audio video file is provided by a user, for which the user is interested to know other similar videos, one or more low level descriptors or high level descriptors are extracted from the audio file comprised in the first audio video file. These descriptors are compared to the corresponding descriptors for a second audio video file. The descriptors of the second file may be obtained by the same feature extraction as it has been conducted for the first audio video file or may be looked up in the descriptor-video database. A similarity measure is determined by comparing the descriptors of the first and second audio file and if the similarity is found to be above a threshold, the second audio video file is provided to the user. This video retrieval may be combined with the video retrieval upon piece of music identification as described above. Web page information, confidence measure or collaborative filtering information may be used to refine the similarity measure determination for this feature based analysis in the same manner as it has been described above for video retrieval method based on piece of music identification.
The execution of video retrieval may be conducted on correspondingly adapted devices or may be conducted on a system comprising a network of such devices.
In
The server 1008, 1010 may be a sever corresponding to the server 800 in
The different devices of the system may be connected by a communication connection 1000. The communication connection 100 may as well be configured to only provide a communication connection between different devices in a hierarchical communication structure.
Corresponding to the method for video retrieval a computer program product including computer program instructions that cause a computer to execute a method for video retrieval comprising—providing a first audio video file, determining a first identifier of a first piece of music in the first audio video file, looking up for the first identifier first meta data in a music database, in which said first meta data are associated with said first identifier, providing second meta data of a second piece of music from the music database, wherein the second piece of music is included in a second audio video file, determining a similarity measure by comparing the first and second meta data, and providing the second audio video file or an identifier thereof depending on the similarity measure.
Such a computer program product may be provided on a corresponding computer readable storage medium.
There may also be an embodiment as follows:
A method for video retrieval, comprising: providing, e.g. by a user, an audio video file; determining an identifier of a first piece of music in the audio video file, wherein the identifier is determined based on characteristic audio features (# fingerprint) of an audio signal of the piece of music; determining meta data and/or collaborative filtering information for the identifier independently from the audio signal based on a look-up table, wherein in the look-up table a plurality of identifiers of pieces of music are associated with respective meta data and/or collaborative filtering information; providing further meta data and/or further collaborative filtering information of a plurality of second pieces of music, wherein each of the second pieces of music is associated with a respective further audio video file of a set of further audio video files; determining a similarity measure for each of the second pieces of music, the similarity measure indicating a similarity of the first piece of music with a respective second piece of music, wherein the similarity measure is determined by comparing the meta data with the respective further meta data and/or by comparing the collaborative filtering information with the further collaborative filtering information; and providing, e.g. to the user, at least one audio video file of the set of further audio video files or an identifier thereof depending on the similarity measure.
Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of ultra net and/or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the described embodiments. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this invention be limited only by the claims and the equivalents thereof.
Number | Date | Country | Kind |
---|---|---|---|
09004184 | Mar 2009 | EP | regional |
Number | Name | Date | Kind |
---|---|---|---|
6993532 | Platt et al. | Jan 2006 | B1 |
7356830 | Dimitrova | Apr 2008 | B1 |
7685132 | Hyman | Mar 2010 | B2 |
7756915 | Ben-Yaacov et al. | Jul 2010 | B2 |
7777125 | Platt et al. | Aug 2010 | B2 |
7870574 | Kenyon et al. | Jan 2011 | B2 |
7881657 | Wang et al. | Feb 2011 | B2 |
7949050 | Xu et al. | May 2011 | B2 |
8073854 | Whitman et al. | Dec 2011 | B2 |
8208643 | Zhang | Jun 2012 | B2 |
20030105743 | Ireton | Jun 2003 | A1 |
20060080356 | Burges et al. | Apr 2006 | A1 |
20060092281 | Herley et al. | May 2006 | A1 |
20070038612 | Sull et al. | Feb 2007 | A1 |
20080059453 | Laderman | Mar 2008 | A1 |
20080109404 | Holm | May 2008 | A1 |
20090228423 | Hicken et al. | Sep 2009 | A1 |
20110035373 | Berg et al. | Feb 2011 | A1 |
Number | Date | Country | |
---|---|---|---|
20100250585 A1 | Sep 2010 | US |