This patent application is related to a co-pending U.S. Patent Application, entitled “Sorting Media Objects By Similarity”, Ser. No. 11/435,494, assigned to the same assignees as the present application.
This invention relates generally to multimedia, and more particularly finding relationships among various multimedia objects.
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the software and data as described below and in the drawings hereto: Copyright© 2005, Sony Electronics, Incorporated, All Rights Reserved.
Clustering and classification tend to be important operations in certain data mining applications. For instance, data within a dataset may need to be clustered and/or classified in a data system with a purpose of assisting a user in searching and automatically organizing content, such as recorded television programs, electronic program guide entries, and other types of multimedia content.
Generally, many clustering and classification algorithms work well when the dataset is numerical (i.e., when datum within the dataset are all related by some inherent similarity metric or natural order). Numerical datasets often describe a single attribute or category. Categorical datasets, on the other hand, describe multiple attributes or categories that are often discrete, and therefore, lack a natural distance or proximity measure between them.
Given that a user is interested in an object in a first medium, it is desirable to find objects in a different medium that the user might be interested in. For example, if the user is interested in a musical artist, it may be desirable to retrieve movies in which the musical artist's songs appear.
Metadata associated with a first multimedia object in a first medium is used to find a second multimedia object in a second medium. The metadata includes category data.
The present invention is described in conjunction with systems, clients, servers, methods, and machine-readable media of varying scope. In addition to the aspects of the present invention described in this summary, further aspects of the invention will become apparent by reference to the drawings and by reading the detailed description that follows.
The present invention is illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.
In the following detailed description of embodiments of the invention, reference is made to the accompanying drawings in which like references indicate similar elements, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that logical, mechanical, electrical, functional, and other changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.
The category data 11 is grouped into clusters, and/or classified into folders by the clustering/classification module 12. Details of the clustering and classification performed by module 12 are below. The output of the clustering/classification module 12 is an organizational data structure 13, such as a cluster tree or a dendrogram. A cluster tree may be used as an indexed organization of the category data or to select a suitable cluster of the data.
Many clustering applications require identification of a specific layer within a cluster tree that best describes the underlying distribution of patterns within the category data. In one embodiment, organizational data structure 13 includes an optimal layer that contains a unique cluster group containing an optimal number of clusters.
A data analysis module 14 may use the folder-based classifiers and/or classifiers generated by clustering operations for automatic recommendation or selection of content. The data analysis module 14 may automatically recommend or provide content in a first medium that may be of interest to a user or otherwise similar or related to content used by a user in a different medium.
In one embodiment, a user identifies multiple folders of category data records that categorize specific content items, and the data analysis module 14 assigns category data records for new content items with the appropriate folders based on similarity. In another embodiment, data analysis module 14 comprises similarity module 18 that automatically determines cross-media object relations. Determining cross-media object relations is further described in
A user interface 15 also shown in
Clustering is a process of organizing category data into a plurality of clusters according to some similarity measure among the category data. The module 12 clusters the category data by using one or more clustering processes, including seed based hierarchical clustering, order-invariant clustering, and subspace bounded recursive clustering. In one embodiment, the clustering/classification module 12 merges clusters in a manner independent of the order in which the category data is received.
In one embodiment, the group of folders created by the user may act as a classifier such that new category data records are compared against the user-created group of folders and automatically sorted into the most appropriate folder. In another embodiment, the clustering/classification module 12 implements a folder-based classifier based on user feedback. The folder-based classifier automatically creates a collection of folders, and automatically adds and deletes folders to or from the collection. The folder-based classifier may also automatically modify the contents of other folders not in the collection.
In one embodiment, the clustering/classification module 12 may augment the category data prior to or during clustering or classification. One method for augmentation is by imputing attributes of the category data. The augmentation may reduce any scarceness of category data while increasing the overall quality of the category data to aid the clustering and classification processes.
Records are represented within the data system 10 as data vectors. The dimension of each vector corresponds to the total number of terms characterizing all attributes found in all records processed by the system (i.e., the global vocabulary of the system). Values assigned to components of a vector represent the presence of a term within a corresponding record. For example, a vector component can be represented in a binary fashion as either a 0 (indicating the absence of a term from a record) or a 1 (indicating the presence of a term in a record).
Although shown in
Category data describes the different categories associated with the content. For example, category data 158 comprises terms: Best, Underway, Sports, GolfCategory, Golf, Art, 0SubCulture, Animation, Family, FamilyGeneration, Child, Kids, Family, FamilyGeneration, and Child. As illustrated, category data 158 comprises fifteen terms describing the program. Some of the terms are related, for example, “Sports, GolfCategory, Golf” are related to sports, and “Family, FamilyGeneration, Child, Kids”, are related to family. Furthermore, category data 158 includes duplicate terms and possibly undefined terms (0SubCulture). Undefined terms may be only associated with one program, because the definition is unknown and, therefore, not very useful.
Artists 170 comprise the list of artists associated with the program. For example, artists 170 comprise artist1, artist2, . . . , artistN. An artist can be, but not limited to, an actor, actress, producer, director, singer, musician, arranger, composer, choreographer, painter, illustrator, author, etc., and/or any person who adds to the creative content of the program.
According to an embodiment of the present invention, objects that belong to different types of multimedia are related together. Accordingly, for instance, given a song that a user is interested in, the artist(s) of the song is related to one or more movies by looking at the music that appears in the movies' soundtrack. For example, the song “Birthday” that is written by Paul McCartney and John Lennon may be used to find movies in which Paul McCartney's and John Lennon's songs appear.
Further, according to an embodiment of the present invention, objects that belong to different types of multimedia are related together, even when there is no direct connection between the objects. Accordingly, for example, a musical artist is related to one or more movies by looking at the music that appears in the movies' soundtracks, even when the musical performer did not perform in any of the songs that appear in the movies' soundtracks.
According to an embodiment of the present invention, a filtering system is provided that presents the user with movies of potential interest based on the user's interest in musical artists.
One embodiment of a method 300 to be performed by a data analysis module 14 to relate multimedia objects across various mediums is described with reference to a flowchart shown in
At block 201, the method 200 reads metadata associated with the object of interest. In an embodiment, the metadata is read in dynamically and may be tailored specific to user entered information. For example, the metadata is read in response to a search query by a user. The metadata is in the form of category data 11.
For instance, metadata for a song may include such attributes as “song writer name(s)”, “performer name(s)”, year of production, genre, name of album and so on. Each attribute may have one or more values. For example, the attribute “song writer name(s)” for the song “Birthday” may have two values—Paul McCartney and John Lennon. The attribute “performer name(s)” for that song may have just one value—the Beatles. The year of production for this song has one value: 1968. The name of album for this song has one value—The White Album. In an embodiment, the artist metadata is generated from an artist metadata database, publicly available sources (Wikipedia®, GRACENOTE®, etc.), and so on. The artist names may be used to search for cross-media objects that may be of interest to a user, e.g., movies.
The metadata may also include name-mapping data, which may map for instance, the name of an artist to various spellings of the name (including common misspellings).
At block 211, the method 300 searches for objects considered relevant to the object of interest in a medium different from that of the object of interest. Accordingly, exact matches for object of interest metadata may be found. For instance, when comparing an artist name related to song of interest to names of artists that appear in soundtracks of movies, the artists' names are represented as strings. Artist name may be mapped to object identifier at distance 0. This means that there is no difference between the artist name related to song of interest and one or more names of artists that appear in soundtracks of movies.
Accordingly, a collection of objects is searched to look for the occurrence of the exact object of interest metadata, e.g., the exact artist name spelling. A list of relevant objects in other mediums is obtained at block 221. In an embodiment, if the list of objects thus obtained is large enough, e.g., greater than a predetermined threshold, the method ends here. Otherwise, the method continues to block 231.
At block 231, the method 200 searches for more objects that are close matches to object of interest metadata. Thus, in the example given above, musical artist's name may be mapped to name at distance 1. Accordingly, the plurality of objects is searched to look for the occurrence of a close object of interest metadata, e.g., the artist name spelling with one letter off.
A list of relevant objects in other mediums may thus be obtained at block 241. In an embodiment, if the list of objects thus obtained is large enough, e.g., greater than a predetermined threshold, the method ends here. Otherwise, the method continues to block 301 of
One of ordinary skill in the art will appreciate that searching for objects that are not as close matches for object of interest metadata may result in getting mismatches. For example, the artist name spelling with two letters off may result in finding objects for different artists than the one intended.
At block 301, a value “current_similarity” is set to zero. At block 311, a loop starts. While current similarity is determined to be less than a predetermined limit and the loop is “not done”, at block 321, the value of done is set to true. Having these two restrictions on the loop helps to ensure that the loop does not go on infinitely. The predetermined limit may be set by a user. For instance, the user may specify a limit for search results in a user profile or when performing a search query. Alternatively, the limit may be learned or estimated by the data system 10 based, for instance, on user behavior.
At block 331, it is determined if enough cross-media objects related to the object of interest have been found. If the answer is no, then at block 341, the method 200 reads in similarity data. The similarity data is used to find objects that are considered similar to the object of interest itself at block 351. At block 351, for each object that is considered similar to the selected object, it is determined if the similar object is related to any objects in a medium different from that of the object of interest.
At block 361, the similar object's results are added to the object of interest search results. The loop continues until it is determined that there are enough objects (e.g., movies) relevant to the current artist. In this way, the method 200 finds objects (e.g., movies) even when no cross-media objects come up for an object of interest.
At block 371, the value of current_similarity is incremented and the loop continues until current_similarity is no longer less than the limit and the value of done is false. At block 381, the list of found cross-media objects (e.g., movies) is output to the user.
In an embodiment, similarity among found target objects is used in addition to similarity among object of interest and other objects to enhance results.
In an embodiment, other attributes that are shared by two objects (the object of interest and the target object) are used to enhance results. For example, the year a song was produced may be used to find movies that were also released in the same year.
In an embodiment, attributes of objects are mapped to each other, e.g., a musical genre can be mapped to a movie genre.
One embodiment of a method 400 to be performed by a data analysis module 14 to relate a song to movies is described with reference to flowchart shown in
At block 401, the method 400 reads metadata related to a song of interest to a user. The metadata may include soundtrack data for each song to which the musical artist contributed, which may include, e.g., song title data, song-writer data for each song, song-performer for each song, genre of song, and so on. The metadata may also include name-mapping data. The metadata may further include similarity data, which may include a list of artists or bands that are considered similar. The method 400 finds movies related to all artists of the song of interest to the user.
For each artist, the method 400, at block 411, maps to name at distance 0. Accordingly, each time the artist's name exactly appears in the soundtrack of another movie, the movie is considered relevant. For example, all songs that contain in their metadata the string “John Lennon” are considered relevant. A list of relevant movies may be obtained at block 421. In an embodiment, if the list of movies is large enough, e.g., greater than a predetermined threshold, the method ends here. Otherwise, the method continues to block 431.
At block 431, for each artist, the method 400, maps to name at distance 1. For example, all songs that contain in their metadata the string “John Lannon” are considered relevant. A list of relevant movies may be thus obtained at block 441. At block 451, current_similarity is set to zero. While current similarity is determined to be less than a predetermined limit and a loop is not finished, for the selected artist, at block 461, it is determined if enough movies have been found at block 481. If the answer is no, then similarity metadata is read in at block 491 and used to find artists that are considered similar to the artist of the song of interest to user at block 501. Artists considered similar to an artist of interest may be determined by device 18 that sorts media objects by similarity if they have the same measure of similarity (e.g., a rank). For example, the similarity metadata may indicate that the musical artist George Harrison is similar to John Lennon.
At block 511, for each artist that is considered similar, it is determined if the artist is related to any movies. Thus, movies may be found for George Harrison. Movies may be found in the same way as for the original artist of interest. At block 521, the similar artist's movies are added to the current artist movie list. The loop continues until it is determined that there are enough movies relevant to the current artist. In this way, the method 400 finds movies even when no movies come up for a musical artist of interest to a user.
At block 511, the value of current_similarity is incremented and the loop continues until current similarity is no longer less than the limit and the value of done is false. A list of movies is output to the user. The method 400 continues for each artist included in the metadata of the song of interest to find movies of potential interest to the user.
The following description of
In practice, the methods described herein may constitute one or more programs made up of machine-executable instructions. Describing the method with reference to the flowchart in
The web server 1108 is typically at least one computer system which operates as a server computer system and is configured to operate with the protocols of the World Wide Web and is coupled to the Internet. Optionally, the web server 1108 can be part of an ISP which provides access to the Internet for client systems. The web server 1108 is shown coupled to the server computer system 1110 which itself is coupled to web content 842, which can be considered a form of a media database. It will be appreciated that while two computer systems 1108 and 1110 are shown in
Client computer systems 1112, 1116, 1124, and 1126 can each, with the appropriate web browsing software, view HTML pages provided by the web server 1108. The ISP 1104 provides Internet connectivity to the client computer system 1112 through the modem interface 1114 which can be considered part of the client computer system 1112. The client computer system can be a personal computer system, a network computer, a Web TV system, a handheld device, or other such computer system. Similarly, the ISP 1106 provides Internet connectivity for client systems 1116, 1124, and 1126, although as shown in
Alternatively, as well-known, a server computer system 1128 can be directly coupled to the LAN 1122 through a network interface 1134 to provide files 1136 and other services to the clients 1124, 1126, without the need to connect to the Internet through the gateway system 1120. Furthermore, any combination of client systems 1112, 1116, 1124, 1126 may be connected together in a peer-to-peer network using LAN 1122, Internet 1102 or a combination as a communications medium. Generally, a peer-to-peer network distributes data across a network of multiple machines for storage and retrieval without the use of a central server or servers. Thus, each peer network node may incorporate the functions of both the client and the server described above.
Network computers are another type of computer system that can be used with the embodiments of the present invention. Network computers do not usually include a hard disk or other mass storage, and the executable programs are loaded from a network connection into the memory 1208 for execution by the processor 1204. A Web TV system, which is known in the art, is also considered to be a computer system according to the embodiments of the present invention, but it may lack some of the features shown in
It will be appreciated that the computer system 1200 is one example of many possible computer systems, which have different architectures. For example, personal computers based on an Intel microprocessor often have multiple buses, one of which can be an input/output (I/O) bus for the peripherals and one that directly connects the processor 1204 and the memory 1208 (often referred to as a memory bus). The buses are connected together through bridge components that perform any necessary translation due to differing bus protocols.
It will also be appreciated that the computer system 1200 is controlled by operating system software, which includes a file management system, such as a disk operating system, which is part of the operating system software. One example of an operating system software with its associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Wash., and their associated file management systems. The file management system is typically stored in the non-volatile storage 1214 and causes the processor 1204 to execute the various acts required by the operating system to input and output data and to store data in memory, including storing files on the non-volatile storage 1214.
In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications may be made thereto without departing from the broader spirit and scope of the invention as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.
Number | Name | Date | Kind |
---|---|---|---|
5566291 | Boulton et al. | Oct 1996 | A |
5764239 | Misue et al. | Jun 1998 | A |
5963746 | Barker et al. | Oct 1999 | A |
6105046 | Greenfield et al. | Aug 2000 | A |
6208963 | Martinez et al. | Mar 2001 | B1 |
6256648 | Hill et al. | Jul 2001 | B1 |
6282548 | Burner et al. | Aug 2001 | B1 |
6373484 | Orell et al. | Apr 2002 | B1 |
6460036 | Herz | Oct 2002 | B1 |
6473851 | Plutowski | Oct 2002 | B1 |
6484199 | Eyal | Nov 2002 | B2 |
6513027 | Powers et al. | Jan 2003 | B1 |
6539354 | Sutton et al. | Mar 2003 | B1 |
6545209 | Flannery et al. | Apr 2003 | B1 |
6584456 | Dom et al. | Jun 2003 | B1 |
6592627 | Agrawal et al. | Jul 2003 | B1 |
6625585 | MacCuish et al. | Sep 2003 | B1 |
6668273 | Rust | Dec 2003 | B1 |
6714897 | Whitney et al. | Mar 2004 | B2 |
6732145 | Aravamudan et al. | May 2004 | B1 |
6738678 | Bharat et al. | May 2004 | B1 |
6748418 | Yoshida et al. | Jun 2004 | B1 |
6785688 | Abajian et al. | Aug 2004 | B2 |
6801229 | Tinkler | Oct 2004 | B1 |
6941300 | Jensen-Grey | Sep 2005 | B2 |
6996575 | Cox et al. | Feb 2006 | B2 |
7003515 | Glaser et al. | Feb 2006 | B1 |
7085736 | Keezer et al. | Aug 2006 | B2 |
7158983 | Willse et al. | Jan 2007 | B2 |
7162691 | Chatterjee et al. | Jan 2007 | B1 |
7165069 | Kahle et al. | Jan 2007 | B1 |
7184968 | Shapiro et al. | Feb 2007 | B2 |
7185001 | Burdick et al. | Feb 2007 | B1 |
7203698 | Yamashita | Apr 2007 | B2 |
7216129 | Aono et al. | May 2007 | B2 |
7325006 | Fortnan et al. | Jan 2008 | B2 |
7330850 | Seibel et al. | Feb 2008 | B1 |
7340455 | Platt et al. | Mar 2008 | B2 |
7371736 | Shaughnessy et al. | May 2008 | B2 |
7392248 | Bakalash et al. | Jun 2008 | B2 |
20010045952 | Tenev et al. | Nov 2001 | A1 |
20020035603 | Lee et al. | Mar 2002 | A1 |
20020042793 | Choi | Apr 2002 | A1 |
20020099696 | Prince | Jul 2002 | A1 |
20020099731 | Abajian | Jul 2002 | A1 |
20020099737 | Porter et al. | Jul 2002 | A1 |
20020107827 | Benitez-Jimenez et al. | Aug 2002 | A1 |
20020138624 | Esenther | Sep 2002 | A1 |
20030011601 | Itoh et al. | Jan 2003 | A1 |
20030033318 | Carlbom et al. | Feb 2003 | A1 |
20030041095 | Konda et al. | Feb 2003 | A1 |
20030041108 | Henrick et al. | Feb 2003 | A1 |
20030084054 | Clewis et al. | May 2003 | A1 |
20030089218 | Gang et al. | May 2003 | A1 |
20030105819 | Kim et al. | Jun 2003 | A1 |
20030154181 | Liu et al. | Aug 2003 | A1 |
20030217335 | Chung et al. | Nov 2003 | A1 |
20040083236 | Rust | Apr 2004 | A1 |
20040090439 | Dillner | May 2004 | A1 |
20040117367 | Smith et al. | Jun 2004 | A1 |
20040133555 | Toong et al. | Jul 2004 | A1 |
20040133639 | Shuang et al. | Jul 2004 | A1 |
20040193587 | Yamashita | Sep 2004 | A1 |
20040215626 | Colossi et al. | Oct 2004 | A1 |
20040260710 | Marston et al. | Dec 2004 | A1 |
20050027687 | Nowitz et al. | Feb 2005 | A1 |
20050033807 | Lowrance et al. | Feb 2005 | A1 |
20050060350 | Baum et al. | Mar 2005 | A1 |
20050114324 | Mayer | May 2005 | A1 |
20050289109 | Arrouye et al. | Dec 2005 | A1 |
20050289168 | Green et al. | Dec 2005 | A1 |
20060023724 | Na et al. | Feb 2006 | A1 |
20060025175 | Lapstun et al. | Feb 2006 | A1 |
20060112141 | Morris | May 2006 | A1 |
20060122819 | Carmel et al. | Jun 2006 | A1 |
20060167942 | Lucas et al. | Jul 2006 | A1 |
20060218153 | Voon et al. | Sep 2006 | A1 |
20060282789 | Kim | Dec 2006 | A1 |
20070005581 | Arrouye et al. | Jan 2007 | A1 |
20070061319 | Bergholz | Mar 2007 | A1 |
20070130194 | Kaiser | Jun 2007 | A1 |
20070192300 | Reuther et al. | Aug 2007 | A1 |
20070233730 | Johnston | Oct 2007 | A1 |
20070245373 | Shivaji-Rao et al. | Oct 2007 | A1 |
20080133466 | Smith et al. | Jun 2008 | A1 |
20080301121 | Suzuki et al. | Dec 2008 | A1 |
20080313214 | Duhig et al. | Dec 2008 | A1 |
Entry |
---|
Pavel Moravec, Michal Kolovrat, and Vaclav Snasel, “LSI vs. Wordnet Ontology in Dimension Reduction for Information Retrieval”, 2004, pp. 18-26, ISBN 80-248-0457-3. |
Ana B. Benitez, John R. Smith, Shih-Fu Chang, “MediaNet: A Multimedia Information Network for Knowledge Representation”, in Proc., SPIE, 2001. |
Lawrence Reeve and Hyoil Han, “Semantic Annotation for Semantic Social Networks”, Using Community Resources, AIS SIGSEMIS Bulletin, vol. 2, Issue (3&4), 2005, pp. 52-56. |
Maria Ruiz-Casado, Enrique Alfonseca and Pablo Castells, “Automatic Extraction of Semantic Relationships for WordNet by Means of Pattern Learning From Wikipedia”, Castelis in Lecture Notes in Computer Science, vol. 3513, 2005. |
George A. Miller, Richard Beckwith Christiane Fellbaum, “Introduction to WordNet: An On-Line Lexical Database”, Int. J. Lexicography, 1990, vol. 3, pp. 235-244. |
Sebastian, F., “Machine Learning in Automated Text Categorization”, Mar. 2002, ACM, vol. 34, Issue 1, pp. 1-47. |
Chen, et al., “Predicting Category Accesses for a User in a Structured Information Space”, Aug. 2002, ACM, pp. 65-72. |
Number | Date | Country | |
---|---|---|---|
20070271264 A1 | Nov 2007 | US |