Disclosed herein are, for example, methods, computer program, computer program product and indexing systems for indexing or updating an index. In one aspect, the disclosure is related to, inter alia, indexing time-based content (e.g., video content, audio content, and audiovisual content) to enable a user to easily find content of interest to the user. The disclosure relates to several embodiments of such a system.
The World Wide Web (WWW) grows larger and larger every day, and users access it more often on a daily basis using a variety of devices, such as Personal computers (PC), phones, tablets, cameras and Internet Protocol Television (IP-TV) devices. Mobile technologies make it easier to capture audiovisual content and different social network and video sharing sites make it possible to share such content on the Internet. As technology becomes smarter the sharing of audiovisual content becomes more common. The amount of content is growing rapidly and researchers are working on smarter ways to search, retrieve and share such content.
Today audiovisual content can be searched on web sites, such as YouTube (www.youtube.com, retrieved on Aug. 16, 2013), so-called “play channels” owned by traditional TV broadcasters, Video-on-Demand sites, etc. Videos are stored and tagged using conventional audiovisual metadata like title, description, duration, keywords, video category (like “Action”, “Comedy” etc.).
There are a number of commercial search engines available today that use proprietary algorithms for indexing and search. A first example is YouTube (www.youtube.com, retrieved on 16 Aug. 2013). Searching for a specific keyword via YouTube can provide three different types of results: 1) Videos (e.g., a link to a video which is relevant to the search query); 2) Playlists (e.g., link to a playlist, an ordered set of videos, which are relevant to the search query); and 3) Channels (e.g., a link to a channel (user account) which contains videos relevant to the search).
Another example is Vimeo (www.vimeo.com, retrieved on 16 Aug. 2013), wherein the search function is segmented into different types of content, and the user is required to select which type of search results that is expected. By default, a search will display a list of matching videos, but the user is able to select to instead search for people, channels, groups or forum.
Yet another example is Google (www.google.com, retrieved on 16 Aug. 2013). Here a standard web search can present results ordered hierarchically. The most common result on a web search is a list of web pages which match the search query. For each of these, the user can expand to see exactly where in the page the search query is matched. Additionally, relevant pages within the same site are displayed adjacently, with the most relevant result for said site being displayed on top. The Google search engine searches web pages rather than video content.
What is desired is, for example, a search engine system having an improved content indexing system that enables a user to more easily find the content the user is searching for.
Described herein are various embodiments for indexing or updating an index. An object of at least one embodiment may be to enable improved search accuracy for multimedia content in relation to the prior art mentioned in the background. Another object of at least one embodiment may be to enable more effective techniques for generating accurate search indices for multimedia content. Embodiments of the indexing and searching system disclosed herein use a time range, hereinafter sometimes called Window of Relevance (WoR), associated with keywords (which are also referred to as “tags”). The WoR enables a more effective display of search results in, for example, a graphical user interface (GUI) displayed by a user equipment (UE) (e.g., a computer, smartphone, etc), by displaying at least one word adjacent to a displayed keyword used in a search. The at least one word makes it easier for the user to guess or know if a hit in the search result is relevant or not. Other possible technical advantages is that the implementation of some embodiments of the indexing and search system becomes more dynamic and/or generic by e.g. making the indexing more unrelated to more “stationary segments” and their size. Some embodiments also make the searching multimedia content faster than conventional methods.
In one aspect there is provided a method performed by an indexing system for indexing or updating an index for content. In some embodiments, the method comprises for each one of a plurality of keywords in a data storage system, wherein each keyword is associated with a modality and at least a first time stamp, retrieving from the data storage system a keyword and its associated first time stamp. The method also comprises defining a time range for the keyword and determining which other keywords in the data storage system are associated with a time stamp within the time range. The method further comprises updating the index with the other keywords that were determined to be associated with a time stamp within the time range. The method may also comprise updating the index with a rank value and modality of each one of the determined other keywords with time stamps within the time range.
In some embodiments, the method also comprises, before the step of retrieving the keyword from the data storage system, on at least a plurality of frames of a video: identifying one or more features present in the frames, wherein each one of the features are of at least one modality. The method according to the embodiments also includes identifying, for each identified feature, a keyword associated with the identified feature and generating the at least one time stamp for each one of the identified keywords. The method according to the embodiments further includes storing the keywords and their respective time stamps in the data storage system.
In some embodiments, the time range is defined as starting a certain time before the time of the first time stamp and ending the certain time after the first time stamp.
In some embodiments, the keyword is further associated with a second time stamp and the time range starts a certain time before the time of the first time stamp and ends a certain time after the second time stamp.
In another aspect there is provided an indexing system. The indexing system comprises a computer system and data storage system. The data storage system contains instructions executable by the computer system. The indexing system is operative to, for each one of a plurality of keywords in the data storage system, wherein each keyword is associated with a modality and at least a first time stamp: retrieve from the data storage system a keyword and its associated first time stamp and define a time range for the keyword. The indexing system is further operative to determine which other keywords in the data storage system are associated with a time stamp within the time range. The indexing system is further operative to update an index with the other keywords that were determined to be associated with a time stamp within the time range.
In another aspect there is provided a computer program for indexing content. In some embodiments, the computer program comprises computer readable instructions. The computer readable instructions are configured such that when run on a computer system they cause the computer system to retrieve a keyword from a plurality of keywords stored in a data storage system, wherein the retrieved keyword is associated with a modality and at least a first time stamp, and define a time range for the keyword. The computer readable instructions further cause the computer system to determine which other keywords in the data storage system are associated with a time stamp within the time range. The computer readable instructions further cause the computer system to update an index with the other keywords that were determined to be associated with a time stamp within the time range.
In another aspect there is provided an indexing system for indexing or updating an index for content. In some embodiments, the indexing system comprises means for retrieving from a data storage system a keyword and a time stamp associated with the keyword. The indexing system further comprises means for defining a time range for the keyword. The indexing system further comprises means for determining which other keywords in the data storage system are associated with a time stamp within the time range. The indexing system further comprises means for updating an index with the other keywords that were determined to be associated with a time stamp within the time range.
In another aspect there is provided a content indexing method performed by a indexing and search system. In some embodiments, the method comprises storing in a data storage system a plurality of keywords, the plurality of keywords including a first keyword and a second keyword. The method further comprises defining a time range for the first keyword. The method further comprises, after defining the time range for the first keyword, determining that the second keyword falls within the defined time range. The method further comprises, in response to determining that the second keyword falls within the defined time range, updating a first set of keywords by adding the second keyword to the first set of keywords, wherein the first set of keywords is associated with the first keyword.
The above and other aspects and embodiments are described below with reference to the accompanying drawings.
The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.
Even though it is not illustrated in index 106, the index 106 may also comprise the ranking/weight value of the respective matching keywords in the list 214.
In some embodiments, the rank assigned to a keyword k depends on the modality of k and whether k overlaps with other keywords. For example, if we assume k overlaps with N other keywords, then we can use a formula of the following form to calculate the rank R_k to assign to keyword k: R_k=Mv_k+Ov_k,1+Ov_k,2+ . . . +Ov_k,N, where Mv_k is a value assigned to the modality of k and Ov_k,i is an overlap value that is dependent on the degree of overlap between the keyword k and keyword i. For instance, Ov_k,i may equal: ((k_end−i_start)/(i_end−i_start))*Mv_i, where k_end is the end TS value for keyword k, i_start is the start TS for keyword i, i_end is the end TS for keyword i, and Mv_i is the modality value for keyword i. Using the above formula and assuming that the modality value for keywords having a “text” modality is 5, we can calculate the rank for the keyword “intro” as follows: R_intro=5+((8−6)/10−6))*5=7.5
Referring now to
In step 302, query processor 108 receives a query that was input by the user into communication device and transmitted by the communication device 101 to query processor 108, where the query comprises a keyword (e.g., Obama). In some embodiments, query processor 108 includes a Hypertext Transfer Protocol (HTTP) server. Thus, in such embodiments, the user may input the basis (such as the keyword) for the query into a web page form and initiate the submission of the query to the query processor 108 in an HTTP GET message. That is, for example, the web page form may have a text input box to enable the user to input a text based query and a “search” button. When the user activates the search button after inputting into the text input field a query, the user's browser causes the communication device 101 on which the browser is running to transmit to query processor 108 an HTTP GET message that includes the text (i.e., query) that the user input into the text input box.
In step 304, query processor 108 determines that the keyword included in the query matches a first keyword (e.g., keyword 204a “Obama”) included in the index 106.
In step 306, query processor 108 obtains a second keyword (e.g., keyword 204b “flag”) that overlaps with the first keyword 204a (e.g., Obama). For example, query processor may obtain the second keyword 204b by retrieving from index 106 record 202a, which includes a set of keywords 214a (i.e., CNN and flag) that overlaps with the Obama keyword 204a.
In step 308, query processor 108 generates search result information, wherein the search result information comprises: a) a first locator identifying a first image file associated with the first keyword (e.g., a thumbnail image of President Obama) and a second locator identifying a second image file associated with the second keyword (e.g., a thumbnail image of a flag); and/or b) the first keyword and the keywords that overlap the first keyword (e.g., the second keyword). Query processor 108 may obtain the first and second locators from information contained in index 106. For example, while not shown, the first locator may be stored in a field of record 202a and the second locator may be stored in a field of record 202b. As another example, query processor 108 may construct the first and second locators using information contained in records 202a and 202b, respectively.
In step 310, query processor 108 adds the generated search result information to a search results page.
In step 312, query processor 108 provides the search results page to the communication device 101 which displays the page to the user. For example, the search results page may be in the form of a Hypertext Markup Language (HTML) document) and query processor may use HTTP to transmit the HTML document to the user's communication device 101 (e.g., laptop computer, smartphone, etc).
In this way, indexing and search system 100 provides the advantage of not only providing the user with information associated with the keywords contained in the user's query (e.g., Obama), but also providing to the user additional information associated with keywords (e.g., flag) that overlap the keywords of the user's query. This additional information can be very advantageous to the user because it provides additional contextual information that may enable the user to find the search result from the search results page that bests matches what the user was searching for.
Referring now to
In step 404, indexing system 104, for each video frame of the video, analyzes the frame to produce frame data, which frame data includes a set of keywords. For example, in step 404, indexing system 104 may use image recognition software to recognize and label images found within each video frame. The label given to an image identified in a video frame becomes a keyword for that video frame. In this way, indexing system 104 generates keywords for a video.
In step 406, for each keyword obtained from performing step 404, indexing system 104 analyzes the frame data to determine for the keyword a time range (a.k.a., WoR) (e.g., a video frame range), which consists of a starting time stamp and an ending time stamp. Time may be measured in units of seconds, units of frames, or any other unit that can be used to identify a position (e.g., frame) within the video. Thus, for example, in some embodiments, a time stamp is a value having units of time (e.g., seconds), while in some other embodiments, a time stamp is a value having units of frames.
In step 440, which may be performed concurrently with one or more of steps 404-406, indexing system 104 analyzes the audio portion of the video to extract keywords from the audio portion. In step 445, indexing system 104 assign a time range to each keyword. After step 445, process 400 proceeds to step 408.
In step 408, for each keyword obtained from performing steps 404 and 440, indexing system 104 determines other keywords that “overlap” the keyword. In some embodiments, a first keyword overlaps a second keyword if there is any overlap in the time ranges for the two keywords.
In step 410, for each keyword obtained from performing steps 404 and 440, indexing system 104 updates the index 106 with the keyword and the determined other keywords that overlap the keyword. For example, in step 410, for each keyword, indexing system 104 inserts into index 106 a new record for the keyword, wherein the keyword is stored in the “keyword” field 204 of the record and the keywords that were determined to overlap with the keyword are stored in the “overlap” field 214 of the record. As shown in
Referring now to
In step 506, for each keyword in frame i, indexing system 104 sets start-index=i and adds to a Temporary Keyword Set (TKS) a keyword vector (keyword, start-index) for the keyword. In step 508, indexing system sets i equal to the value i+x, where x is a whole number greater than zero. In step 510, indexing system 104 determines whether frame i exists in the video. If it does not, then process proceeds to step 514, otherwise it proceeds to step 512. In step 512, indexing system 104 analyzes frame i of the video to produce a set of keywords for frame i.
In step 514, for each keyword vector included in TKS that does not include a keyword that is included in the set of keywords for frame i, indexing system 104 removes from TKS the keyword vector (keyword, start-index) for the keyword, sets end-index equal to the value of i, and stores in a data storage system 1206 (see
In step 516, for each keyword in the set of keywords for frame i that is not included in TKS, indexing system 104 sets start-index=i, and adds a keyword vector (keyword, start-index) for the keyword to TKS.
In the above manner, indexing system can determine a time range (e.g., a video frame range) for each keyword.
Referring now to
Referring now to
In step 704, query processor 108 determines that the keyword included in the query matches a first keyword (e.g., a keyword included in index 106).
In step 706, query processor 108 obtains a first time stamp (FTS) associated with the first keyword. For example, the FTS may be a start time stamp associated with the first keyword (e.g., the start TS value that is stored in the same record of index 106 in which the first keyword is stored).
In step 708, query processor 108 defines the time range using FTS and OV. For example, the time range may be defined by a start time stamp and an end time stamp, where the start time stamp is equal to (FTS−OV) and an end time stamp is equal to a second time stamp (STS) plus OV (i.e., STS+OV). STS may be an end time stamp associated with the first keyword.
In step 710, query processor 108 determines a set of keywords other than the first keyword that are located within the defined time range. That is, as described above, the keywords that overlap with the first keyword are found using, for example, the start and end time range values stored in index 106.
In step 712, query processor 108 generates a search result using the first keyword and the determined set of other keywords. In step 714, query processor 108 provides the search result to the communication device 101.
In the above manner, the overlapping keywords are determined in real-time in response to a query.
In step 804, indexing system 104, for each video frame of the video, analyzes the frame to produce frame data, which frame data includes a set of keywords. For example, in step 804, indexing system 104 may use image recognition software to recognize and label images found within each video frame. The label given to an image identified in a video frame becomes a keyword for that video frame.
In step 806, for each keyword obtained from performing step 804, indexing system 104 analyzes the frame data to determine for the keyword at least a starting time stamp. In some embodiments, for each keyword, indexing system 104 determines a starting and an ending time stamp for the key word.
In step 840, which may be performed concurrently with one or more of steps 804-806, indexing system 104 analyzes the audio portion of the video to extract keywords from the audio portion. In step 845, indexing system 104 assigns a starting time stamp to each of the extracted keywords from step 840. After step 845, process 800 proceeds to step 810.
In step 810, for each keyword obtained from performing steps 804 and 840, indexing system 104 updates the index 106 with the keyword and at least the keyword's starting time stamp. For example, in step 810, for each keyword, indexing system 104 inserts into index 106 a new record for the keyword, wherein the keyword is stored in the “keyword” field 204 of the record, the starting time stamp is stored in the “start TS” field 208, and, if an ending time stamp exists, the ending time stamp is stored in the “end TS” field 210. As shown in
As explained upon above, a feature of various embodiments is an algorithm that extracts and indexes keywords within a window of relevance (WoR). The WoR is in one embodiment set for each key word, t(keyword)±t(WoR), wherein t(keyword) is a time stamp associated with a tag/keyword in e.g. a multimedia content, such as a video, and t(WoR) is a time, which may be fixed for all modalities, fixed but different for different modalities, such as one t(WoR) value for features/faces in images/frames in a video and another value for derived keywords from speech and still another value for derived keywords from sound and yet another value for text (such as derived words from subtitles). Other ways of setting the WoR than fixed values in the above mentioned way, may be to set it more dynamic so that at least one other keyword comes within the window of relevance, so that it is easily enabled that at least one keyword is displayed in a search result list adjacent to the main keyword, which in its turn may be highlighted to make it easier for the user to judge the relevance of the hit in the result list or result view.
The WoR may in some embodiments also be made to be dependent on the type of keyword itself or a general tag associated with the content. An action movie might for example be made to have different window of relevance than e.g. a romantic, slow paced movie, whereas talk shows may have still another set of values for the WoR. Yet other ways of setting the WoR is to make it dependent on whether the keyword is present in more than one modality at the same time, e.g. has a rank value above a certain threshold, and/or dependent on the length of the video.
In some embodiments, the algorithm included into the indexing process enables an improvement of the search capabilities for audio visual content. The indexing and window of relevance algorithm here, in some embodiments, takes a given number of frames per second in a video and processes each frame as an image. The images are then analyzed for metadata like text (using e.g. OCR), faces (using face detection algorithms), voice, objects and other metadata and could in some embodiments be combined with manual tagging of the content. All metadata will be indexed, see
As illustrated in
I. A multimedia content, in this example a video, is captured or uploaded to a server, e.g. the ISS.
II. The WoR algorithm start analyzing the uploaded video, e.g. already when the uploading is made, and in the following way:
a) For each modality (image, voice, text, sound etc) analyze each frame in the video:
i) For each frame:
ii) For each keyword in Vector Space/RAM:
The steps define WoR, Add related keywords and add modality builds up the meta data and the structure for the data which shall be inserted into the database. In the Add modality step, it is defined what you actually intends to store in the database, such as info indicating modality and what the rank value is for that modality, but in some embodiments it may be sufficient with only indicating the rank value if there is no need to indicate modality in the index and/or database. In case json-files are created, the json-files may be aggregated and kept in the vector space and sorted according to the first time stamp. Crosslinking can be done to provide the ranking and matching keywords. In preparation for at least one of the two next updating steps, there may in one embodiment in practice be that a new list is created and which include ranking and modality is included and wherein every row is formatted with every first time stamp.
In case of a video and persons tagged in the video, the algorithm may find a certain amount of people in the video and it may be checked if the person shows up in a plurality of times in the video (plurality=more than one). The ranking of the person/object in the index can then be made dependent on how long the person/object is visible in the movie and even get a higher ranking if he/she/it is mentioned or talks a lot in the movie.
Index 106 shown in
Hence, when a search is made for the keyword “Obama”, the index table displaying the most relevant hits in the GUI in e.g. a web browser connecting to the indexing and search system 100, may also display e.g. adjacent to a highlighted word “Obama” or a picture of Mr President Obama taken from the video (to more readily show that Obama is actually visible in that hit and not only talked about etc) display the text “CNN”. By displaying CNN next to the picture of Mr President Obama, the searcher should be able to more readily guess the context and relevance of the video in relation to what is actually searched for.
Another, somewhat different embodiment is described in the following:
The window of relevance algorithm is here included in the search function, e.g. as an added feature to an existing query processor 108 (a.k.a., search engine). In this case the window of relevance can be set by the user via a GUI in the communication device 101, such as a User Equipment like a tablet, mobile phone, laptop, or a communications device embedded in a vehicle i, and be modified for each search query.
In this embodiment the indexing table/index contains all key words, modalities and time stamps according to the example below.
The algorithm here analyzes videos that have been or is currently being uploaded to the server. For example, if a video is currently being uploaded, the algorithm is run by the server or the UE and here looks and tries to recognize faces and text and optionally also for sound. A json-file (.json) including keywords, related times and modality is created.
In a first step, the communication device 101 submits a search query to the server.
In steps 2-3, the server run another part of the WoR algorithm on the query results
a) The query results consist of a ranked list of matching keywords, t_start, and modality
b) For each keyword in list
c) Optionally keywords which are included in overlapping WoRs may be boosted in rank.
In step 3, return updated ranked list of search results.
The embodiments described herein may enable detailed and precise search of audio visual content.
Referring to
retrieving from the data storage system 1206 the keyword and the at least one time stamp (step 1002),
define a time range (WoR) for the keyword (step 1004),
determining if there are other keywords in the data storage system 1206 which are associated with a time stamp within the time range (i.e., determine which other keywords in the data storage system 1206 are associated with a time stamp within the defined time range) (step 1006),
updating index 106 with, for each one of the keywords in the data storage system 1206, found other keywords with time stamps (matching keywords) within the time range (step 1008).
The method may also comprise updating the index and/or database with a rank value and modality of each one the other keywords with time stamps within the time range (i.e. the matching keywords).
Referring to
identifying one or more features present in the frames, wherein each one of the features are of at least one modality/media type (step 1102);
identifying, for each identified feature, one or more respective keywords associated with the identified feature (step 1104);
generating the at least one time stamp for each one of the identified keywords (step 1106); and
storing the keywords and their respective relevances and time stamps in the data storage system 1206 (step 1108).
The method may further include determining, for each identified keyword associated with an identified feature, a respective relevance of the keyword, wherein the respective relevance is dependent on a weight associated with the respective modality/media type of the identified feature.
The time range may be defined as starting a certain time before the time of the first time stamp and ending the certain time after the first time stamp. The time range, may alternatively, e.g. for images, start a certain time before the time of the first time stamp and end the certain time after a second time stamp associated with the keyword.
Apparatus, computer program and computer program products corresponding to the embodiment illustrated by the method is disclosed by the above disclosure. An embodiment of an apparatus, such as a server/server device/server host, may correspondingly be said to be a server device comprising a processor and memory, said memory containing instructions executable by said processor whereby said server device is operative to, for each one of a plurality of keywords in the memory, wherein each keyword is associated with a modality and at least one time stamp: retrieve the keyword and the at least one time stamp from the memory or the second memory; define a time range (WoR) for the keyword; determine if there are other keywords in the memory which are associated with a time stamp within the time range; and update an index and/or a database with, for each one of the keywords in the memory, found other keywords with time stamps (matching keywords) within the time range.
The memory should above not be seen as comprising just one memory, but should also be interpreted as also comprising embodiments where two or more memories are utilized, for example when the instructions are stored in one physical memory part and the keywords and time stamps etc are stored in another memory part in the server device.
While various embodiments of the present disclosure are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.
Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.
This application is a 35 U.S.C. § 371 National Phase Entry Application from PCT/SE2014/050027, filed Jan. 13, 2014, designating the United States, which claims priority to U.S. Patent Application No. 61/871,460, filed Aug. 29, 2013. The above identified applications are incorporated by reference.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/SE2014/050027 | 1/13/2014 | WO | 00 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2015/030645 | 3/5/2015 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
5678040 | Vasudevan et al. | Oct 1997 | A |
5832495 | Gustman | Nov 1998 | A |
5963940 | Liddy et al. | Oct 1999 | A |
6026388 | Liddy et al. | Feb 2000 | A |
6363380 | Dimitrova | Mar 2002 | B1 |
6366296 | Boreczky et al. | Apr 2002 | B1 |
6374260 | Hoffert | Apr 2002 | B1 |
6484156 | Gupta et al. | Nov 2002 | B1 |
6643643 | Lee et al. | Nov 2003 | B1 |
7131059 | Obrador | Oct 2006 | B2 |
7149359 | Omoigui | Dec 2006 | B1 |
7242810 | Chang | Jul 2007 | B2 |
7382933 | Dorai et al. | Jun 2008 | B2 |
7493312 | Liu et al. | Feb 2009 | B2 |
7627556 | Liu et al. | Dec 2009 | B2 |
7676495 | Qian | Mar 2010 | B2 |
7725453 | Chen et al. | May 2010 | B1 |
8060906 | Begeja et al. | Nov 2011 | B2 |
8065301 | Ginsburg et al. | Nov 2011 | B2 |
8166029 | Park et al. | Apr 2012 | B2 |
8438157 | Adams, Jr. | May 2013 | B2 |
8442994 | Chen et al. | May 2013 | B1 |
8566370 | Jin et al. | Oct 2013 | B2 |
8611422 | Yagnik et al. | Dec 2013 | B1 |
8914632 | Shankar et al. | Dec 2014 | B1 |
9292552 | Arngren | Mar 2016 | B2 |
9633015 | Arngren | Apr 2017 | B2 |
20010047379 | Jun et al. | Nov 2001 | A1 |
20020054083 | Boreczky et al. | May 2002 | A1 |
20020097983 | Wallace et al. | Jul 2002 | A1 |
20020161747 | Li et al. | Oct 2002 | A1 |
20030033347 | Bolle | Feb 2003 | A1 |
20030105589 | Liu et al. | Jun 2003 | A1 |
20030107592 | Li et al. | Jun 2003 | A1 |
20030108334 | Nevenka et al. | Jun 2003 | A1 |
20040025180 | Begeja et al. | Feb 2004 | A1 |
20040111432 | Adams, Jr. et al. | Jun 2004 | A1 |
20040162870 | Matsuzaki et al. | Aug 2004 | A1 |
20040215663 | Liu et al. | Oct 2004 | A1 |
20040220925 | Liu et al. | Nov 2004 | A1 |
20050102312 | Ohya et al. | May 2005 | A1 |
20050114357 | Chengalvarayan et al. | May 2005 | A1 |
20050222981 | Lawrence et al. | Oct 2005 | A1 |
20050265607 | Chang | Dec 2005 | A1 |
20060093190 | Cheng et al. | May 2006 | A1 |
20060122984 | Byers et al. | Jun 2006 | A1 |
20060149624 | Baluja et al. | Jul 2006 | A1 |
20060218191 | Gopalakrishnan | Sep 2006 | A1 |
20060282336 | Huang | Dec 2006 | A1 |
20070033515 | Sull et al. | Feb 2007 | A1 |
20070055695 | Dorai et al. | Mar 2007 | A1 |
20070056046 | Claudatos et al. | Mar 2007 | A1 |
20070067304 | Ives | Mar 2007 | A1 |
20070106646 | Stern et al. | May 2007 | A1 |
20070106660 | Stern et al. | May 2007 | A1 |
20070220025 | Hyman | Sep 2007 | A1 |
20070250810 | Tittizer et al. | Oct 2007 | A1 |
20080016101 | Ginsburg et al. | Jan 2008 | A1 |
20080016293 | Saika | Jan 2008 | A1 |
20080086688 | Chandratillake et al. | Apr 2008 | A1 |
20080109881 | Dasdan | May 2008 | A1 |
20080112690 | Shahraray et al. | May 2008 | A1 |
20080124055 | Shahraray et al. | May 2008 | A1 |
20080232775 | Ljolje | Sep 2008 | A1 |
20080310628 | Fujioka et al. | Dec 2008 | A1 |
20090006368 | Mei | Jan 2009 | A1 |
20090019034 | Franks et al. | Jan 2009 | A1 |
20090041356 | Barbieri et al. | Feb 2009 | A1 |
20090110296 | Sekiguchi et al. | Apr 2009 | A1 |
20090116645 | Jeong et al. | May 2009 | A1 |
20090138472 | MacLean | May 2009 | A1 |
20090154806 | Chang et al. | Jun 2009 | A1 |
20090210779 | Badoiu | Aug 2009 | A1 |
20090240674 | Wilde | Sep 2009 | A1 |
20090299725 | Grigsby et al. | Dec 2009 | A1 |
20090300351 | Lei et al. | Dec 2009 | A1 |
20100005121 | Benitez et al. | Jan 2010 | A1 |
20100070485 | Parsons et al. | Mar 2010 | A1 |
20100094630 | Yoakum | Apr 2010 | A1 |
20100138292 | Park et al. | Jun 2010 | A1 |
20100158470 | Tzoukermann et al. | Jun 2010 | A1 |
20100161580 | Chipman et al. | Jun 2010 | A1 |
20100211781 | Auradkar et al. | Aug 2010 | A1 |
20110010372 | Sahasrabudhe et al. | Jan 2011 | A1 |
20110040967 | Waller et al. | Feb 2011 | A1 |
20110047163 | Chechik et al. | Feb 2011 | A1 |
20110072012 | Ah-Pine | Mar 2011 | A1 |
20110154405 | Isaias | Jun 2011 | A1 |
20110208722 | Hannuksela | Aug 2011 | A1 |
20110249956 | Komai | Oct 2011 | A1 |
20110258188 | AbdAlmageed et al. | Oct 2011 | A1 |
20110299721 | He et al. | Dec 2011 | A1 |
20120023084 | Lalji et al. | Jan 2012 | A1 |
20120089580 | Yamashita et al. | Apr 2012 | A1 |
20120110080 | Panyam et al. | May 2012 | A1 |
20120124055 | Deubel et al. | May 2012 | A1 |
20120158713 | Jin et al. | Jun 2012 | A1 |
20130061035 | Hook et al. | Mar 2013 | A1 |
20130151534 | Luks et al. | Jun 2013 | A1 |
20130166587 | Berry | Jun 2013 | A1 |
20130219024 | Flack | Aug 2013 | A1 |
20130226930 | Arngren et al. | Aug 2013 | A1 |
20130282687 | Ah-Pine et al. | Oct 2013 | A1 |
20140032538 | Arngren et al. | Jan 2014 | A1 |
20140032562 | Arngren et al. | Jan 2014 | A1 |
20140108020 | Sharma et al. | Apr 2014 | A1 |
20140142958 | Sharma et al. | May 2014 | A1 |
20140222755 | Söderberg et al. | Aug 2014 | A1 |
20140229488 | Arngren et al. | Aug 2014 | A1 |
20150234824 | Arngren et al. | Aug 2015 | A1 |
20150312259 | Alpha et al. | Oct 2015 | A1 |
20160085860 | Wallenberg et al. | Mar 2016 | A1 |
20160210443 | Arngren et al. | Jul 2016 | A1 |
Number | Date | Country |
---|---|---|
1 024 437 | Aug 2000 | EP |
1 220 541 | Jul 2002 | EP |
1 480 100 | Nov 2004 | EP |
2216731 | Aug 2010 | EP |
2323046 | May 2011 | EP |
2 444 921 | Oct 2011 | EP |
2657884 | Oct 2013 | EP |
2001017163 | Mar 2001 | WO |
02084980 | Oct 2002 | WO |
2010150226 | Dec 2010 | WO |
2011104428 | Sep 2011 | WO |
2012001216 | Jan 2012 | WO |
2014185834 | Nov 2014 | WO |
2015030645 | Mar 2015 | WO |
2015030646 | Mar 2015 | WO |
Entry |
---|
Final Office Action for U.S. Appl. No. 13/559,258, dated Sep. 9, 2016, 39 pages. |
Extended European Search Report for European Patent Application No. 14839431.5, dated Jul. 22, 2016, 9 pages. |
Excerpt of Prosecution History of U.S. Appl. No. 13/559,258 Sep. 9, 2016-Dec. 2, 2016 45 pages. |
Excerpt of Prosecution History of U.S. Appl. No. 13/408,448 Nov. 1, 2016-May 9, 2014 45 pages. |
Supplementary European Search Report for European Patent Application No. 14893538 dated Mar. 6, 2017, 3 pages. |
International Search Report and Written Opinion issued in International Application No. PCT/SE2014/050655, dated Mar. 2, 2015, 15 pages. |
Saracoglu, Ahmet et. al.,“Content Based Copy Detection with Coarse, Audio-Visual Fingerprints”, 2009 Seventh International Workshop on Content-Based Multimedia Indexing, 2009, IEEE, 6 pages. |
Youtube and Shazam homepages retrieved on May 26, 2014, 2 pages. |
9th International Workshop on Content-Based Multimedia Indexing (CBMI 2011), homepage, 2 pages, http://www-vpu.eps.uam.es/cbmi2011/. |
Excerpt of Prosecution History of U.S. Appl. No. 13/764,219, Aug. 2014-May 2016 , 156 pages. |
Excerpt of Prosecution History of U.S. Appl. No. 14/184,207, Jun. 2016-May 2017, 124 pages. |
International Search Report and Written Opinion of the ISA issued for the corresponding International Patent Application No. PCT/SE2014/050027 dated Nov. 3, 2014, 10 pages. |
Non-Final Office Action issued in U.S. Appl. No. 14/914,849 dated Dec. 28, 2017, 27 pages. |
Lew, M.S., et al. “Content-Based Multimedia Information Retrieval: State of the Art and Challenges,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 2, No. 1, Feb. 2006, pp. 1-19. |
“MPEG-7 Overview,” International Organisation for Standardisation, ISO/IEC JTC1/SC29/WG11, Coding of Moving Pictures and Audio, Oct. 2004, pp. 1-95. |
Non-Final Office Action for U.S. Appl. No. 13/688,566, dated Sep. 10, 2014, 10 pages. |
Non-Final Office Action for U.S. Appl. No. 13/688,566, dated Mar. 12, 2015, 11 pages. |
Final Office Action for U.S. Appl. No. 13/688,566, dated Sep. 10, 2015, 16 pages. |
Examiner-Initiated Interview Summary for U.S. Appl. No. 13/688,566 dated Nov. 16, 2015, 1 page. |
Notice of Allowance for U.S. Appl. No. 13/688,566, dated Dec. 30, 2015, 19 pages. |
Non-Final Office Action for U.S. Appl. No. 13/408,448 dated Feb. 6, 2013, 15 pages. |
Non-Final Office Action for U.S. Appl. No. 13/408,448 dated Jun. 14, 2013, 19 pages. |
Final Office Action for U.S. Appl. No. 13/408,448 dated Dec. 5, 2013, 21 pages. |
Non-Final Office Action for U.S. Appl. No. 13/408,448 dated Apr. 7, 2014, 21 pages. |
Final Office Action for U.S. Appl. No. 13/408,448 dated Aug. 29, 2014, 28 pages. |
Advisory Action for U.S. Appl. No. 13/408,448 dated Nov. 7, 2014, 3 pages. |
Weisstein, Eric W. “Probability Axioms,” From MathWorld—A Wolfram Web Resource. http://mathworld.wolfram.com/ProbabilityAxioms.html, 1 page. |
Examiner's Answer to Appeal Brief for U.S. Appl. No. 13/408,448 dated May 18, 2015, 10 pages. |
Non-Final Office Action for U.S. Appl. No. 13/559,258, dated Aug. 15, 2013, 15 pages. |
Final Office Action for U.S. Appl. No. 13/559,258, dated Feb. 4, 2014, 20 pages. |
Advisory Action for U.S. Appl. No. 13/559,258, dated Apr. 21, 2014, 3 pages. |
Non-Final Office Action for U.S. Appl. No. 13/559,258, dated Dec. 31, 2014, 17 pages. |
Final Office Action for U.S. Appl. No. 13/559,258, dated Jun. 19, 2015, 23 pages. |
Advisory Action for U.S. Appl. No. 13/559,258, dated Aug. 27, 2015, 3 pages. |
Non-Final Office Action for U.S. Appl. No. 13/559,258, dated Jan. 20, 2016, 23 pages. |
International Search Report and Written Opinion of the ISA issued for International Patent Application No. PCT/SE2013/050536, dated Apr. 10, 2014, 17 pages. |
Lawto et al., “A Scalable Video Search Engine Based on Audio Content Indexing and Topic Segmentation,” Turin, Italy, Sep. 2011, http://hal.archivesouvertes.fr/docs/00/64/52/28/PDF/NEMSummit2011_finaI2.pdf [retrieved Apr. 2, 2014], 6 pages. |
Yang, W. et al., “Discriminative Tag Learning on YouTube Videos with Latent Sub-tags,” in proceeding of The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, Jun. 20-25, 2011, pp. 3217-3224. |
Garcia, E., “The Term Count Model: Demystifying Term Vector Calculations,” Oct. 27, 2006, 7 pages, http://web.archive.org/web/20121029050356/http://www.miislita.com/term-vector/term-vector-2.html. |
International Search Report and Written Opinion of the ISA issued for International Patent Application No. PCT/SE2014/050240, dated Dec. 8, 2014, 14 pages. |
Cisco Systems, Inc., “Cisco Visual Networking Index: Forecast and Methodology, 2012-2017,” Cisco White Paper, May 29, 2013, 15 pages. |
Cisco Systems, Inc., “Press Release: Cisco Global Cloud Index Forecasts Cloud Traffic to Grow Sixfold by 2016,” retrieved Jul. 8, 2014, dated Oct. 23, 2012, 5 pages. |
Cisco Systems, Inc., “Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update,” 2012-2017, Cisco White Paper, Feb. 6, 2013, 34 pages. |
www.google.com Print screen of “Obama” search results, Aug. 16, 2013, 3 pages. |
Non-Final Office Action issued in U.S. Appl. No. 14/914,849 dated Jul. 27, 2018, 25 pages. |
Non-Final Office Action issued in U.S. Appl. No. 14/891,259 dated Aug. 16, 2018, 54 pages. |
Non-Final Office Action issued in U.S. Appl. No. 15/312,848 dated Nov. 20, 2018, 32 pages. |
Final Office Action issued in U.S. Appl. No. 14/891,259 (3602-603US1) dated Mar. 6, 2019, 22 pages. |
Final Office Action issued in U.S. Appl. No. 15/312,848 (3602-1334US1) dated Apr. 2, 2019, 25 pages. |
Number | Date | Country | |
---|---|---|---|
20160217171 A1 | Jul 2016 | US |
Number | Date | Country | |
---|---|---|---|
61871460 | Aug 2013 | US |