Patent Grant
10205781
The present invention concerns linking traditional media to new interactive media, such as that provided over the Internet for example. In particular, the present invention concerns identifying a work (e.g., content or an advertisement delivered via print media, or via a radio or television broadcast) without the need to modify the work.
§ 1.2.1 Opportunities Arising from Linking Works Delivered Via Some Traditional Media Channel or Conduit to a More Interactive System
The rapid adoption of the Internet and associated World Wide Web has recently spurred interest in linking works, delivered via traditional media channels or conduits, to a more interactive system, such as the Internet for example. Basically, such linking can be used to (a) promote commerce, such as e-commerce, and/or (b) enhance interest in the work itself by facilitating audience interaction or participation. Commerce opportunities include, for example, facilitating the placement of direct orders for products, providing product coupons, providing further information related to a product, product placement, etc.
In the context of e-commerce, viewers could request discount vouchers or coupons for viewed products that are redeemable at the point of purchase. E-commerce applications also extend beyond advertisements. It is now common for television shows to include product placements. For example, an actor might drink a Coke rather than a Pepsi brand of soda, actors and actresses might wear designer-labeled clothing such as Calvin Klein, etc. Viewers may wish to purchase similar clothing but may not necessarily be able to identify the designer or the particular style directly from the show. However, with an interactive capability, viewers would be able to discover this and other information by going to an associated Web site. The link to this Web site can be automatically enabled using the invention described herein.
In the context of facilitating audience interaction or participation, there is much interest in the convergence of television and computers. Convergence encompasses a very wide range of capabilities. Although a significant effort is being directed to video-on-demand applications, in which there is a unique video stream for each user of the service, as well as to transmitting video signals over the Internet, there is also interest in enhancing the television viewing experience. To this end, there have been a number of experiments with interactive television in which viewers can participate in a live broadcast. There are a variety of ways in which viewers can participate. For example, during game shows, users can answer the questions and their scores can be tabulated. In recent reality-based programming such as the ABC television game show, “Big Brother”, viewers can vote on contestants who must leave the show, and be eliminated from the competition.
§ 1.2.2 Embedding Work Identifying Code or Signals Within Works
Known techniques of linking works delivered via traditional media channels to a more interactive system typically require some type of code, used to identify the work, to be inserted into the work before it is delivered via such traditional media channels. Some examples of such inserted code include (i) signals inserted into the vertical blanking interval (“VBI”) lines of a (e.g., NTSC) television signal, (ii) watermarks embedded into images, (iii) bar codes imposed on images, and (iv) tones embedded into music.
The common technical theme of these proposed implementations is the insertion of visible or invisible signals into the media that can be decoded by a computer. These signals can contain a variety of information. In its most direct form, the signal may directly encode the URL of the associated Web site. However, since the alphanumeric string has variable length and is not a particularly efficient coding, it is more common to encode a unique ID. The computer then accesses a database, which is usually proprietary, and matches the ID with the associated web address. This database can be considered a form of domain name server, similar to those already deployed for network addresses. However, in this case, the domain name server is proprietary and the addresses are unique ID's.
There are two principal advantages to encoding a proprietary identifier into content. First, as previously mentioned, it is a more efficient use of the available bandwidth and second, by directing all traffic to a single Web site that contains the database, a company can maintain control over the technology and gather useful statistics that may then be sold to advertisers and publishers.
As an example of inserting signals into the vertical blanking interval lines of a television signal, RespondTV of San Francisco, Calif. embeds identification information into the vertical blanking interval of the television signal. The VBI is part of the analog video broadcast that is not visible to television viewers. For digital television, it may be possible to encode the information in, for example, the motion picture experts group (“MPEG”) header. In the USA, the vertical blanking interval is currently used to transmit close-captioning information as well as other information, while in the UK, the VBI is used to transmit teletext information. Although the close captioning information is guaranteed to be transmitted into the home in America, unfortunately, other information is not. This is because ownership of the vertical blanking interval is disputed by content owners, broadcasters and local television operators.
As an example of embedding watermarks into images, Digimarc of Tualatin, Oreg. embeds watermarks in print media. Invisible watermarks are newer than VBI insertion, and have the advantage of being independent of the method of broadcast. Thus, once the information is embedded, it should remain readable whether the video is transmitted in NTSC, PAL or SECAM analog formats or newer digital formats. It should be more reliable than using the vertical blanking interval in television applications. Unfortunately, however, watermarks still require modification of the broadcast signal which is problematic for a number of economic, logistical, legal (permission to alter the content is needed) and quality control (the content may be degraded by the addition of a watermark) reasons.
As an example of imposing bar codes on images, print advertisers are currently testing a technology that allows an advertisement to be shown to a camera, scanner or bar code reader that is connected to a personal computer (“PC”). The captured image is then analyzed to determine an associated Web site that the PC's browser then accesses. For example, GoCode of Draper, Utah embeds small two-dimensional bar codes for print advertisements. The latter signal is read by inexpensive barcode readers that can be connected to a PC. AirClic of Blue Bell, Pa. provides a combination of barcode and wireless communication to enable wireless shopping through print media. A so-called “CueCat” reads bar codes printed in conjunction with advertisements and articles in Forbes magazine. Similar capabilities are being tested for television and audio media.
Machine-readable bar codes are one example of a visible signal. The advantage of this technology is that it is very mature. However, the fact that the signal is visible is often considered a disadvantage since it may detract from the aesthetic of the work delivered via a traditional media channel or conduit.
As an example of embedding tones into music, Digital Convergence of Dallas, Tex. proposes to embed identification codes into audible music tones broadcast with television signals.
All the foregoing techniques of inserting code into a work can be categorized as active techniques in that they must alter the existing signal, whether it is music, print, television or other media, such that an identification code is also present. There are several disadvantages that active systems share. First, there are aesthetic or fidelity issues associated with bar codes, audible tones and watermarks. More importantly, all media must be processed, before it is delivered to the end user, to contain these active signals. Even if a system is enthusiastically adopted, the logistics involved with inserting bar codes or watermarks into, say every printed advertisement, are formidable.
Further, even if the rate of adoption is very rapid, it nevertheless remains true that during the early deployment of the system, most works will not be tagged. Thus, consumers that are early-adopters will find that most media is not identified. At best, this is frustrating. At worst, the naive user may conclude that the system is not reliable or does not work at all. This erroneous conclusion might have a very adverse effect on the adoption rate.
Further, not only must there be modification to the production process, but modifications must also be made to the equipment in a user's home. Again, using the example of watermarking of print media, a PC must be fitted with a camera and watermark detection software must be installed. In the case of television, the detection of the identification signal is likely to occur at the set-top-box—this is the equipment provided by the local cable television or satellite broadcasting company. In many cases, this may require modifications to the hardware, which is likely to be prohibitively expensive. For example, the audible tone used by Digital Convergence to recognize television content, must be fed directly into a sound card in a PC. This requires a physical connection between the television and the PC, which may be expensive or at least inconvenient, and a sound card may have to be purchased.
§ 1.2.3 Unmet Needs
In view of the foregoing disadvantages of inserting an identification code into a work, thereby altering the existing signal, there is a need for techniques of identifying a work without the need of inserting an identification code into a work. Such an identification code can then be used to invoke a work-related action, such as work-related commerce methods and/or to increase audience interest by facilitating audience interaction and/or participation.
Some embodiments consistent with the present invention provide a computer-implemented method, apparatus, or computer-executable programs for linking a media work to an action. Such embodiments might (a) extract features from the media work, (b) determine an identification of the media work based on the features extracted using a sub-linear time search, such as an approximate nearest neighbor search for example, and (c) determine an action based on the identification of the media work determined. In some embodiments consistent with the present invention, the media work is an audio signal. The audio signal might be obtained from a broadcast, or an audio file format. In other embodiments consistent with the present invention, the media work is a video signal. The video signal might be obtained from a broadcast, or a video file format.
In some of the embodiments pertaining to audio files, the audio file might be an mp3 file or some other digital representation of an audio signal. The information might include a song title, an album title, and/or a performer name.
In some of the embodiments pertaining to video files, the video file might be an MPEG file or some other digital representation of a video signal. The video file might be a video work, and the information might include a title of the video work, a director of the video work, and names of performers in the video work.
The present invention may involve novel methods, apparatus and data structures for identifying works without the need of embedding signals therein. Once identified, such information can be used to determine a work-related action. The following description is presented to enable one skilled in the art to make and use the invention, and is provided in the context of particular embodiments and methods. Various modifications to the disclosed embodiments and methods will be apparent to those skilled in the art, and the general principles set forth below may be applied to other embodiments, methods and applications. Thus, the present invention is not intended to be limited to the embodiments and methods shown and the inventors regard their invention as the following disclosed methods, apparatus, data structures and any other patentable subject matter to the extent that they are patentable.
The present invention functions to identify a work without the need of inserting an identification code into a work. The present invention may do so by (i) extracting features from the work to define a feature vector, and (ii) comparing the feature vector to feature vectors associated with identified works. Alternatively, or in addition, the present invention may do so by (i) accepting extra-work information, such as the time of a query or of a rendering of the work, the geographic location at which the work is rendered, and the station that the audience member has selected, and (ii) use such extra-work information to lookup an identification of the work. In either case, an identification code may be used to identify the work.
The present invention may then function to use such an identification code to initiate a work-related action, such as for work-related commerce methods and/or to increase audience interest by facilitating audience interaction and/or participation.
As just introduced in § 4.1 above, the present invention may use intra-work information and/or extra-work information to identify a work. Once identified, such identification can be used to initiate an action, such as an action related to commerce, or facilitating audience participation or interaction. Exemplary embodiments of the present invention, in which work is recognized or identified based on intra-work information, are described in § 4.2.1. Then, exemplary embodiments of the present invention, in which work is recognized or identified based on extra-work information, are described in § 4.2.2.
§ 4.2.1 Embodiments in which Work is Recognized Based on Intra-Work Information, Such as a Feature Vector
Operations related to this embodiment are described in § 4.2.1.1 below. Then, various architectures which may be used to effect such operations are described in § 4.2.1.2.
§ 4.2.1.1 Operations and Exemplary Methods and Techniques for Effecting Such Operations
Further, work identifier-action information storage 130 may include a number of items or records 132. Each item or record 132 may associate a, preferably unique, work identifier 134 with associated information 136, such as an action for example. The work identifier-action information storage 130 may be generated by a database generation operation(s) 138 which may, for example, accept manual entries.
As can be appreciated from the foregoing, the work-information storage 110 records 112 and the work identification-action 130 records 132 can be combined into a single record. That is, there need not be two databases. A single database is also possible in which the work identifier, or a feature vector extracted from the work, serves as a key and the associated field contains work-related information, such as a URL for example.
The feature extraction operation(s) 140 can accept a work, such as that being rendered by a user, at a second time (WORK@t2), and extract features from that work. The extracted features may be used to define a so-called feature vector.
The extracted features, e.g., as a feature vector, can be used by a feature (vector) lookup operation(s) 150 to search for a matching feature vector 114. If a match, or a match within a predetermined threshold is determined, then the associated work identifier 116 is read.
The read work identifier can then be used by a work-associated information lookup operation(s) 160 to retrieve associated information, such as an action, 136 associated with the work identifier. Such information 136 can then be passed to action initiation operation(s) 170 which can perform some action based on the associated information 136.
§ 4.2.1.1.1 Exemplary Techniques for Feature Extraction
When the user initiates a request, the specific television or radio broadcast or printed commercial, each of which is referred to as a work, is first passed to the feature extraction operation. The work may be an image, an audio file or some portion of an audio signal or may be one or more frames or fields of a video signal, or a multimedia signal. The purpose of the feature extraction operation is to derive a compact representation of the work that can subsequently be used for the purpose of recognition. In the case of images and video, this feature vector might be a pseudo-random sample of pixels from the frame or a low-resolution copy of the frame or the average intensities of n.times.n blocks of pixels. It might also be a frequency-based decomposition of the signal, such as produced by the Fourier, wavelet and or discrete cosine transforms. It might involve principal component analysis. It might also be a combination of these. For television and audio signals, recognition might also rely on a temporal sequence of feature vectors. The recognition literature contains many different representations. For block-based methods, blocks may be accessed at pseudo-random locations in each frame or might have a specific structure. For audio, common feature vectors are based on Fourier frequency decompositions, but other representations are possible. See, e.g., R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis (Wiley-Interscience, New York, 1973). See also K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd Ed. (Academic Press, New York, 1990). (These references are incorporated herein by reference.)
As previously stated, one object of the vector extraction stage is to obtain a more concise representation of the frame. For example, each video frame is initially composed of 480.times.720 pixels which is equivalent to 345,600 pixels or 691,200 bytes. In comparison, an exemplary feature vector might only consist of 1 Kbyte of data.
A second purpose of the feature extraction process is to acquire a representation that is robust or invariant to possible noise or distortions that a signal might experience. For example, frames of a television broadcast may experience a small amount of jitter, i.e., horizontal and or vertical translation, or may undergo lossy compression such as by MPEG-2. It is advantageous that these and other processes do not adversely affect the extracted vectors. For still images there has been considerable work on determining image properties that are invariant to affine and other geometric distortions. For example, the use of Radon and Fourier-Mellin transforms have been proposed for robustness against rotation, scale and translation, since these transforms are either invariant or bare a simple relation to the geometric distortions. See, e.g., C. Lin, M. Wu, Y. M. Lui, J. A. Bloom, M. L. Miller, I. J. Cox, “Rotation, Scale, and Translation Resilient Public Watermarking for Images,” IEEE Transactions on Image Processing (2001). See also, U.S. Pat. Nos. 5,436,653, 5,504,518, 5,582,246, 5,612,729, and 5,621,454. (Each of these references is incorporated herein by reference.)
§ 4.2.1.1.2 Exemplary Techniques for Database Generation and Maintenance
A number of possibilities exist for generating and maintaining work identification (WID) and identification-action translation (WIDAT) databases. However, in all cases, works of interest are processed to extract a representative feature vector and this feature vector is assigned a unique identifier. This unique identifier is then entered into the work identification (WID) database 110 as well as into the WIDAT database 130 together with all the necessary associated data. This process is referred to as tagging. For example, in the case of an advertisement, the WIDAT database 130 might include the manufacturer (Ford), the product name (Taurus), a product category (automotive) and the URL associated with the Ford Taurus car together with the instruction to translate the query into the associated URL.
The determination of all works of interest and subsequent feature vector extraction and tagging depends on whether content owners are actively collaborating with the entity responsible for creating and maintaining the database. If there is no collaboration, then the database entity must collect all works of interest and process and tag them. While this is a significant effort, it is not overwhelming and is certainly commercially feasible. For example, competitive market research firms routinely tabulate all advertisements appearing in a very wide variety of print media. Newspapers and magazines can be scanned in and software algorithms can be applied to the images to identify likely advertisements. These possible advertisements can then be compared with advertisements already in the WID database 110. If there is a match, nothing further need be done. If there is not a match, the image can be sent to a human to determine if the page does indeed contain an advertisement. If so, the operator can instruct the computer to extract the representative feature vector and assign it a unique identifier. Then, the operator can insert this information into the content identification database and as well as update the corresponding WIDAT database 130 with all the necessary associated data. This is continually performed as new magazines and papers include new advertisements to maintain the databases. This is a cost to the database entity. Television and radio broadcasts can also be monitored and, in fact, broadcast monitoring is currently performed by companies such as Nielsen Media research and Competitive Media Reporting. Television and radio broadcasts differ from print media in the real-time nature of the signals and the consequent desire for real-time recognition.
In many cases, advertisers, publishers and broadcasters may wish to collaborate with the database provider. In this case, feature extraction and annotation and/or extra-work information may be performed by the advertiser, advertisement agency, network and/or broadcaster and this information sent to the database provider to update the database. Clearly, this arrangement is preferable from the database provider's perspective. However, it is not essential.
§ 4.2.1.1.3. Exemplary Techniques for Matching Extracted Features with Database Entries
The extracted feature vector is then passed to a recognition (e.g., feature look-up) operation, during which, the vector is compared to entries of known vectors 114 in a content identification (WID) database 110. It is important to realize that the matching of extracted and known vectors is not equivalent to looking up a word in an electronic dictionary. Since the extracted vectors contain noise or distortions, binary search might not be possible. Instead, a statistical comparison is often made between an extracted vector and each stored vector. Common statistical measures include linear correlation and related measures such as correlation coefficient, but other methods can also be used including mutual information, Euclidean distance and Lp-norms. These measures provide a statistical measure of the confidence of the match. A threshold can be established, usually based on the required false positive and false negative rates, such that if the correlation output exceeds this threshold, then the extracted and known vectors are said to match. See, e.g., R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis (Wiley-Interscience, New York, 1973). See also, U.S. Pat. No. 3,919,474 by W. D. Moon, R. J. Weiner, R. A. Hansen and R. N. Linde, entitled “Broadcast Signal Identification System”. (Each of these references is incorporated herein by reference.)
If binary search was possible, then a database containing N vectors would require at most log(N) comparisons. Unfortunately, binary search is not possible when taking a noisy signal and trying to find the most similar reference signal. This problem is one of nearest neighbor search in a (high-dimensional) feature space. In previous work, it was not uncommon to perform a linear search of all N entries, perhaps halting the search when the first match is found. On average, this will require N/2 comparisons. If N is large, this search can be computationally very expensive.
Other forms of matching include those based on clustering, kd-trees, vantage point trees and excluded middle vantage point forests are possible and will be discussed in more detail later. See, e.g., P. N. Yianilos “Excluded Middle Vantage Point Forests for nearest Neighbor Search”, Presented at the Sixth DIMACS Implementation Challenge: Near Neighbor Searches workshop, (Jan. 15, 1999). See also, P. N. Yianilos, “Locally lifting the curse of Dimensionality for nearest Neighbor Search” SODA 2000: 361-370. (Each of these references is incorporated herein by reference.) Thus, for example, a sub-linear search time can be achieved. Unlike the kd-tree method which finds the nearest neighbor with certainty, randomized constructions, like the one described in P. N. Yianilos, “Locally lifting the curse of Dimensionality for nearest Neighbor Search” SODA 2000: 361-370, that succeed with some specified probability may be used. One example of a sub-linear time search is an approximate nearest neighbor search. A nearest neighbor search always finds the closest point to the query. An approximate nearest neighbor search does not always find the closest point to the query. For example, it might do so with some probability, or it might provide any point within some small distance of the closest point.
If the extracted vector “matches” a known vector in the content identification database, then the work has been identified. Of course, there is the risk that the match is incorrect. This type of error is known as a false positive. The false positive rate can be reduced to any desired value, but at the expense of the false negative rate. A false negative occurs when the vector extracted from a work is not matched to the database even though the work is present in the database. There are several reasons why a work's feature vector may fail to match a feature vector database entry. First, the recognition system may not be capable of 100% accuracy. Second, the extracted vector will often contain noise as a result of the transmission process. This noise may alter the values of a feature vector to the extent that a match is no longer possible.
Finally, there is the case where the observed work is not present in the database. In this case, the work can be sent to an operator for identification and insertion in the database.
§ 4.2.1.1.4 Exemplary Work Based Actions
Assuming that the work is correctly identified, then the identifier can be used to retrieve associated information from the second work identification-action translation (WIDAT) database 130 that contains information 136 associated with the particular work 134. This information may simply be a corresponding URL address, in which case, the action can be considered to be a form of network address translation. However, in general, any information about the work could be stored therein, together with possible actions to be taken such as initiating an e-commerce transaction. After looking up the work identifier 134 in the WIDAT database 130, an action is performed on behalf of the user, examples of which has been previously described.
In addition to using the system to allow audience members of a work to connect to associated sites on the Internet, a number of other uses are possible. First, the work identification database 130 allows competitive market research data to be collected (e.g., the action may include logging an event). For example, it is possible to determine how many commercials the Coca Cola Company in the Chicago market aired in the month of June. This information is valuable to competitors such as Pepsi. Thus, any company that developed a system as described above could also expect to generate revenue from competitive market research data that it gathers.
Advertisers often wish to ensure that they receive the advertising time that was purchased. To do so, they often hire commercial verification services to verify that the advertisement or commercial did indeed run at the expected time. To do so, currently deployed systems by Nielsen and CMR embedded active signals in the advertisement prior to the broadcast. These signals are then detected by remote monitoring facilities that then report back to a central system which commercials were positively identified. See for example U.S. Pat. No. 5,629,739 by R. A. Dougherty entitled “Apparatus and method for injecting an ancillary signal into a low energy density portion of a color television frequency spectrum”, U.S. Pat. No. 4,025,851 by D. E. Haselwood and C. M. Solar entitled “Automatic monitor for programs broadcast”, U.S. Pat. No. 5,243,423 by J. P. DeJean, D. Lu and R. Weissman, entitled “Spread spectrum digital data transmission over TV video”, and U.S. Pat. No. 5,450,122 by L. D. Keene entitled “In-station television program encoding and monitoring system and method”. (Each of these patents is incorporated herein by reference.) Active systems are usually preferred for advertisement verification because the required recognition accuracy is difficult to achieve with passive systems. The passive monitoring system described herein supports commercial verification.
§ 4.2.1.2 Exemplary Architectures
Three alternative architectural embodiments in which the first technique may be employed are now described with reference to
As described above, the two databases might be integrated into a single database. However, conceptually, they are described here as separate.
An example illustrating operations that can occur in the first embodiment of
An important issue then becomes recognition rate. While this may be problematic, all the images are two-dimensional—three-dimensional object recognition is not required. Of course, since a low cost camera captures the printed advertisement, there may be a number of geometric distortions that might be introduced together with noise. Nevertheless, the application is sufficiently constrained that adequate recognition rates should be achievable with current state-of-the-art computer vision algorithms. See, e.g., P. N. Yianilos “Excluded Middle Vantage Point Forests for nearest Neighbor Search”, Presented at the Sixth DIMACS Implementation Challenge: Near Neighbor Searches workshop, Jan. 15, 1999. See also, P. N. Yianilos “Locally lifting the curse of Dimensionality for nearest Neighbor Search” SODA 2000: 361-370. (Each of these references is incorporated herein by reference.) Thus, for example, a sub-linear search time can be achieved. Unlike the kd-tree method which finds the nearest neighbor with certainty, randomized constructions, like the one described in P. N. Yianilos, “Locally lifting the curse of Dimensionality for nearest Neighbor Search” SODA 2000: 361-370, that succeed with some specified probability may be used. One example of a sub-linear time search is an approximate nearest neighbor search. Estimates of the size of the WIDAT database 130a depend on what associated information (recall fields 136) is stored. If, for example, only a URL address is needed, about 20 characters can typically represent most URLs. Thus, the size of the WIDAT database 130a would be less than 1 Mbyte.
The configuration just described with reference to
As new works are created and made publicly available, the databases residing on a user's local computer become obsolete. Just as the database provider 240 must continually update the databases in order to remain current, there is also a need to update local databases on devices at audience member premises. This update process can be performed over the Internet 230 in a manner very similar to how software is currently upgraded. It is not necessary to download an entirely new database although this is an option. Rather, only the changes need to be transmitted. During this update process, the user's computer 210 might also transmit information to a central monitoring center 240 informing it of which advertisements the computer user has queried. This type of information is valuable to both advertisers and publishers. Of course, care must be taken to ensure the privacy of individual users of the system. However, it is not necessary to know the identity of individual users for the system to work.
The second embodiment is most similar to active systems in which an embedded signal is extracted and decoded and the identifier is used to interrogate a central database. Consequently it has many of the advantages of such systems, while avoiding the need to insert signals into all works. One such advantage, is that the database provider receives real-time information relating to users' access patterns.
The WIDAT database 130b might physically reside at more than one location. In such a case, some requests will go to one site, and other requests will go to another. In this way, overloading of a single site by too many users can be avoided. Other load balancing techniques are also applicable.
Since the storage and searching of the work-identifier (WID) database require the most computation and storage, it may be more economical to perform these actions remotely. Thus, for example, if a user is playing an MP3 music file and wants to go to a corresponding website, the MP3 file is passed to an operation that determines one or more feature vectors. In the third embodiment, instead of performing the matching locally 410, the one or more vectors are transmitted to a central site 440 at which is stored the WID and WIDAT databases 110c and 130c together with sufficiently powerful computers to resolve this request and those of other computer users. This configuration is illustrated in
After successfully matching the feature vector, the central site 440 can provide the user with information directly, or can direct the user to another Web site that contains the information the user wants. In cases where the recognition is ambiguous, the central site 440 might return information identifying one of several possible matches and allow the user to select the intended one.
The third embodiment is particularly attractive if the cost of extracting the feature vector is small. In this case, it becomes economical to have feature vector extraction 140c in digital set-top-boxes and in video recorders 410. The latter may be especially useful for the new generation of consumer digital video recorders such as those manufactured by TIVO and Replay TV. These devices already have access to the Internet via a phone line. Thus, when someone watching a recorded movie from television reacts to an advertisement, the video recorder would extract one or more feature vectors and transmit them to a central site 440. This site 440 would determine if a match existed between the query vector and the database of pre-stored vectors 110c. If a match is found, the central server 440 would transmit the associated information, which might include a Web site address or an 800 number for more traditional ordering, back to the audience user device 410. Of course, a consumer device 410 such as a digital video recorder might also store personal information of the owner to facilitate online e-commerce. Such a device 410 could store the owner's name, address, and credit card information and automatically transmit them to an on-line store to complete a purchase. Very little user interaction other than to authorize the purchase might be needed. This type of purchasing may be very convenient to consumers.
Another advantage of the third embodiment is that it obviates the need to update local databases while, at the same time, the centrally maintained databases can be kept current with very frequent updating.
§ 4.2.2 Embodiments in which Work is Recognized Based on Extra-Work Information
Operations related to this embodiment are described in § 4.2.2.1 below. Then, various architectures which may be used to effect such operations are described in § 4.2.2.2.
If the cost of extracting a feature vector is too large, then the cost of deploying any of the embodiments described in § 4.2.1 above may be prohibitive. This is particularly likely in very cost sensitive consumer products, including set-top-boxes and next generation digital VCR's. Acknowledging this fact, a different technique, one that is particularly well suited for broadcasted media such as television and radio as well as to content published in magazines and newspapers, is now described. This technique relies on the fact that a work need not be identified by a feature vector extracted from the work (which is an example of “intra-work information”), but can also be identified by when and where it is published or broadcast (which are examples of “extra-work information”)
An example serves to illustrate this point. Consider the scenario in which a viewer sees a television commercial and responds to it. The embodiments described in § 4.2.1 above required the user device (e.g., a computer or set-top-box) 210/310/410 to extract a feature vector. Such an extracted vector was attempted to be matched to another feature vector(s), either locally, or at a remote site. In the embodiments using a remote site, if the central site is monitoring all television broadcasts, then the user's query does not need to include the feature vector. Instead, the query simply needs to identify the time, geographic location and the station that the viewer is watching. A central site can then determine which advertisement was airing at that moment and, once again, return the associated information. The same is true for radio broadcasts. Moreover, magazines and newspapers can also be handled in this manner. Here the query might include the name of the magazine, the month of publication and the page number.
§ 4.2.2.1 Operations and Exemplary Methods and Techniques for Effecting Such Operations
Further, work identifier-action information (WIDAT) storage 530 may include a number of items or records 532. Each item or record 532 may associate a, preferably unique, work identifier 534 with associated information 536, such as an action for example. The work identifier-action (WIDAT) information storage 530 may be generated by a database generation operation(s) 538 which may, for example, accept manual entries.
As can be appreciated from the foregoing, the query work-information (QWID) storage 510 records 512 and the work identification-action (WIDAT) storage 530 records 532 can be combined into a single record.
The extra-work information aggregation (e.g., query generation) operation(s) 540 can accept a information related to a work, such as the time of a user request or of a rendering of the work, the geographic location at which the work is rendered, and the station that the audience member has selected, and generate a query from such extra-work information.
The query including the extra-work information can be used by a lookup operation(s) 550 to search for a “matching” set of information 514. If a match, or a match within a predetermined threshold is determined, then the associated work identifier 516 is read.
The read work identifier can then be used by a work-associated information lookup operation(s) 560 to retrieve associated information, such as an action, 536 associated with the work identifier. Such information 536 can then be passed to action initiation operation(s) 570 which can perform some action based on the associated information 536.
If the extra-work information of a work is known (in advance), generating the query work identifier (QWID) information 510 is straight-forward. If this were always the case, an intra-work information-based recognition operation would not be needed. However, very often this is not the case. For example, local television broadcasts typically have discretion to insert local advertising, as well as national advertising. Thus, it often is not possible to know in advance when, on what station, and where a particular advertisement will play.
In such instances, a real-time (e.g., centralized) monitoring facility 580 may be used to (i) extract feature vectors from a work, (ii) determine a work identifier 116 from the extracted features, and (iii) communicate one or more messages 590 in which extra-work information (e.g., time, channel, geographic market) 592 is associated with a work identifier 594, to operation(s) 520 for generating query work identification (QWID) information 510.
§ 4.2.2.1.1 Exemplary Extra-Work Information
In the context of national broadcasts, geographic information may be needed to distinguish between, for example, the ABC television broadcast in Los Angeles and that in New York. While both locations broadcast ABC's programming, this programming airs at different times on the East and West coasts of America. More importantly, the local network affiliates that air ABC's shows have discretion to sell local advertising as well as a responsibility to broadcast the national commercials that ABC sells. In short, the works broadcast by ABC in Los Angeles can be different from that in other geographic locations. Geographic information is therefore useful to distinguish between the different television markets. In some circumstances, geographic information may not be necessary, especially in parts of the world with highly regulated and centralized broadcasting in which there are not regional differences.
§ 4.2.2.1.2 Exemplary Techniques for Generating Databases
As introduced above, although it appears that this architecture does not require a recognition facility, such a facility may be needed. The feature extraction operation(s) 140d, as well as the work identification operation(s) 150d and other databases 110d, may be moved to one or more remote sites 580.
Although TV Guide and other companies provide detailed information regarding what will be broadcast when, these scheduling guides do not have any information regarding what advertisements will air when. In many cases, this information is unknown until a day or so before the broadcast. Even then, the time slots that a broadcaster sells to an advertiser only provide a time range, e.g. 12 pm to 3 pm. Thus it is unlikely that all commercials and aired programming can be determined from TV schedules and other sources prior to transmission. Further, occasionally programming schedules are altered unexpectedly due to live broadcasts that overrun their time slots. This is common in sports events and awards shows. Another example of interrupts to scheduled programming occurs when a particularly important news event occurs.
During transmission, it may therefore be necessary for a central site 580 to determine what work is being broadcast and to update its and/or other's database 520 accordingly based on the work identified 594 and relevant extra-work information 592. There are a variety of ways that this can be accomplished.
First, it may be economically feasible to manually monitor all television stations that are of interest, and manually update the database with information regarding the work being monitored. In fact, Nielsen used such procedures in the early 1960's for the company to tabulate competitive market data. More than one person can be employed to watch the same channel in order to reduce the error rate. It should be noted that the recent ruling by the FCC that satellite broadcasters such as DirecTV, DishTV and EchoStar can carry local stations significantly reduces the cost of monitoring many geographic markets. Currently, DirecTV, for example, carries the four main local stations in each of the 35 largest markets. Thus, these 4.times.35=140 channels can all be monitored from a single site 580. This site would be provided with satellite receivers to obtain the television channels.
Unfortunately, however, humans are error prone and the monitoring of many different stations from many different geographic locations can be expensive. In order to automate the recognition process, a central site 580 could employ a computer-based system to perform automatic recognition. Because the recognition is centralized, only one or a few sites are needed. This is in comparison with the first architecture we described in which a complete recognition system was required in every user's home or premise. This centralization makes it more economic to employ more expensive computers, perhaps even special purpose hardware, and more sophisticated software algorithms. When video frames or clips cannot be identified or are considered ambiguous, this video can be quickly passed to human viewers to identify. Further, it should be possible for the automated recognition system to use additional information such as television schedules, time of day, etc in order to improve its recognition rate.
§ 4.2.2.1.2 Exemplary Techniques for Generating Queries Based on Extra-Work Information
At the audience member (user) premises, all that is needed is for the device to send a query to a database-server with information that includes extra-work information, such as geographic location, time and channel. Usually, this extra-work information would be transmitted in real-time, while the work (e.g., an advertisement) is being broadcast. However, this is not necessary. If the television does not have access to the Internet, and most TV's do not yet, then an audience member (user) may simply remember or record which channel he or she was viewing at what time. In fact, the user device could store this information for later retrieval by the user. At a convenient later time, the user might access the Internet using a home PC. At this time, he or she can query the database by entering this extra-work information (e.g., together with geographic information) into an application program or a web browser plug-in.
Another possibility is allowing an audience member (user), at the time he or she is consuming (e.g., viewing, reading, listening to, etc.) the work, to enter query information into a handheld personal digital assistant (“PDA”) such as a Palm Pilot, so as not to forget it. This information can then be manually transferred to a device connected to a network, or the information can be transferred automatically using, for example, infrared communications or via a physical link such as a cradle. Recently, PDAs also have some wireless networking capabilities built in, and thus might support direct access to the information desired. Further, software is available that allows a Palm Pilot or other PDA to function as a TV remote control device. As such, the PDA already knows the time of day and channel being viewed. It also probably knows the location of the audience member, since most PDA users include their own name and address in the PDA's phonebook and identify it as their own. Thus, with one or a few clicks, an audience member PDA user could bookmark the television content he or she is viewing. If the PDA is networked, then the PDA can, itself, retrieve the associated information immediately. Otherwise, the PDA can transfer this bookmarked data to a networked device, which can then provide access to the central database.
§ 4.2.2.2 Exemplary Architectures
These embodiments have many advantages for television and radio broadcasters who desire to provide Internet links or other action. First, the audience member (user) equipment, whether it is a computer, set-top-box, television, radio, remote control, personal digital assistant (pda), cell phone or other device, does not need to perform any processing of the received signal. As such, there is almost no cost involved to equipment manufacturers.
These last embodiments have some similarity with services such as those provided by the companies Real Names of Redwood City, Calif., America Online (“AOL”) and especially iTag from Xenote. The popular press has reported on the difficulties associated with assigning domain names. The simplest of these problems is that almost all the one-word names in the “.com” category have been used. Consequently, domain names can often be difficult to remember. To alleviate this problem, RealNames and AOL provide alternative, proprietary name spaces (AOL calls these keywords). For a fee, a company may register a name with these companies. Thus, rather than type the URL http://www.bell-labs.com, the simple keyword “bell” might be sufficient to access the same Web site. These capabilities are convenient to users. However, these systems are very different from the fourth and fifth embodiments described. First, and foremost, these systems are not designed to identify content. Rather, they are simply alternative network address translation systems based on easily remembered mnemonics which are sold to interested companies. As such, the user is still expected to type in an address, but this address is easier to remember than the equivalent URL. In contrast, while a user may manually enter the information describing the work, the preferred embodiment is for the computer, set-top-box or other device to automatically generate this information. Further, the mapping of keywords to network addresses is an arbitrary mapping maintained by AOL or Real Names. For example, the keyword “bell” might just as reasonably point to the Web site for Philadelphia's Liberty Bell as to Lucent's Bell Labs. In contrast, the query used in the fourth and fifth embodiments is designed to contain all the necessary data to identify the work, e.g. the time, place and television channel during which the work was broadcast. There is nothing arbitrary about this mapping. It should also be pointed out that the proposed system is dynamic—the same work, e.g. a commercial, potentially has an infinite number of addresses depending on when and where it is broadcast. If an advertisement airs 100,000 unique times, then there are 100,000 different queries that uniquely identify it. Moreover, the exemplary query includes naturally occurring information such as time, place, channel or page number. This is not the case for AOL or RealNames, which typically assigns one or more static keywords to the address of a Web site.
Xenote's iTag system is designed to identify radio broadcasts and uses a query similar to that which may be used in the fourth and fifth embodiments, i.e. time and station information. However, the work identification information is not dynamically constructed but is instead based on detailed program scheduling that radio stations must provide it. As such, it suffers from potential errors in scheduling and requires the detailed cooperation of broadcasters. While the fourth and fifth embodiments might choose to use program scheduling information and other ancillary information to aid in the recognition process, they do not exclusively rely on this. The concept of resolving a site name by recognizing the content is absent from the above systems.
§ 4.2.3 Exemplary Apparatus for Audience Member (User) Premise Device
While personal computers may be the primary computational device at a user's location, it is not essential to use a PC. This is especially true of the embodiments depicted in
As previously noted, personal digital assistants (PDAs) can be used to record the identification information. This information can then be transferred to a device with a network communication such as a PC. However, increasingly, PDAs will already have wireless network communication capabilities built-in, as with the Palm VII PDA. These devices will allow immediate communication with the query resolution center and all information will be downloaded to them or they can participate in facilitating an e-commerce transaction. Similarly, wireless telephones are increasingly offering web-enabled capabilities. Consequently, wireless phones could be programmed to act as a user interface.
New devices can also be envisaged, including a universal remote control for home entertainment systems with a LCD or other graphical display and a network connection. This connection may be wireless or the remote control might have a phone jack that allows it to be plugged directly into an existing phone line. As home networks begin to be deployed, such devices can be expected to communicate via an inexpensive interface to the home network and from there to access the Internet.
In many homes, it is not uncommon for a computer and television to be used simultaneously, perhaps in the same room. A person watching television could install a web browser plug-in or applet that would ask the user to identify his location and the station being watched. Then, periodically, every 20 seconds for example, the plug-in would update a list of web addresses that are relevant to the television programs being watched, including the commercials. The audience member would then simply click on the web address of interest to obtain further information. This has the advantage that the viewer does not have to guess the relevant address associated with a commercial and, in fact, can be directed to a more specialized address, such as www.fordvehicles.com/ibv/tausrus2kflash/flash.html, rather than the generic www.ford.com site. Of course, this applet or plug-in could also provide the database entity with information regarding what is being accessed from where and at what time. This information, as noted earlier, is valuable to advertisers and broadcasters. For PC's that have infra-red communication capabilities, it is straightforward to either control the home entertainment center from the PC or for the PC to decode the signals from a conventional remote control. Thus, as a user changes channels, the PC is able to automatically track the channel changes.
Recording devices such as analog VCR's and newer digital recording devices can also be exploited in the embodiments depicted in
Digital set-top-boxes are also expected to exploit the capabilities described herein. In particular, such devices will have two-way communication capabilities and may even include cable modem capabilities of course, the two-way communication need not be over a television cable. For example, satellite set-top-boxes provide up-link communications via a telephone connection. Clearly, such devices provide a convenient location to enable the services described herein. Moreover, such services can be provided as part of the OpenCable and DOCSIS (data over cable service interface specification) initiatives.
§ 4.2.4 Information Retrieval Using Features Extracted from Audio and/or Video Works
Some embodiments consistent with the present invention provide a computer-implemented method, apparatus, or computer-executable program for providing information about an audio file or (a video file) played on a device. Such embodiments might (a) extract features from the audio (or video) file, (b) communicate the features to a database, and (c) receive the information about the audio (or video) file from the database. In some embodiments consistent with the present invention, the act of extracting the features is performed by a microprocessor of the device, and/or a digital signal processor of the device. The received information might be rendered on an output (e.g., a monitor, a speaker, etc.) of the device. The received information might be stored (e.g., persistently) locally on the device. The information might be stored on a disk, or non-volatile memory.
In some of the embodiments pertaining to audio files, the audio file might be an mp3 file or some other digital representation of an audio signal. The information might include a song title, an album title, and/or a performer name.
In some of the embodiments pertaining to video files, the video file might be an MPEG file or some other digital representation of a video signal. The video file might be a video work, and the information might include a title of the video work, a director of the video work, and names of performers in the video work.
An example illustrating operations of an exemplary embodiment of the present invention, that uses intra-work information to identify the work, is provided in § 4.3.1. Then, an example illustrating operations of an exemplary embodiment of the present invention, that uses extra-work information to identify the work, is provided in § 4.3.2.
§ 4.3.1 Operational Example where Intra-Work Information is Used to Identify the Work
A generic system for monitoring television commercials is now described. Obviously, the basic ideas extend beyond this specific application.
The process of recognition usually begins by recognizing the start of a commercial. This can be accomplished by looking for black video frames before and after a commercial. If a number of black frames are detected and subsequently a similar number are detected 30 seconds later, then there is a good chance that a commercial has aired and that others will follow. It is also well known than the average sound volume during commercials is higher than that for television shows and this too can be used as an indicator of a commercial. Other methods can also be used. The need to recognize the beginning of a commercial is not essential. However, without this stage, all television programming must be assumed to be commercials. As such, all video frames must be analyzed. The advantage of determining the presence of a commercial is that less video content must be processed. Since the percentage of advertising time is relatively small, this can lead to considerable savings. For example, commercials can be buffered and then subsequently processed while the television show is being broadcast. This reduces the real-time requirements of a system at the expense of buffering, which requires memory or disk space. Of course, for the applications envisioned herein, a real-time response to a user requires real-time processing.
Once it is determined that an advertisement is being broadcast, it is necessary to analyze the video frames. Typically, a compact representation of each frame is extracted. This vector might be a pseudo-random sample of pixels from the frame or a low-resolution copy of the frame or the average intensities of n.times.n blocks of pixels. It might also be a frequency-based decomposition of the signal, such as produced by the Fourier, Fourier-Mellin, wavelet and or discrete cosine transforms. It might involve principal component analysis or any combination thereof. The recognition literature contains many different representations. For block-based methods, the n.times.n blocks may be located at pseudo-random locations in each frame or might have a specific structure, e.g. a complete tiling of the frame. The feature vector might then be composed of the pixels in each block or some property of each block, e.g. the average intensity or a Fourier or other decomposition of the block. The object of the vector extraction stage is to obtain a more concise representation of the frame. Each frame is initially composed of 480.times.720 pixels which is equivalent to 345,600 bytes, assuming one byte per pixel. In comparison, the feature vector might only consist of 1 Kbyte of data. For example, if each frame is completely tiled with 16.times.16 blocks, then the number of blocks per frame is 345,600/256=1350. If the average intensity of each block constitutes the feature vector, then the feature vector consists of 1350 bytes, assuming 8-bit precision for the average intensity values. Alternatively, 100 16.times.16 blocks can be pseudo-randomly located on each frame of the video. For each of these 100 blocks, the first 10 DCT coefficients can be determined. The feature vector then consists of the 100.times.10=1000 DCT coefficients. Many other variations are also possible. In many media applications, the content possesses strong temporal and spatial correlations. If necessary, these correlations can be eliminated or substantially reduced by pre-processing the content with a whitening filter.
A second purpose of the feature extraction process is to acquire a representation that is robust or invariant to possible noise or distortions that a signal might experience. For example, frames of a television broadcast may experience a small amount of jitter, i.e. horizontal and or vertical translation, or may undergo lossy compression such as MPEG-2. It is advantageous, though not essential, that these and other processes do not adversely affect the extracted vectors.
Each frame's feature vector is then compared with a database of known feature vectors. These known vectors have previously been entered into a content recognition database together with a unique identifier. If a frame's vector matches a known vector, then the commercial is recognized. Of course, there is the risk that the match is incorrect. This type of error is known as a false positive. The false positive rate can be reduced to any desired value, but at the expense of the false negative rate. A false negative occurs when a frame's vector is not matched to the database even though the advertisement is present in the database. There are several reasons why a frame's feature vector may fail to match. First, the recognition system may not be capable of 100% accuracy. Second, the extracted vector will contain noise as a result of the transmission process. This noise may alter the values of a feature vector to the extent that a match is no longer possible. Finally, there is the case where the observed commercial is not yet present in the database. In this case, it is necessary to store the commercial and pass it (e.g., to a person) for identification and subsequent entry in the database.
It is important to realize that the matching of extracted and known vectors is not equivalent to looking up a word in an electronic dictionary. Since the extracted vectors contain noise or distortions, binary search is often not possible. Instead, a statistical comparison is often made between an extracted vector and each stored vector. Common statistical measures include linear correlation and related measures such as correlation coefficient, but other methods can also be used, including clustering techniques. See, e.g., the Duda and Hart reference. These measures provide a statistical measure of the confidence of the match. A threshold can be established, usually based on the required false positive and negative rates, such that if the correlation output exceeds this threshold, then the extracted and known vectors are said to match.
If binary search was possible, then a database containing N vectors would require at most log(N) comparisons. However, in current advertisement monitoring applications there is no discussion of efficient search methods. Thus, a linear search of all N entries may be performed, perhaps halting the search when the first match is found. On average, this will require N/2 comparisons. If N is large, this can be computationally expensive. Consider a situation in which one out of 100,000 possible commercials is to be identified. Each 30-second commercial consists of 900 video frames. If all 900 frames are stored in the database, then N=90,000,000. Even if only every 10th video frame is stored in the database, its size is still nine million. While databases of this size are now common, they rely of efficient search to access entries, i.e., they do not perform a linear search. A binary search of a 90,000,000-item database requires less than 20 comparisons. In contrast, a linear search will require an average of 45,000,000!
With 9 million entries, if each vector is 1 Kbyte, then the storage requirement is 9 Gigabytes. Disk drives with this capacity are extremely cheap at this time. However, if the database must reside in memory due to real-time requirements, then this still represents a substantial memory requirement by today's standards. One reason that the data may need to be stored in memory is because of the real-time requirements of the database. If 10 channels are being simultaneously monitored within each of 50 geographic areas, then there will be 15,000 queries per second to the content recognition database, assuming each and every frame is analyzed. This query rate is low. However, if a linear search is performed then 675 billion comparisons per second will be required. This is an extremely high computational rate by today's standards. Even if only key frames are analyzed, this is unlikely to reduce the computational rate by more than an order of magnitude.
If an advertisement is not recognized, then typically, the remote monitoring system will compress the video and transmit it back to a central office. Here, the clip is identified and added to the database and the remote recognition sites are subsequently updated. Identification and annotation may be performed manually. However, automatic annotation is also possible using optical character recognition software on each frame of video, speech recognition software, close captioning information and other information sources. As these methods improve in accuracy, it is expected that they will replace manual identification and annotation.
The recognition system described can be considered to be a form of nearest neighbor search in a high dimensional feature space. This problem has been very well studied and is known to be very difficult as the dimensionality of the vectors increases. A number of possible data structures are applicable including kd-trees and vantage point trees. These data structures and associated search algorithms organize a N-point dataset (N=90,000,000 in out previous example) so that sub-linear time searches can be performed on average. However, worst-case search times can be considerably longer. Recently, Yianilos proposed an excluded middle vantage point forest for nearest neighbor search. See, e.g., the Yianilos reference. This data structure guarantees sub-linear worst-case search times, but where the search is now for a nearest neighbor within a fixed radius, .tau. The fixed radius search means that if the database contains a vector that is within .tau. of the query, then there is a match. Otherwise, no match is found. In contrast, traditional vantage point trees will always return a nearest neighbor, even if the distance between the neighbor and the query is very large. In these cases, if the distance between the query and the nearest neighbor exceeds a threshold, then they are considered not to match. This is precisely what the excluded middle vantage point forest implicitly does.
Using an excluded middle vantage point forest, will allow accurate real-time recognition of 100,000 broadcasted advertisements. This entails constructing an excluded middle vantage point forest based on feature vectors extracted from say 90,000,000 frames of video. Of course, using some form of pre-filtering that eliminates a large number of redundant frames or frames that are not considered to be good unique identifiers can reduce this number. One such pre-filter would be to only examine the I-frames used when applying MPEG compression. However, this is unlikely to reduce the work identification database (WID) size by more than one order of magnitude. Assuming 10 channels are monitored in each of 50 geographic regions, then the query rate is 15,000=10.times.50.times.30 queries per second.
§ 4.3.2 Operational Example where Extra-Work Information is Used to Identify the Work
The database centers 840b can use traditional database technology. In general, the query search initiated by an audience member is not a nearest neighbor search but can be a classical textual search procedure such as a binary search. The nearest neighbor search is appropriate for the monitoring sub-system 840a. The database centers 840b are continually updated as each new advertisement, television show or portion thereof is recognized. Standard updating algorithms can be used. However, random new entries to the database are unlikely. Rather, each new entry, or set of entries, denotes a new time segment that is later than all previously inserted items. As such, each new entry can be appended to the end of the database while still maintaining an ordered data structure that is amenable to binary and other efficient search techniques. If two entries have the same time in their time field, items can be sorted based on secondary fields such as the channel and geographic location, as depicted in
When a user query is received, the time, channel and geographic information are used to retrieve the corresponding unique identifier that is then used to access a second database that contains information associated with the identified work.
An entry 1000 in this second database is depicted in
An audience member (user) 810 watching a television commercial for example may react to the advertisement by initiating a query to the database center 840b. The device whereby the user initiates the query might be a television or set-top-box remote control, or a computer or a wireless PDA or a (WAP-enabled) cell phone or a specialized device. Typically, the query will occur during the airing of the commercial or a shortly thereafter. However, the time between the broadcasting of the advertisement and the time of the associated query is not critical and can, in some instances be much longer. For example, the audience member might bookmark the query information in a device such as a PDA or a specialized device similar to those developed by Xenote for their Itag radio linking. Later, the audience member may transmit the query to the database center 840b. This might happen hours or even days later.
The query contains information that the database center 840b uses to identify the work being viewed. This information might include the time and place where the audience member was, together with the channel being viewed. Other identifying information is also possible. The query may also contain additional information that may be used to facilitate the user's transaction and will include the return address of the user. For example, if the user is intending to order a pizza after seeing a Pizza Hut advertisement, the query may also contain personal information including his or her identity, street address and credit card information.
When the database center 840b receives a query, data in the query is used to identify the work and associated information. A number of possible actions are possible at this point. First, the database center 840b may simply function as a form of proxy server, mapping the audience member's initial query into a web address associated with the advertisement. In this case, the audience member will be sent to the corresponding Web site. The database center 840b may also send additional data included in the initial query to this Web site 850 in order to facilitate an e-commerce transaction between the audience member and the advertiser. In some cases, this transaction will not be direct, but may be indirect via a dealer or third party application service provider. Thus, for example, though an advertisement by Ford Motor Company may air nationally, viewers may be directed to different Web sites for Ford dealerships depending on both the audience member's and the dealerships' geographic locations. In other cases, advertisers may have contracted with the database center 840b to provide e-commerce capabilities. This latter arrangement has the potential to reduce the amount of traffic directed over the public Internet, restricting it, instead to a private network associated with the owner of the database center.
If the audience member (user) is not watching live television but is instead watching a taped and therefore time-shifted copy, then additional processes are needed. For the new generation of digital video recorders, irrespective of the recording media (tape or disk), it is likely to be very easy to include information identifying the location of the recorder, as well as the time and channel recorded. Location information can be provided to the recorder during the setup and installation process, for example. Digital video recorders, such as those currently manufactured by TIVO of Alviso, Calif. or Replay TV of Santa Clara, Calif. have a network connection via telephone, which can then send the query of an audience member to the database center 840b using the recorded rather than the current information.
In cases where query information has not been recorded, it is still possible to initiate a successful query. However, in this case, it may be necessary to extract the feature vector from the work of interest and send this information to the monitoring center 840a where the feature vector can be identified. This form of query is computationally more expensive but the relative number of such queries compared to those sent to the database centers 840b is expected to be small. It should also be noted that the physical separation of the monitoring and database centers, depicted in
Although the implementation architectures described above focus on the television media, it is apparent that the present invention is applicable to audio, print and other media.
None of the embodiments of the invention require modification to the work or content, i.e., no active signal is embedded. Consequently, there is no change to the production processes. More importantly, from a user perspective, deployment of this system need not suffer from poor initial coverage. Provided the database is sufficiently comprehensive, early adopters will have comprehensive coverage immediately. Thus, there is less risk that the consumer will perceive that the initial performance of the deployed system is poor. Further, the present invention permits statistics to be gathered that measure users' responses to content. This information is expected to be very useful to advertisers and publishers and broadcasters.
The present application is a continuation of U.S. patent application Ser. No. 15/365,403 (incorporated herein by reference, issued as U.S. Pat. No. 9,832,266 on Nov. 28, 2017), titled, “METHODS FOR USING EXTRACTED FEATURES TO PERFORM AN ACTION ASSOCIATED WITH IDENTIFIED ACTION INFORMATION,” filed on Nov. 30, 2016, and listing Ingemar J. Cox as the inventor, which is a continuation of U.S. patent application Ser. No. 14/981,269 (incorporated herein by reference, issued as U.S. Pat. No. 9,558,190 on Jan. 31, 2017), titled, “SYSTEM AND METHOD FOR TAKING ACTION WITH RESPECT TO AN ELECTRONIC MEDIA WORK,” filed on Dec. 28, 2015, and listing Ingemar J. Cox as the inventor, which is a continuation of U.S. patent application Ser. No. 13/842,068 (incorporated herein by reference, issued as U.S. Pat. No. 9,348,820 on May 24, 2016), titled, “SYSTEM AND METHOD FOR TAKING ACTION WITH RESPECT TO AN ELECTRONIC MEDIA WORK AND LOGGING EVENT INFORMATION RELATED THERETO,” filed on Mar. 15, 2013, and listing Ingemar J. Cox as the inventor, which is a continuation of U.S. patent application Ser. No. 13/800,573 (incorporated herein by reference, issued as U.S. Pat. No. 8,904,464 on Dec. 2, 2014), titled “METHOD FOR TAGGING AN ELECTRONIC MEDIA WORK TO PERFORM AN ACTION,” filed on Mar. 13, 2013, and listing Ingemar J. Cox as the inventor, which is a continuation of U.S. patent application Ser. No. 13/338,079 (incorporated herein by reference, issued as U.S. Pat. No. 8,640,179 on Jan. 28, 2014), titled “METHOD FOR USING EXTRACTED FEATURES FROM AN ELECTRONIC WORK,” filed on Dec. 27, 2011, and listing Ingemar J. Cox as the inventor, which is a continuation of U.S. patent application Ser. No. 11/977,202 (incorporated herein by reference, issued as U.S. Pat. No. 8,205,237 on Jun. 19, 2012), titled “IDENTIFYING WORKS, USING A SUB-LINEAR TIME SEARCH, SUCH AS AN APPROXIMATE NEAREST NEIGHBOR SEARCH, FOR INITIATING A WORK-BASED ACTION, SUCH AS AN ACTION ON THE INTERNET,” filed Oct. 23, 2007, and listing Ingemar J. Cox as the inventor, which is a continuation of U.S. patent application Ser. No. 11/445,928 (incorporated herein by reference, issued as U.S. Pat. No. 8,010,988 on Aug. 30, 2011), titled “USING FEATURES EXTRACTED FROM AN AUDIO AND/OR VIDEO WORK TO OBTAIN INFORMATION ABOUT THE WORK,” filed on Jun. 2, 2006, and listing Ingemar J. Cox as the inventor, which is a continuation-in-part of U.S. patent application Ser. No. 09/950,972 (incorporated herein by reference, issued as U.S. Pat. No. 7,058,223 on Jun. 6, 2006), titled “IDENTIFYING WORKS FOR INITIATING A WORK-BASED ACTION, SUCH AS AN ACTION ON THE INTERNET,” filed on Sep. 13, 2001, and listing Ingemar J. Cox as the inventor, which application claims benefit to the filing date of provisional patent application Ser. No. 60/232,618 (incorporated herein by reference), titled “IDENTIFYING AND LINKING TELEVISION, AUDIO, PRINT AND OTHER MEDIA TO THE INTERNET,” filed on Sep. 14, 2000 and listing Ingemar J. Cox as the inventor.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3919479 | Moon et al. | Nov 1975 | A |
| 4230990 | Lert, Jr. et al. | Oct 1980 | A |
| 4381522 | Lambert | Apr 1983 | A |
| 4450531 | Kenyon et al. | May 1984 | A |
| 4454594 | Heffron et al. | Jun 1984 | A |
| 4495526 | Baranoff-Rossine | Jan 1985 | A |
| 4499601 | Matthews | Feb 1985 | A |
| 4511917 | Kohler et al. | Apr 1985 | A |
| 4547804 | Greenberg | Oct 1985 | A |
| 4634966 | Nakatani et al. | Jan 1987 | A |
| 4639779 | Greenberg | Jan 1987 | A |
| 4677455 | Okajima | Jun 1987 | A |
| 4677466 | Lert, Jr. et al. | Jun 1987 | A |
| 4682370 | Matthews | Jul 1987 | A |
| 4697209 | Kiervit et al. | Sep 1987 | A |
| 4706121 | Young | Nov 1987 | A |
| 4739398 | Thomas et al. | Apr 1988 | A |
| 4776017 | Fujimoto | Oct 1988 | A |
| 4805020 | Greenberg | Feb 1989 | A |
| 4843526 | Price, III | Jun 1989 | A |
| 4843562 | Kenyon et al. | Jun 1989 | A |
| 4918730 | Schulze | Apr 1990 | A |
| 5210820 | Kenyon | May 1993 | A |
| 5283819 | Glick et al. | Feb 1994 | A |
| 5410326 | Goldstein | Apr 1995 | A |
| 5437050 | Lamb et al. | Jul 1995 | A |
| 5438355 | Palmer | Aug 1995 | A |
| 5465353 | Hull et al. | Nov 1995 | A |
| 5479268 | Young et al. | Dec 1995 | A |
| 5481294 | Thomas et al. | Jan 1996 | A |
| 5504518 | Ellis et al. | Apr 1996 | A |
| 5550735 | Slade et al. | Aug 1996 | A |
| 5581658 | O'Hagan et al. | Dec 1996 | A |
| 5584025 | Keithley et al. | Dec 1996 | A |
| 5589892 | Knee et al. | Dec 1996 | A |
| 5594934 | Lu et al. | Jan 1997 | A |
| 5594936 | Rebec et al. | Jan 1997 | A |
| 5607356 | Schwartz | Mar 1997 | A |
| 5629739 | Dougherty | May 1997 | A |
| 5634012 | Stefik et al. | May 1997 | A |
| 5638443 | Stefik et al. | Jun 1997 | A |
| 5648824 | Dunn et al. | Jul 1997 | A |
| 5686954 | Yoshinobu et al. | Nov 1997 | A |
| 5692213 | Goldberg et al. | Nov 1997 | A |
| 5701452 | Siefert | Dec 1997 | A |
| 5701542 | Sasayama | Dec 1997 | A |
| 5706364 | Kopec et al. | Jan 1998 | A |
| 5721827 | Logan et al. | Feb 1998 | A |
| 5724605 | Wissner | Mar 1998 | A |
| 5734853 | Hendricks et al. | Mar 1998 | A |
| 5745900 | Burrows | Apr 1998 | A |
| 5748783 | Rhoads | May 1998 | A |
| 5751956 | Kirsch | May 1998 | A |
| 5758258 | Shoff et al. | May 1998 | A |
| 5759101 | Von Kohorn | Jun 1998 | A |
| 5768426 | Rhoads | Jun 1998 | A |
| 5774664 | Hidary et al. | Jun 1998 | A |
| 5774666 | Portuesi | Jun 1998 | A |
| 5778181 | Hidary et al. | Jul 1998 | A |
| 5778187 | Monteiro et al. | Jul 1998 | A |
| 5782692 | Stelovsky | Jul 1998 | A |
| 5790176 | Craig | Aug 1998 | A |
| 5796393 | MacNaughton et al. | Aug 1998 | A |
| 5796952 | Davis et al. | Aug 1998 | A |
| 5798785 | Hendricks et al. | Aug 1998 | A |
| 5809471 | Brodsky | Sep 1998 | A |
| 5809512 | Kato | Sep 1998 | A |
| 5812123 | Rowe et al. | Sep 1998 | A |
| 5813014 | Gustman | Sep 1998 | A |
| 5818441 | Throckmorton et al. | Oct 1998 | A |
| 5818935 | Maa | Oct 1998 | A |
| 5822436 | Rhoads | Oct 1998 | A |
| 5826102 | Escobar | Oct 1998 | A |
| 5832119 | Rhoads | Nov 1998 | A |
| 5832182 | Zhang et al. | Nov 1998 | A |
| 5832499 | Gustman | Nov 1998 | A |
| 5841978 | Rhoads | Nov 1998 | A |
| 5850218 | LaJoie et al. | Dec 1998 | A |
| 5850490 | Johnson | Dec 1998 | A |
| 5855008 | Goldhaber et al. | Dec 1998 | A |
| 5859898 | Checco | Jan 1999 | A |
| 5862260 | Rhoads | Jan 1999 | A |
| 5867821 | Ballantyne | Feb 1999 | A |
| 5874686 | Ghias et al. | Feb 1999 | A |
| 5892536 | Logan et al. | Apr 1999 | A |
| 5903816 | Broadwin et al. | May 1999 | A |
| 5905865 | Palmer et al. | May 1999 | A |
| 5905988 | Schwartz et al. | May 1999 | A |
| 5907322 | Kelly et al. | May 1999 | A |
| 5914746 | Matthews | Jun 1999 | A |
| 5918223 | Blum et al. | Jun 1999 | A |
| 5926230 | Niijima et al. | Jul 1999 | A |
| 5929849 | Kikinis | Jul 1999 | A |
| 5929850 | Broadwin et al. | Jul 1999 | A |
| 5931901 | Wolfe | Aug 1999 | A |
| 5931908 | Gerba et al. | Aug 1999 | A |
| 5937331 | Kalluri et al. | Aug 1999 | A |
| 5953005 | Liu | Sep 1999 | A |
| 5953415 | Nielsen | Sep 1999 | A |
| 5956716 | Kenner et al. | Sep 1999 | A |
| 5961603 | Kunkel et al. | Oct 1999 | A |
| 5963966 | Mitchell et al. | Oct 1999 | A |
| 5973685 | Schaffa et al. | Oct 1999 | A |
| 5973723 | Deluca | Oct 1999 | A |
| 5978791 | Farber et al. | Nov 1999 | A |
| 5983171 | Yokoyama et al. | Nov 1999 | A |
| 5983176 | Hoffert et al. | Nov 1999 | A |
| 5991801 | Rebec et al. | Nov 1999 | A |
| 5996006 | Speicher | Nov 1999 | A |
| 5999689 | Iggulden | Dec 1999 | A |
| 6005561 | Hawkins et al. | Dec 1999 | A |
| 6006256 | Zdepski et al. | Dec 1999 | A |
| 6006265 | Rangan et al. | Dec 1999 | A |
| 6008803 | Rowe et al. | Dec 1999 | A |
| 6009410 | LeMole et al. | Dec 1999 | A |
| 6011758 | Dockes et al. | Jan 2000 | A |
| 6021433 | Payne et al. | Feb 2000 | A |
| 6023693 | Masuoka et al. | Feb 2000 | A |
| 6025837 | Matthews et al. | Feb 2000 | A |
| 6026439 | Chowdhury et al. | Feb 2000 | A |
| 6038591 | Wolfe | Mar 2000 | A |
| 6044376 | Kurtzman, II | Mar 2000 | A |
| 6044402 | Jacobson et al. | Mar 2000 | A |
| 6047327 | Tso et al. | Apr 2000 | A |
| 6049823 | Hwang | Apr 2000 | A |
| 6052693 | Smith et al. | Apr 2000 | A |
| 6057872 | Candelore | May 2000 | A |
| 6061056 | Menard et al. | May 2000 | A |
| 6067369 | Kamei | May 2000 | A |
| 6088455 | Logan et al. | Jul 2000 | A |
| 6088707 | Bates et al. | Jul 2000 | A |
| 6092080 | Gustman | Jul 2000 | A |
| 6096961 | Bruti et al. | Aug 2000 | A |
| 6098106 | Philyaw et al. | Aug 2000 | A |
| 6100883 | Hoarty | Aug 2000 | A |
| 6118442 | Tanigawa | Sep 2000 | A |
| 6118450 | Proehl et al. | Sep 2000 | A |
| 6119124 | Broder et al. | Sep 2000 | A |
| 6121530 | Sonoda | Sep 2000 | A |
| 6148081 | Szymanski et al. | Nov 2000 | A |
| 6154737 | Inaba et al. | Nov 2000 | A |
| 6161142 | Wolfe | Dec 2000 | A |
| 6169986 | Bowman et al. | Jan 2001 | B1 |
| 6173406 | Wang et al. | Jan 2001 | B1 |
| 6177931 | Alexander et al. | Jan 2001 | B1 |
| 6188010 | Iwamura | Feb 2001 | B1 |
| 6195693 | Berry et al. | Feb 2001 | B1 |
| 6201176 | Yourlo | Mar 2001 | B1 |
| 6205582 | Hoarty | Mar 2001 | B1 |
| 6215483 | Zigmond | Apr 2001 | B1 |
| 6229922 | Sasakawa et al. | May 2001 | B1 |
| 6230192 | Roberts et al. | May 2001 | B1 |
| 6233682 | Fritsch | May 2001 | B1 |
| 6236758 | Sodagar et al. | May 2001 | B1 |
| 6237146 | Richards | May 2001 | B1 |
| 6240409 | Aiken | May 2001 | B1 |
| 6243725 | Hempleman et al. | Jun 2001 | B1 |
| 6247133 | Palage et al. | Jun 2001 | B1 |
| 6253193 | Ginter et al. | Jun 2001 | B1 |
| 6263348 | Kathrow et al. | Jul 2001 | B1 |
| 6263505 | Walker et al. | Jul 2001 | B1 |
| 6269275 | Slade | Jul 2001 | B1 |
| 6279010 | Anderson | Aug 2001 | B1 |
| 6285407 | Yasuki et al. | Sep 2001 | B1 |
| 6289346 | Milewski et al. | Sep 2001 | B1 |
| 6305016 | Marshall et al. | Oct 2001 | B1 |
| 6317885 | Fries | Nov 2001 | B1 |
| 6326982 | Wu et al. | Dec 2001 | B1 |
| 6330593 | Roberts et al. | Dec 2001 | B1 |
| 6339810 | Basham et al. | Jan 2002 | B1 |
| 6345256 | Milsted et al. | Feb 2002 | B1 |
| 6349296 | Broder et al. | Feb 2002 | B1 |
| 6357042 | Srinivasan | Mar 2002 | B2 |
| 6360215 | Judd et al. | Mar 2002 | B1 |
| 6363377 | Kravets et al. | Mar 2002 | B1 |
| 6374225 | Hejna, Jr. | Apr 2002 | B1 |
| 6374260 | Hoffert et al. | Apr 2002 | B1 |
| 6381601 | Fujiwara et al. | Apr 2002 | B1 |
| 6385596 | Wiser et al. | May 2002 | B1 |
| 6400407 | Zigmond et al. | Jun 2002 | B1 |
| 6407680 | Lai et al. | Jun 2002 | B1 |
| 6408128 | Abecassis | Jun 2002 | B1 |
| 6415280 | Farber et al. | Jul 2002 | B1 |
| 6415438 | Blackketter et al. | Jul 2002 | B1 |
| 6418421 | Hurtado et al. | Jul 2002 | B1 |
| 6438556 | Malik et al. | Aug 2002 | B1 |
| 6446068 | Kortge | Sep 2002 | B1 |
| 6449226 | Kumagai | Sep 2002 | B1 |
| 6452874 | Otsuka et al. | Sep 2002 | B1 |
| 6453252 | Laroche | Sep 2002 | B1 |
| 6460050 | Pace et al. | Oct 2002 | B1 |
| 6460180 | Park et al. | Oct 2002 | B1 |
| 6469749 | Dimitrova | Oct 2002 | B1 |
| 6473804 | Kaiser et al. | Oct 2002 | B1 |
| 6477704 | Cremia | Nov 2002 | B1 |
| 6490279 | Chen et al. | Dec 2002 | B1 |
| 6496802 | Van Zoest et al. | Dec 2002 | B1 |
| 6496857 | Dustin et al. | Dec 2002 | B1 |
| 6505160 | Levy et al. | Jan 2003 | B1 |
| 6539548 | Hendricks et al. | Mar 2003 | B1 |
| 6542869 | Foote | Apr 2003 | B1 |
| 6546393 | Khan | Apr 2003 | B1 |
| 6550001 | Corwin et al. | Apr 2003 | B1 |
| 6550011 | Sims, III | Apr 2003 | B1 |
| 6552254 | Hasegawa et al. | Apr 2003 | B2 |
| 6563515 | Reynolds et al. | May 2003 | B1 |
| 6564379 | Knudson et al. | May 2003 | B1 |
| 6564380 | Murphy | May 2003 | B1 |
| 6567982 | Howe et al. | May 2003 | B1 |
| 6571392 | Zigmond et al. | May 2003 | B1 |
| 6577746 | Evans et al. | Jun 2003 | B1 |
| 6591245 | Klug | Jul 2003 | B1 |
| 6597405 | Iggulden | Jul 2003 | B1 |
| 6598228 | Hejna, Jr. | Jul 2003 | B2 |
| 6604242 | Weinstein et al. | Aug 2003 | B1 |
| 6609105 | Van Zoest et al. | Aug 2003 | B2 |
| 6615408 | Kaiser et al. | Sep 2003 | B1 |
| 6631523 | Matthews, III et al. | Oct 2003 | B1 |
| 6636247 | Hamzy et al. | Oct 2003 | B1 |
| 6654757 | Stern | Nov 2003 | B1 |
| 6658423 | Pugh et al. | Dec 2003 | B1 |
| 6665661 | Crow et al. | Dec 2003 | B1 |
| 6668378 | Leak et al. | Dec 2003 | B2 |
| 6675174 | Bolle et al. | Jan 2004 | B1 |
| 6675385 | Wang | Jan 2004 | B1 |
| 6693236 | Gould et al. | Feb 2004 | B1 |
| 6698020 | Zigmond et al. | Feb 2004 | B1 |
| 6711617 | Bantz et al. | Mar 2004 | B1 |
| 6738978 | Hendricks | May 2004 | B1 |
| 6754904 | Cooper et al. | Jun 2004 | B1 |
| 6757294 | Maruyama | Jun 2004 | B1 |
| 6766523 | Herley | Jul 2004 | B2 |
| 6774926 | Ellis et al. | Aug 2004 | B1 |
| 6782186 | Covell et al. | Aug 2004 | B1 |
| 6785275 | Boivie et al. | Aug 2004 | B1 |
| 6785902 | Zigmond et al. | Aug 2004 | B1 |
| 6804825 | White et al. | Oct 2004 | B1 |
| 6810388 | Sato | Oct 2004 | B1 |
| 6833865 | Fuller et al. | Dec 2004 | B1 |
| 6834110 | Marconcini et al. | Dec 2004 | B1 |
| 6834308 | Ikezoye et al. | Dec 2004 | B1 |
| 6845396 | Kanojia et al. | Jan 2005 | B1 |
| 6850252 | Hoffberg | Feb 2005 | B1 |
| 6868507 | Gurumoorthy et al. | Mar 2005 | B1 |
| 6871200 | MacQueen et al. | Mar 2005 | B2 |
| 6871231 | Morris | Mar 2005 | B2 |
| 6873982 | Bates et al. | Mar 2005 | B1 |
| 6898762 | Ellis et al. | May 2005 | B2 |
| 6912571 | Serena | Jun 2005 | B1 |
| 6928423 | Yamanaka | Aug 2005 | B1 |
| 6928442 | Farber et al. | Aug 2005 | B2 |
| 6931451 | Logan et al. | Aug 2005 | B1 |
| 6937766 | Wilf et al. | Aug 2005 | B1 |
| 6938270 | Blackkeffer et al. | Aug 2005 | B2 |
| 6941275 | Swierczek | Sep 2005 | B1 |
| 6941574 | Broadwin et al. | Sep 2005 | B1 |
| 6944632 | Stern | Sep 2005 | B2 |
| 6968337 | Wold | Nov 2005 | B2 |
| 6970886 | Conwell et al. | Nov 2005 | B1 |
| 6978419 | Kantrowitz | Dec 2005 | B1 |
| 6978461 | Shapiro et al. | Dec 2005 | B2 |
| 6983371 | Hurtado et al. | Jan 2006 | B1 |
| 6990453 | Wang et al. | Jan 2006 | B2 |
| 6990677 | Pietraszak et al. | Jan 2006 | B1 |
| 6999111 | McIntyre et al. | Feb 2006 | B2 |
| 7013301 | Holm et al. | Mar 2006 | B2 |
| 7020635 | Hamilton et al. | Mar 2006 | B2 |
| 7028327 | Dougherty et al. | Apr 2006 | B1 |
| 7035914 | Payne et al. | Apr 2006 | B1 |
| 7039935 | Knudson et al. | May 2006 | B2 |
| 7043473 | Rassool et al. | May 2006 | B1 |
| 7055166 | Logan et al. | May 2006 | B1 |
| 7058223 | Cox | Jun 2006 | B2 |
| 7065709 | Ellis et al. | Jun 2006 | B2 |
| 7092953 | Haynes | Aug 2006 | B1 |
| 7096486 | Ukai et al. | Aug 2006 | B1 |
| 7100185 | Bennington et al. | Aug 2006 | B2 |
| 7103905 | Novak | Sep 2006 | B2 |
| 7103906 | Katz et al. | Sep 2006 | B1 |
| 7106904 | Shuma | Sep 2006 | B2 |
| 7140033 | Durden et al. | Nov 2006 | B1 |
| 7146631 | Tanaka et al. | Dec 2006 | B1 |
| 7152236 | Wugofski et al. | Dec 2006 | B1 |
| 7155449 | Pingel et al. | Dec 2006 | B2 |
| 7158929 | Wouters et al. | Jan 2007 | B2 |
| 7165266 | Zigmond | Jan 2007 | B2 |
| 7168083 | Kalker et al. | Jan 2007 | B2 |
| 7171016 | Rhoads | Jan 2007 | B1 |
| 7174293 | Kenyon et al. | Feb 2007 | B2 |
| 7181756 | Zigmond et al. | Feb 2007 | B1 |
| 7184100 | Wilf et al. | Feb 2007 | B1 |
| 7188353 | Crinon | Mar 2007 | B1 |
| 7191190 | Debique et al. | Mar 2007 | B2 |
| 7194754 | Tomsen et al. | Mar 2007 | B2 |
| 7213005 | Mourad et al. | May 2007 | B2 |
| 7222163 | Girouard et al. | May 2007 | B1 |
| 7225455 | Bennington et al. | May 2007 | B2 |
| 7237253 | Blackketter et al. | Jun 2007 | B1 |
| 7243139 | Ullman | Jul 2007 | B2 |
| 7243153 | McIntyre et al. | Jul 2007 | B2 |
| 7251475 | Kawamoto | Jul 2007 | B2 |
| 7254829 | Brown et al. | Aug 2007 | B1 |
| 7272788 | Anderson et al. | Sep 2007 | B2 |
| 7272843 | Nejime et al. | Sep 2007 | B1 |
| 7277870 | Mourad et al. | Oct 2007 | B2 |
| 7302574 | Conwell et al. | Nov 2007 | B2 |
| 7305693 | Blackketter et al. | Dec 2007 | B2 |
| 7308413 | Tota et al. | Dec 2007 | B1 |
| 7313805 | Rosin et al. | Dec 2007 | B1 |
| 7334250 | Blackketter et al. | Feb 2008 | B2 |
| 7340763 | Harris | Mar 2008 | B1 |
| 7346472 | Moskowitz et al. | Mar 2008 | B1 |
| 7349668 | Ilan et al. | Mar 2008 | B2 |
| 7363278 | Schmelzer et al. | Apr 2008 | B2 |
| 7366718 | Pugh et al. | Apr 2008 | B1 |
| 7366787 | Salas et al. | Apr 2008 | B2 |
| 7369677 | Petrovic et al. | May 2008 | B2 |
| 7370017 | Lindeman et al. | May 2008 | B1 |
| 7386512 | Allibhoy et al. | Jun 2008 | B1 |
| 7392532 | White et al. | Jun 2008 | B2 |
| 7404200 | Hailey et al. | Jul 2008 | B1 |
| 7409437 | Ullman et al. | Aug 2008 | B2 |
| 7421723 | Harkness et al. | Sep 2008 | B2 |
| 7423771 | Ohata et al. | Sep 2008 | B2 |
| 7426558 | Allibhoy et al. | Sep 2008 | B1 |
| 7444353 | Chen et al. | Oct 2008 | B1 |
| 7471834 | Sull et al. | Dec 2008 | B2 |
| 7477739 | Haitsma et al. | Jan 2009 | B2 |
| 7483958 | Elabbady et al. | Jan 2009 | B1 |
| 7487527 | Ellis et al. | Feb 2009 | B2 |
| 7493643 | Ellis | Feb 2009 | B2 |
| 7500007 | Ikezoye et al. | Mar 2009 | B2 |
| 7506352 | Blackketter et al. | Mar 2009 | B2 |
| 7523312 | Kalker et al. | Apr 2009 | B2 |
| 7523478 | Blackketter et al. | Apr 2009 | B2 |
| 7529659 | Wold | May 2009 | B2 |
| 7548565 | Sull et al. | Jun 2009 | B2 |
| 7562012 | Wold et al. | Jul 2009 | B1 |
| 7562392 | Rhoads et al. | Jul 2009 | B1 |
| 7565294 | Rhoads | Jul 2009 | B2 |
| 7565327 | Schmelzer | Jul 2009 | B2 |
| 7587728 | Wheeler et al. | Sep 2009 | B2 |
| 7595914 | Haining | Sep 2009 | B2 |
| 7606883 | Allibhoy et al. | Oct 2009 | B1 |
| 7624337 | Sull et al. | Nov 2009 | B2 |
| 7631072 | Allibhoy et al. | Dec 2009 | B2 |
| 7644429 | Bayassi et al. | Jan 2010 | B2 |
| 7647604 | Ramaswamy | Jan 2010 | B2 |
| 7650616 | Lee | Jan 2010 | B2 |
| 7660700 | Moskowitz et al. | Feb 2010 | B2 |
| 7690020 | Lebar | Mar 2010 | B2 |
| 7707088 | Schmelzer | Apr 2010 | B2 |
| 7711652 | Schmelzer | May 2010 | B2 |
| 7712125 | Herigstad et al. | May 2010 | B2 |
| 7720707 | Mowry | May 2010 | B1 |
| 7738704 | Lienhart et al. | Jun 2010 | B2 |
| 7743092 | Wood | Jun 2010 | B2 |
| 7757248 | Harkness et al. | Jul 2010 | B2 |
| 7757254 | Shoff et al. | Jul 2010 | B2 |
| 7765575 | Zigmond | Jul 2010 | B2 |
| 7783489 | Kenyon et al. | Aug 2010 | B2 |
| 7797249 | Schmelzer et al. | Sep 2010 | B2 |
| 7802281 | Tani et al. | Sep 2010 | B1 |
| 7818768 | Blackketter et al. | Oct 2010 | B2 |
| 7835920 | Snyder et al. | Nov 2010 | B2 |
| 7840975 | Matheny et al. | Nov 2010 | B2 |
| 7849226 | Zigmond et al. | Dec 2010 | B2 |
| 7853664 | Wang et al. | Dec 2010 | B1 |
| 7861275 | Vellaikal et al. | Dec 2010 | B1 |
| 7870088 | Chen et al. | Jan 2011 | B1 |
| 7877438 | Schrempp et al. | Jan 2011 | B2 |
| 7882518 | Finseth et al. | Feb 2011 | B2 |
| 7917645 | Ikezoye | Mar 2011 | B2 |
| 7926079 | Lebar | Apr 2011 | B2 |
| 7930719 | Ellis et al. | Apr 2011 | B2 |
| 7941816 | Harkness et al. | May 2011 | B2 |
| 7945929 | Knudson et al. | May 2011 | B2 |
| 7949494 | Moskowitz et al. | May 2011 | B2 |
| 7949749 | Allibhoy et al. | May 2011 | B2 |
| 7962414 | Allibhoy et al. | Jun 2011 | B1 |
| 7987492 | Liwerant et al. | Jul 2011 | B2 |
| 7996565 | Allibhoy et al. | Aug 2011 | B2 |
| 8001569 | Marler et al. | Aug 2011 | B2 |
| 8006263 | Ellis et al. | Aug 2011 | B2 |
| 8006264 | Reynolds et al. | Aug 2011 | B2 |
| 8006314 | Wold | Aug 2011 | B2 |
| 8065615 | Murray et al. | Nov 2011 | B2 |
| 8082150 | Wold | Dec 2011 | B2 |
| 8086445 | Wold et al. | Dec 2011 | B2 |
| 8090605 | Tota et al. | Jan 2012 | B2 |
| 8094949 | Rhoads | Jan 2012 | B1 |
| 8108886 | Murahashi et al. | Jan 2012 | B1 |
| 8112776 | Schein et al. | Feb 2012 | B2 |
| 8171509 | Girouard et al. | May 2012 | B1 |
| 8171510 | Kamen et al. | May 2012 | B2 |
| 8185923 | Slaney et al. | May 2012 | B2 |
| 8205237 | Cox | Jun 2012 | B2 |
| 8214175 | Moskowitz et al. | Jul 2012 | B2 |
| 8214254 | Mowry | Jul 2012 | B1 |
| 8219446 | Mowry | Jul 2012 | B1 |
| RE43578 | Sorensen | Aug 2012 | E |
| 8249924 | Mowry | Aug 2012 | B1 |
| 8255952 | Boylan, III et al. | Aug 2012 | B2 |
| RE43671 | Sorensen | Sep 2012 | E |
| 8272011 | Yuen et al. | Sep 2012 | B2 |
| 8296792 | Sahota et al. | Oct 2012 | B2 |
| 8301758 | Allibhoy et al. | Oct 2012 | B2 |
| 8340994 | Tota et al. | Dec 2012 | B2 |
| 8406607 | Nesvadba et al. | Mar 2013 | B2 |
| 8438608 | Liwerant et al. | May 2013 | B2 |
| 8464302 | Liwerant et al. | Jun 2013 | B1 |
| 8479233 | Ellis et al. | Jul 2013 | B2 |
| 8479246 | Hudson et al. | Jul 2013 | B2 |
| 8572279 | Payne et al. | Oct 2013 | B2 |
| 8601154 | Payne et al. | Dec 2013 | B2 |
| 9143718 | Nagorski et al. | Sep 2015 | B2 |
| 9414008 | Nagorski et al. | Aug 2016 | B2 |
| 20010001160 | Shoff et al. | May 2001 | A1 |
| 20010003818 | Pingel et al. | Jun 2001 | A1 |
| 20010014210 | Kang | Aug 2001 | A1 |
| 20010018771 | Walker et al. | Aug 2001 | A1 |
| 20010033736 | Yap | Oct 2001 | A1 |
| 20010037376 | Ullman | Nov 2001 | A1 |
| 20010047298 | Moore et al. | Nov 2001 | A1 |
| 20010049625 | Mowry | Dec 2001 | A1 |
| 20010052132 | Fryer | Dec 2001 | A1 |
| 20020023020 | Kenyon et al. | Feb 2002 | A1 |
| 20020026369 | Miller et al. | Feb 2002 | A1 |
| 20020032698 | Cox | Mar 2002 | A1 |
| 20020035600 | Ullman | Mar 2002 | A1 |
| 20020035601 | Ullman | Mar 2002 | A1 |
| 20020035614 | Ullman | Mar 2002 | A1 |
| 20020035615 | Ullman | Mar 2002 | A1 |
| 20020038296 | Margolus et al. | Mar 2002 | A1 |
| 20020038383 | Ullman et al. | Mar 2002 | A1 |
| 20020042813 | Ullman et al. | Apr 2002 | A1 |
| 20020046178 | Morito et al. | Apr 2002 | A1 |
| 20020049832 | Ullman et al. | Apr 2002 | A1 |
| 20020056091 | Bala et al. | May 2002 | A1 |
| 20020056104 | Burnhouse et al. | May 2002 | A1 |
| 20020056123 | Liwerant et al. | May 2002 | A1 |
| 20020056129 | Blackketter et al. | May 2002 | A1 |
| 20020059610 | Ellis | May 2002 | A1 |
| 20020069218 | Sull et al. | Jun 2002 | A1 |
| 20020073424 | Ward et al. | Jun 2002 | A1 |
| 20020082731 | Pitman et al. | Jun 2002 | A1 |
| 20020083005 | Lowenstein et al. | Jun 2002 | A1 |
| 20020087661 | Matichuk et al. | Jul 2002 | A1 |
| 20020087885 | Peled et al. | Jul 2002 | A1 |
| 20020088336 | Stahl | Jul 2002 | A1 |
| 20020092019 | Marcus | Jul 2002 | A1 |
| 20020099555 | Pitman et al. | Jul 2002 | A1 |
| 20020104099 | Novak | Aug 2002 | A1 |
| 20020112002 | Abato | Aug 2002 | A1 |
| 20020120925 | Logan | Aug 2002 | A1 |
| 20020133499 | Ward et al. | Sep 2002 | A1 |
| 20020138843 | Samaan et al. | Sep 2002 | A1 |
| 20020150164 | Felts et al. | Oct 2002 | A1 |
| 20020156760 | Lawrence et al. | Oct 2002 | A1 |
| 20020156909 | Harrington | Oct 2002 | A1 |
| 20020176702 | Frantz | Nov 2002 | A1 |
| 20020178276 | McCartney et al. | Nov 2002 | A1 |
| 20020184634 | Cooper | Dec 2002 | A1 |
| 20020186887 | Rhoads | Dec 2002 | A1 |
| 20020188699 | Ullman et al. | Dec 2002 | A1 |
| 20030005151 | Ullman et al. | Jan 2003 | A1 |
| 20030019527 | Oppermann et al. | Jan 2003 | A1 |
| 20030028489 | Williamson | Feb 2003 | A1 |
| 20030037010 | Schmelzer | Feb 2003 | A1 |
| 20030061490 | Abajian | Mar 2003 | A1 |
| 20030065719 | Ullman | Apr 2003 | A1 |
| 20030081939 | Kim et al. | May 2003 | A1 |
| 20030088674 | Ullman | May 2003 | A1 |
| 20030093790 | Logan et al. | May 2003 | A1 |
| 20030095660 | Lee | May 2003 | A1 |
| 20030101144 | Moreno | May 2003 | A1 |
| 20030101232 | Ullman | May 2003 | A1 |
| 20030106017 | Kanchirayappa et al. | Jun 2003 | A1 |
| 20030123276 | Yokozeki | Jul 2003 | A1 |
| 20030146940 | Ellis et al. | Aug 2003 | A1 |
| 20030167300 | Ullman | Sep 2003 | A1 |
| 20030182113 | Huang | Sep 2003 | A1 |
| 20030191816 | Landress | Oct 2003 | A1 |
| 20030202660 | Zhou et al. | Oct 2003 | A1 |
| 20030208756 | Macrae et al. | Nov 2003 | A1 |
| 20030226150 | Berberet | Dec 2003 | A1 |
| 20030233930 | Ozick | Dec 2003 | A1 |
| 20040003398 | Donian et al. | Jan 2004 | A1 |
| 20040010602 | Van Vleck et al. | Jan 2004 | A1 |
| 20040015608 | Ellis et al. | Jan 2004 | A1 |
| 20040025174 | Cerrato et al. | Feb 2004 | A1 |
| 20040030759 | Hidary | Feb 2004 | A1 |
| 20040078825 | Murphy | Apr 2004 | A1 |
| 20040123314 | Bova | Jun 2004 | A1 |
| 20040163106 | Schrempp et al. | Aug 2004 | A1 |
| 20040170335 | Pearlman et al. | Sep 2004 | A1 |
| 20040199387 | Wang et al. | Oct 2004 | A1 |
| 20040221118 | Slater et al. | Nov 2004 | A1 |
| 20040234936 | Ullman | Nov 2004 | A1 |
| 20040240562 | Bargeron et al. | Dec 2004 | A1 |
| 20040243540 | Moskowitz et al. | Dec 2004 | A1 |
| 20050015795 | Iggulden | Jan 2005 | A1 |
| 20050015815 | Shoff et al. | Jan 2005 | A1 |
| 20050044189 | Ikezoye et al. | Feb 2005 | A1 |
| 20050044577 | Jerding | Feb 2005 | A1 |
| 20050080846 | McCleskey et al. | Apr 2005 | A1 |
| 20050097622 | Zigmond et al. | May 2005 | A1 |
| 20050102515 | Jaworski et al. | May 2005 | A1 |
| 20050154892 | Mihcak et al. | Jul 2005 | A1 |
| 20050160363 | Bhogal et al. | Jul 2005 | A1 |
| 20050193016 | Seet et al. | Sep 2005 | A1 |
| 20050213826 | Neogi | Sep 2005 | A1 |
| 20050235318 | Grauch et al. | Oct 2005 | A1 |
| 20050235319 | Carpenter et al. | Oct 2005 | A1 |
| 20050240961 | Jerding et al. | Oct 2005 | A1 |
| 20050246752 | Liwerant et al. | Nov 2005 | A1 |
| 20050283800 | Ellis et al. | Dec 2005 | A1 |
| 20050289065 | Weare | Dec 2005 | A1 |
| 20060015925 | Logan | Jan 2006 | A1 |
| 20060031870 | Jarman et al. | Feb 2006 | A1 |
| 20060080356 | Burges et al. | Apr 2006 | A1 |
| 20060085816 | Funk et al. | Apr 2006 | A1 |
| 20010110137 | Tsuda et al. | May 2006 | |
| 20060101069 | Bell et al. | May 2006 | A1 |
| 20010187358 | Lienhart et al. | Aug 2006 | |
| 20060195859 | Konig et al. | Aug 2006 | A1 |
| 20060195860 | Eldering et al. | Aug 2006 | A1 |
| 20060206462 | Barber | Sep 2006 | A1 |
| 20060212927 | Riku et al. | Sep 2006 | A1 |
| 20060271947 | Lienhart et al. | Nov 2006 | A1 |
| 20070041667 | Cox | Feb 2007 | A1 |
| 20070071330 | Oostveen et al. | Mar 2007 | A1 |
| 20070083510 | McArdle | Apr 2007 | A1 |
| 20070101360 | Gutta et al. | May 2007 | A1 |
| 20070118375 | Kenyon et al. | May 2007 | A1 |
| 20070124698 | Majumdar | May 2007 | A1 |
| 20070130580 | Covell et al. | Jun 2007 | A1 |
| 20070180537 | He et al. | Aug 2007 | A1 |
| 20070203911 | Chiu | Aug 2007 | A1 |
| 20070214482 | Nguyen | Sep 2007 | A1 |
| 20070282472 | Seidman | Dec 2007 | A1 |
| 20070288518 | Crigler et al. | Dec 2007 | A1 |
| 20070294173 | Levy et al. | Dec 2007 | A1 |
| 20080052783 | Levy | Feb 2008 | A1 |
| 20080072260 | Rosin et al. | Mar 2008 | A1 |
| 20080091684 | Ellis et al. | Apr 2008 | A1 |
| 20080162478 | Pugh et al. | Jul 2008 | A1 |
| 20080189749 | White et al. | Aug 2008 | A1 |
| 20080250241 | Ginter et al. | Oct 2008 | A1 |
| 20090052784 | Covell et al. | Feb 2009 | A1 |
| 20090226148 | Nesvadba et al. | Sep 2009 | A1 |
| 20090328236 | Schmelzer | Dec 2009 | A1 |
| 20100158488 | Roberts et al. | Jun 2010 | A1 |
| 20100211969 | Schein et al. | Aug 2010 | A1 |
| 20100211975 | Boyer et al. | Aug 2010 | A1 |
| 20100290666 | Rhoads | Nov 2010 | A1 |
| 20110167449 | Klosterman et al. | Jul 2011 | A1 |
| 20110173660 | Schein et al. | Jul 2011 | A1 |
| 20120078871 | Pugh et al. | Mar 2012 | A1 |
| 20130086608 | Slaney et al. | Apr 2013 | A1 |
| 20130125158 | Brown | May 2013 | A1 |
| 20130302011 | Nagorski et al. | Nov 2013 | A1 |
| Number | Date | Country |
|---|---|---|
| 0194143 | Oct 1986 | EP |
| 0838960 | Apr 1998 | EP |
| 0849946 | Jun 1998 | EP |
| 1 164 796 | Dec 2001 | EP |
| 1 35427681 | Dec 2007 | EP |
| 1354276 | Dec 2007 | EP |
| 1 48581581 | Jul 2009 | EP |
| 2369203 | May 2002 | GB |
| 2386807 | Sep 2003 | GB |
| 07-284035 | Oct 1995 | JP |
| 11-150692 | Jun 1999 | JP |
| 2003-242281 | Aug 2003 | JP |
| 94006084 | Mar 1994 | WO |
| 9406084 | Mar 1994 | WO |
| 1997005616 | Feb 1997 | WO |
| 1998041020 | Sep 1998 | WO |
| 9904568 | Jan 1999 | WO |
| 9904568 | Jan 1999 | WO |
| 9910822 | Mar 1999 | WO |
| 9941684 | Aug 1999 | WO |
| 99050778 | Oct 1999 | WO |
| 9950778 | Oct 1999 | WO |
| 0016205 | Mar 2000 | WO |
| 0020960 | Apr 2000 | WO |
| 01010124 | Feb 2001 | WO |
| 01010127 | Feb 2001 | WO |
| 2001011865 | Feb 2001 | WO |
| 01022688 | Mar 2001 | WO |
| 2001022730 | Mar 2001 | WO |
| 01038960 | May 2001 | WO |
| 2002008948 | Jan 2002 | WO |
| WO 2002011033 | Feb 2002 | WO |
| WO 2002103968 | Dec 2002 | WO |
| 2004019527 | Mar 2004 | WO |
| 2005006758 | Jan 2005 | WO |
| Entry |
|---|
| Adams, Open Cable Architecture (Cisco Press 2000), Chapter 8-16. |
| ATSC Standard: Program and System Information Protocol for Terrestrial Broadcast and Cable (PSIP) (Dec. 23, 1997). |
| Attack of the $500 Killer Network Computers: Time-Warner Cable's Full Service Network, Network Computing (Aug. 19, 2000) Internet Archive, https://web.archive.org/web/20000819050301/http://www.networkcomputing.com/616/616tw.html. |
| Full Service Network and the Orlando Sentinel add interactive dining guide to GOtv, The Free Library, http://www.thefreelibrary.com/Full+Service+Network+and+The+Orlando+Sentinel+add+interactive+dining...-a018299720 (published May 20, 1996, last visited Mar. 4, 2015). |
| Full Service Network(FSN) in Orlando, Florida, Hong Kong University of Science and Technology (May 4, 1997) Internet Archive, https://web.archive.org/web/19970504203603/http://www.ust.hk/˜webiway/content/USA/Trial/fsn.html. |
| Full Service Network, Time Warner Cable, http://m.history.timewarnercable.com/the-twc-story/era-1990-1995/Story.aspx?story=56 (last visited Mar. 4, 2015). |
| Time Warner Cable, DRAFT Asset Distribution System (“Catcher's Mitt”) Functional Requirements, Version 1.0 (Jan. 26, 2000). |
| Time Warner Cable, Request for Proposal and Functional Requirements Specification for Video-on-Demand (VOD) Systems, Version 2.0 (Apr. 25, 1997). |
| Time Warner Cable's Full Service Network connects live to Innoventions, AllBusiness (published Apr. 10, 1996, archived Mar. 25, 2008) Internet Archive, https://web.archive.org/web/20080325024937/http://www.allbusiness.com/media-telecommunications/telecommunications/7218809-1.html. |
| Time Warner introduces world's first full service network in Orlando; Network offers First . . . , AllBusiness (published Dec. 14, 1994, archived May 22, 2009) Internet Archive, https://web.archive.org/web/20090522134441/http://www.allbusiness.com/media-telecommunications/telecommunications/7087127-1.html. |
| Time Warner will close its Full Service Network, Orlando Business Journal, http://www.bizjournals.com/orlando/stories/1997/04/28/daily7.html (last visited Mar. 4, 2015). |
| Time Warner's ‘Time Machine’ for Future Video, The New York Times, http://www.nytimes.com/.../12/business/time-warner-s-time-machine-for-future-video.html?pagewanted=2&pagewanted=print (publsihed Dec. 12, 1994, last visited Mar. 4, 2015). |
| The Industry Standard, Web Entertainment Gets Personal, S. Rosenbaum, Jan. 10-17, 2000, www.thestandard.com. |
| U.S. Appl. No. 60/188,082, filed Mar. 9, 2000. |
| Duda et al., Pattern Classification, 2nd Edition, John Wiley & Sons, Inc., 2001. |
| Chang et al., Searching Near-Replicas of Images via Clustering, SPIE Symposium of Voice, Video, and Data Communications, 1999. |
| Fukunaga et al., A Branch and Bound Algorithm for Computing k-Nearest Neighbors, IEEE Transactions on computers, Jul. 1975, pp. 750-753, vol. 24, No. 7. |
| Fukunaga, Introduction to Statistical Pattern Recognition, 2nd Edition, Academic Press Inc., San Diego, CA, 1990. |
| Ester et al., A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise, KDD-96 Proceedings, 1996. |
| Salton et al., Generation and Search of Clustered Files, Cornell University, Technical Report 77-299, 1977. |
| Tishby et al., The Information Bottleneck Method, Sep. 30, 1999. |
| Yianilos, Excluded Middle Vantage Point Forests for Nearest Neighbor Search, Jul. 20, 1998, revised Aug. 1, 1999. |
| Opinion and Judgment entered (Vacated-in-Part and Remanded), Google LLC v. Network-1 Technologies, Inc., Case No. 16-2509 in the U.S. Court of Appeals for the Federal Circuit, Doc. 74-2, Mar. 26, 2018, 16 pages. |
| Brin Sergey et al. “Copy Detection Mechanisms for Digital Documents” Proc. of ACM SIGMOD 44 Annual Conf. (San Jose 1995) http://www-db.stanford.edu!˜sergey/copy.html on Nov. 27, 2000 21 pages. |
| Broder Andrei Z. “Some applications of Rabin's fingerprinting method” R. Capocelli A. 47 DeSantis U. Vaccaro Eds; Sequences II: Methods in Communications Security and Computer Science pp. 143-152 (Springer-Verlag 1993) 10 pages. |
| Ardizzone, Edoardo et al., “Motion and Color-Based Video Indexing and Retrieval,” Universita di palermo, Departimento di Ingegneria Elettrica, pp. 135-139, Viale delle Scienze, Palermo, Italy, IEEE 1996. |
| Deng, Yining et al., “Content-based Search of Video Using Color, Texture, and Motion,” Dept. of Electrical and Computer Engineering, University of California, Santa Barbara, CA, pp. 534-537, IEEE 1997. |
| Fang, Min et al., “Computing Iceberg Queries Efficiently,” Dept. of Computer Science, Stanford, CA, Paper No. 234, pp. 1-25. |
| Flickner, Myron et al., “Query by Image and Video Content: The QBIC System,” IBM Almaden Research Center, Sep. 1995, pp. 23-32, IEEE 1995. |
| Gargi, U et al., “Performance Characterization and Comparison of Video Indexing Algorithms,” Dept. of Computer Science & Engineering, The Pennsylvania State University, University Park, PA. |
| Gionis, Aristides et al., “Similarity Search in High Dimensions via Hashing,” Dept. of Computer Science, Stanford University, Stanford, CA, pp. 518-529, Proceeding of the 25th VLDB Conference, Edinburgh, Scotland, 1999. |
| Indyk, Piotr et al., “Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality” (preliminary version) Dept. of Computer Science, Stanford University, Stanford, CA, pp. 1-13 & i-vii, Jul. 21, 1999. |
| Iyengar, Giridharan et al., “Models for automatic classification of video sequences,” MIT Media Laboratory, Cambridge, MA. |
| Jain, Anil K., et al., “Image Retrieval using Color and Shape,” Dept. of Computer Science, Michigan State University, Eas Lansing, MI, pp. 1-24, May 15, 1995. |
| Ogle, Virginia E., et al., “Chabot: Retrieval from a Relational Database of Images,” University of California at Berkeley, Computer pp. 40-48, IEEE 1995. |
| Pentland, A. et al., “Photobook: Content-Based Manipulation of Image Databases,” Perceptual Computing Section, The Media Laboratory, Massachusetts Institute of Tech., International Journal of Computer Vision 18(3), pp. 233-254 (1996), 1996 Kluwer Academic Publishers. Manuf. in the Netherlands. |
| Shivakumar, Narayanan et al., “SCAM: A Copy Detection Mechanism for Digital Documents,” Dept. of Computer Science, Stanford University, Stanford, CA, pp. 1-13. |
| Shivakumar, Narayanan et al., “Building a Scalable and Accurate Copy Detection Mechanism,” Dept. of Computer Science, Stanford University, Stanford, CA. |
| Srihari, Rohini K., “Automatic Indexing and Content-Based Retrieval of Captioned Images,” State University of New York, Buffalo, Theme Feature, pp. 49-56, Sep. 1995, IEEE 1995. |
| Swain, Michael and Ballard, Dana H., “Color Indexing,” International Journal of Computer Vision 7:1, p. 11-32 (1991), 1991 Kluwer Academic Publishers. Manuf. in The Netherlands. |
| Wactlar Howard D. et al “Intelligent Access to Digital Video: Informedia Project,” Carnegie Mellon University, Digital Library Initiative: Carnegie Mellon University, Computer, pp. 46-52, IEEE 1996. |
| Yeo, Boon-Lock et al., “Rapid Scene Analysis on Compressed Video,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 5, No. 6, pp. 533-544, Dec. 1995, Dept. of Electrical Engineering, Princeton University, Princeton, NJ, IEEE Log No. 9415901. |
| Indyk, Piotr et al., “Finding pirated video sequences on the Internet,” Dept. of Computer Science, Stanford University, Palo Alto, CA, Paper No. 199. |
| U.S. Appl. No. 60/133,247, filed May 5, 1999. |
| U.S. Appl. No. 60/155,064, filed Sep. 21, 1999. |
| U.S. Appl. No. 60/218,824, filed Jul. 18, 2000. |
| Indyk, Piotr et al., “Locality-Preserving Hashing in Multidimensional Spaces,” Feb. 25, 1997. |
| Gibson, David, “Name That Clip: Music retrieval using audio clips,” Aug. 19, 1999. |
| Declaration of David A. Gibson, Inter Partes Review of U.S. Pat. No. 7,174,293, Aug. 30, 2013. |
| Declaration of David A. Gibson, Inter Partes Review of U.S. Pat. No. 7,783,489, Aug. 30, 2013. |
| Intersil, “Glossary of Communication Terms,” Dec. 1996. |
| Declaration of Dr. Ton Kalker, Inter Partes Review of U.S. Pat. No. 7,174,293, Aug. 30, 2013. |
| Declaration of Dr. Ton Kalker, Inter Partes Review of U.S. Pat. No. 7,783,489, Aug. 30, 2013. |
| U.S. Appl. No. 60/222,023, filed Jul. 31, 2000; Avery Li-Chun Wang and Julius O. Smith III, Inventors; Palo Alto, CA. |
| Martin Ester et al., “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise,” Proceedings of 2nd International Conference on Knowledge Discovery and Data Mining (KDD-96), 1996. |
| Yossi Rubner et al., “Adaptive Color Image Embeddings for Database Navigation*,” Proceedings of the 1998 IEEE Asian Conference on Computer Vision. |
| Roger Weber et al., “A Quantitative Analysis and Performance Study for Similarity-Search Methods in High-Dimensional Spaces,” Proceedings of 24th VLDB Conference, 1998. |
| P. Yianilos, “Data Structures and Algorithms for Nearest Neighbor Search in General Metric Spaces,” Proceedings of the ACM-SIAM Symposium on Discrete algorithms, 1993, pp. 311.321. |
| Decision Instituting Covered Business Method Review, Google Inc. v. Network-1 Technologies, Inc., No. CBM2015-00113, Paper No. 7 (PTAB Oct. 19, 2015), re U.S. Pat. No. 8,904,464. |
| Petition for Inter Partes Review of U.S. Pat. No. 8,640,179, IPR2015-00343, Paper 1 (PTAB Dec. 3, 2014). |
| Petition for Inter Partes Review of U.S. Pat. No. 8,205,237, IPR2015-00345, Paper 1 (PTAB Dec. 3, 2014). |
| Petition for Inter Partes Review of U.S. Pat. No. 8,010,988, IPR2015-00347, Paper 1 (PTAB Dec. 3, 2014). |
| Petition for Inter Partes Review of U.S. Pat. No. 8,656,441, IPR2015-00348, Paper 1 (PTAB Dec. 3, 2014). |
| Declaration of Dr. Pierre Moulin in Support of Petition for Inter Partes Review of U.S. Pat. No. 8,640,179, IPR2015-00343 (PTAB Dec. 3, 2014). |
| Declaration of Dr. Pierre Moulin in Support of Petition for Inter Partes Review of U.S. Pat. No. 8,205,237, IPR2015-00345 (PTAB Dec. 3, 2014). |
| Declaration of Dr. Pierre Moulin in Support of Petition for Inter Partes Review of U.S. Pat. No. 8,010,988, IPR2015-00347 (PTAB Dec. 3, 2014). |
| Declaration of Dr. Pierre Moulin in Support of Petition for Inter Partes Review of U.S. Pat. No. 8,656,441, IPR2015-00348 (PTAB Dec. 3, 2014). |
| Prosecution History of U.S. Appl. No. 09/438,469. |
| Aristides, Gionis, et al., “Similarity Search in High Dimensions via Hashing”, Proceedings of the 25th VLDB Conference, 1999, Edinburgh, Scotland. |
| Timo, Raita, “Tuning the Boyer-Moore-Horspool String Searching Algorithm”, Software-Practice and Experience, Oct. 1992, 879-884, vol. 22(10). |
| Richard O. Duda & Peter E. Hart, Pattern Classification and Scene Analysis (1973). |
| Keinosuke Fukunaga, Introduction to Statistical Pattern Recognition (1990). |
| Paul C. Denny, Search and Enumeration Techniques for Incidence Structures, Research Report CDMTCS-085, Centre for Discrete Mathematics and Theoretical Computer Science (May 1998). |
| John Orwant et al., Mastering Algorithms with PERL (1999). |
| Carlborg O., Andersson L. & Kinghorn B, The Use of a Genetic Algorithm for Simultaneous Mapping of Multiple Interacting Quantitative Trait Loci, Genetics 155: 2003-2010 (2000). |
| Dufaux, F. & Moscheni, F., Background Mosaicking for Low Bit Rate Video Coding, Image Processing, Proceedings of the IEEE International Conference on Image Processing, vol. 3, pp. 673-676 (Sep. 1996). |
| Robertson, G., Miller, J., & Thomson, P., Non-Exhaustive Search Methods and their Use in the Minimisation of Reed-Muller Canonical Expansions, Int. J. Electronics vol. 80, No. 1, pp. 1-12 (1996). |
| Sudhir Kumar, A Stepwise Algorithm for Finding Minimum Evolution Trees, Mol. Biol. Evol., vol. 13 (1996), pp. 584-593. |
| U.S. Appl. No. 60/230,931, filed Sep. 13, 2000. |
| Network-1 Technologies, Inc. v. Google, Inc. et al., No. 1:14-cv-02396 (S.D.N.Y. filed Apr. 4, 2014). |
| Petition for Covered Business Method Review of U.S. Pat. No. 8,904,464, CBM2015-00113 (filed Apr. 13, 2015). |
| Declaration of Pierre Moulin, Google Inc. v. Network-1 Technologies, Inc., Ex. 1003, CBM2015-00113. |
| W.J.E. Crissy and Gary A. Marple, What about Reader Service Cards?, 27 Journal of Marketing, No. 1, at 56-60 (Jan. 1963). |
| Thomas Publishing, Industrial Equipment News (Jun. 29, 1998), available at http://web.archive.org/web/19980629025648/http://www.thomaspublishing.com/annivienist.html. |
| Joint Claim Construction Chart, Network-1 Technologies, Inc. v. Google, Inc. et al., No. 1:14-cv-02396 (S.D.N.Y. Feb. 6, 2015). |
| Patent Owner's Preliminary Response, Google Inc. v. Network-1 Technologies, Inc., No. IPR2015-00343 (PTAB Mar. 30, 2015). |
| Turau, Volker, “Fixed-Radius Near Neighbors Search”, Information Processing Letters, Aug. 30, 1991, 201-203, vol. 39. |
| Uitdenbogerd, Alexandra L. et al., “Manipulation of Music for Melody Matching”, Multimedia '98 Proceedings of the sixth ACM international conference on Multimedia, 1998, pp. 235-240. |
| Fuller, Chuck, “Deploying Video on the Web”, Web Techniques, Dec. 1999, 67-71, 4(12), United Business Media LLC, San Francisco. |
| “ImaginOn to Showcase Instant Interactive Internet ‘Television Station in a Box’ at PC EXPO 2000!,” Business Wire, Jun. 22, 2000. |
| Jacso, Peter et al., “Now Featuring . . . Movie Databases, Part II:The Software,” Database, Apr./May 1995, pp. 29-39; 18(2); ProQuest Technology Collection. |
| “Smarter TV to Add $25B in Revenues:Holy Grail of Advertising Coming in the Form of Metadata,” Broadcaster, Aug. 2000; 8(59), Business Information Group, Canada. |
| “The Future of Internet Multimedia on Display at Streaming Media West '99,” Business Wire, Nov. 30, 1999; 11 (27), Business Wire, San Jose, California. |
| Bohm, Christian, et al., “Efficient Similarity Search in Digital Libraries,” IEEE Advances in Digital Libraries, ADL May 22-24, 2000, Washington DC. |
| Hall, Patrick, et al., “Approximate String Matching,” ACM Computing Surveys, Dec. 1980, 12(4). |
| Arya, Sunil, et al., “An Optimal Algorithm for Approximate Nearest Neighbor Searching in Fixed Dimensions,” Journal of the ACM, Nov. 1998, 891-923, 45(6). |
| Berchtold, Stefan, et al., “Using Extended Feature Objects for Partial Similarity Retrieval,” The VLDB Journal, Mar. 28, 1997, 333-348 (6). |
| Schwartz, David M., “ImaginOn Technology and Interactive Television,” Imaginon—White Papers, May 11, 1998, available at http://www.imaginon.com/nn/content/white_papers/i_tv.html (last accessed Sep. 29, 2014). |
| Schwartz, David M. et al., “Internet Television: The Economics of Webcasting,” Imaginon—White Papers, Mar. 14, 2000, available at http://www.imaginon.com/nn/content/white_papers/econ_web.html (last accessed Sep. 29, 2014). |
| “ImaginAuthor—Streaming Video Branching Seamlessly From Clip to Clip,” ImaginOn, Inc., available at http://www.imaginon.com/imon/page5.html (last accessed Sep. 29, 2014). |
| “ImOn.comTV Internet Television,” ImOn.comTV, available at http://www.imaginon.com/imon/page4.html (last accessed Sep. 29, 2014). |
| “ImOn.comTV TurnKey Package,” ImOn.comTV, available at http://www.imaginon.com/imon/page1.html (last accessed Sep. 29, 2014). |
| “ImaginOn presents ImOn.comTV,” ImOn.comTV, available at http://www.imaginon.com/imon/index1.html (last accessed Sep. 29, 2014). |
| Alvear, Jose., “ImOn.comTV to Debut Webcasting Solution,” Streaming Media Magazine, Apr. 10, 2000, available at http://www.streamingmedia.com/Articles/ReadArticle.aspx?ArticleID=62301 (last accessed Sep. 29, 2014). |
| “Press Release—ImOn.comTV Technology Will Change Television Advertising Forever,” ImOn.comTV, available at http://www.imaginon.com/pressrel/p1999/advertising.html (last accessed Sep. 29, 2014). |
| Google Inc. and YouTube, LLC's Corrected Preliminary Invalidity Contentions, Network-1 Technologies, Inc., v. Google, Inc. and YouTube, LLC, No. 1:14-cv-02396-PGG (S.D.N.Y. Sep. 8, 2014). |
| Appendix 1 to Google's Responses to Network-1's Second Interrogatories dated Oct. 20, 2014. |
| Google Inc. and YouTube, LLC'S Responses and Objections to Plaintiff's Second Set of Interrogatories (Nos. 7-13) dated Oct. 20, 2014. |
| Patent Owner's Response Pursuant to 37 C.F.R. § 42.120, Google Inc. v. Network-1 Technologies, Inc., IPR2015-00343 (P.T.A.B. Sep. 18, 2015). |
| Petitioner's Reply to Patent Owner's Response to Petition, Google Inc. v. Network-1 Technologies, Inc., IPR2015-00343 (P.T.A.B. Dec. 14, 2015). |
| Petitioner's Demonstratives, Google Inc. v. Network-1 Technologies, Inc., IPR2015-00345 (P.T.A.B. Mar. 4, 2016). |
| Patent Owner's Demonstratives, Google Inc. v. Network-1 Technologies, Inc., IPR2015-00345 (P.T.A.B. Mar. 4, 2016). |
| Record of Oral Hearing, Google Inc. v. Network-1 Technologies, Inc., IPR2015-00345 (P.T.A.B. Apr. 13, 2016). |
| Patent Owner's Preliminary Response to the Petition for Covered Business Method Review Under 35 U.S.C. § 321 Pursuant to 37 C.F.R. § 42.207, Google Inc. v. Network-1 Technologies, Inc., CBM2015-00113 (P.T.A.B. |
| Patent Owner's Response Pursuant to 37 C.F.R. § 42.220, Google Inc. v. Network-1 Technologies, Inc., CBM2015-00113 (P.T.A.B. Dec. 28, 2015). |
| Petitioner's Reply, Google Inc. v. Network-1 Technologies, Inc., CBM2015-00113 (P.T.A.B. Mar. 1, 2016). |
| Declaration of Dr. George Karypis, Google Inc. v. Network-1 Technologies, Inc., CBM2015-00113 (P.T.A.B. Dec. 28, 2015). |
| Transcript of Deposition of George Karypis, Ph.D., Google Inc. v. Network-1 Technologies, Inc., CBM2015-00113 (P.T.A.B. Feb. 17, 2016). |
| Transcript of Deposition of Pierre Moulin, Ph.D., Google Inc. v. Network-1 Technologies, Inc., CBM2015-00113 (P.T.A.B. Dec. 28, 2015). |
| Petitioner's Demonstratives, Google Inc. v. Network-1 Technologies, Inc., CBM2015-00113 (P.T.A.B. May 6, 2016). |
| Patent Owner's Demonstratives, Google Inc. v. Network-1 Technologies, Inc., CBM2015-00113 (P.T.A.B. May 6, 2016). |
| Computer Desktop Encyclopedia, 9th Ed., Osbourne/McGraw-Hill 2001, p. 470. |
| GCSE Bitesize Science, Satellite communication: Revision, BBC. |
| Graf, Modern Dictionary of Electronics, Seventh Ed. Revised and Updated, Newnes 1999, p. 359. |
| Kernighan et al., The C Programming Language, Second Ed., Prentice Hall PTR 1988, pp. vi-xii, 1-15, 36-39. |
| Lindorff, In the clutter race, cable is catching up with broadcast, Media Life Magazine, available at http://www.medialifemagazine/com:8080/news2000/apr00/news20403.html. |
| Margolis, Random House Webster's Computer & Internet Dictionary, Third Ed., Random House, Inc. 1999, p. 268. |
| MediaDailyNews, Universe Collapses: Well, TV's, Anyway (Jul. 1, 2004). |
| Microsoft Computer Dictionary, Fifth Ed., Microsoft Press 2002, p. 264. |
| Microsoft Computer Dictionary, Fifth Ed., Microsoft Press 2002, p. 276. |
| Microsoft Computer Dictionary, Fifth Ed., Microsoft Press 2002, p. 338. |
| Microsoft Computer Dictionary, Fourth Ed., Microsoft Press 1999, p. 228. |
| The New Oxford American Dictionary, Oxford University Press 2001, pp. 1727-1728. |
| New Oxford American Dictionary, Third Ed., Oxford University Press 2010, pp. 1767-1768. |
| Pountain, The Penguin Concise Dictionary of Computing, Penguin Books 2003, p. 221. |
| Quizlet, Facebook Terms AA flashcards, available at https://quizlet.com/76248870/facebook-terms-aa-flash-cards/. |
| The IEEE Standard Dictionary of Electrical and Electronics Terms, Sixth Ed., p. 499. |
| Wikipedia, Identifier, Internet Archive Jan. 11, 2004, available at https://web.archive.org/web/20040111093522/http://en.wikipedia.org/wiki/Identifier. |
| Wikipedia, Name and Identifier, Internet Archive Apr. 6, 2005, available at https://web.archive.org/web/20050406004314/http://en.wikipedia.org/wiki/Identifier. |
| Final Written Decision, Google Inc. v. Network-1 Technologies, Inc., IPR2015-00343 (P.T.A.B. Jun. 20, 2016). |
| Final Written Decision, Google Inc. v. Network-1 Technologies, Inc., IPR2015-00345 (P.T.A.B. Jun. 20, 2016). |
| Final Written Decision, Google Inc. v. Network-1 Technologies, Inc., IPR2015-00347 (P.T.A.B. Jun. 20, 2016). |
| Final Written Decision, Google Inc. v. Network-1 Technologies, Inc., IPR2015-00348 (P.T.A.B. Jun. 20, 2016). |
| Google Inc. v. Network-1 Technologies, Inc., CBM2015-00113, Paper 35, Termination Decision Document (PTAB Oct. 18, 2016). |
| Peter N. Yianlos, Excluded Middle Vantage Point Forests for Nearest Neighbor Search, Jul. 20, 1998, pp. 1-12. |
| Peter N. Yianlos “Locally Lifting the Curse of Dimensionality for Nearest Neighbor Search” SODA 2000, pp. 361-370. |
| L. Baum et al., “A Maximization Technique Occuring in the Statistical Analysis of Probabilistic Functions of Markov Chains,” The Annals of Mathematical Statistics, vol. 41, No. 1, pp. 164-171 (1970). |
| A. P. Dempster et al., “Maximum Likelihood from Incomplete Data via the $EM$ Algorithm,” Journal of the Royal Statistical Society, Series B (Methodological), vol. 39, Issue 1, pp. 1-38 (1977). |
| D. Reynolds et al., “Robust Text-Independent Speaker Identification Using Gaussian Mixture Speaker Models,” IEEE Transactions on Speech and Audio Processing, vol. 3, No. 1, pp. 72-83 (Jan. 1995). |
| A. Djouadi and E. Bouktache, “A fast Algorithm for the nearest neighbor classifier,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Mar. 1997, pp. 277-282. |
| Nene et al., “A simple algorithm for nearest neighbor search in high dimensions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Sep. 1997, pp. 989-1003. |
| Sunil Arya et al., “Approximate Nearest Neighbor Queries in Fixed Dimensions,” Proceedings of the 4th annual ACM-SIAM Symposium on Discrete algorithms, 1993, pp. 271-280. |
| K. Fukunaga et al., A branch and bound algorithm for computing k-nearest neighbors, IEEE Trans. Comput., C24:750-753, Jul. 1975. |
| Charles D. Feustel et al., “The nearest neighbor problem in an abstract metric space,” Pattern Recognition Letters, pp. 125-128, Dec. 1982. |
| Dennis Shasha et al., “New Techniques for Best-Match Retrieval,” ACM Transactions on Information Systems, 8(2); 140{158, Apr. 1990. |
| J. Uhlmann, “Satisfying general proximity / similarity queries with metric trees”, Information Processing Letters, 40 (4): 175{9, Nov. 1991. |
| Sergey Brin, “Near Neighbor Search in Large Metric Spaces,” Proceedings of the 21st VLDB Conference, Zurich, Switzerland, Sep. 1995. |
| Daniel P. Huttenlocher, et al., “Comparing Images Using the Hausdorff Distance,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 15, No. 9, pp. 850-863, Sep. 1993. |
| Thomas Seidl et al., “Optimal Multi-Step K-Nearest Neighbor Search,” Proceedings of ACM SIGMOD International Conference of Management of Data, Jun. 1998, pp. 154-165. |
| W. A. Burkhard et al., “Some Approaches to Best-Match File Searching,” Communications of the ACM, vol. 16, No. 4, Apr. 1973. |
| Eyal Kushilevitz et al., “Efficient Search for Approximate Nearest Neighbor in High Dimensional Spaces,” Proceedings of the 30th annual ACM Symposium on the Theory of computing, 1998, pp. 457-474, vol. 30, No. 2. |
| J. Nievergelt et al., “The Grid File: An Adaptable, Symmetric Multikey File Structure,” ACM Transactions on Database Systems, vol. 9, No. 1, pp. 38-71 (Mar. 1984). |
| Nevin Heintze, “Scalable Document Fingerprinting,” Proc. USENIX Workshop on Electronic Commerce (1996). |
| Erling Wold et al., “Content-Based Classification, Search, and Retrieval of Audio,” IEEE Multimedia, vol. 3, Issue 3, pp. 27-63 (1996). |
| Sir Bhanu et al., “Learning Feature Relevance and Similarity Metrics in Image Databases,” Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries, pp. 14-19 (1998). |
| A. Del Bimbo et al., “Using Weighted Spatial Relationships in Retrieval by Visual Contents,” Image Description and Retrieval, pp. 161-192 (1998). |
| P. Indyk et al., “Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality,” Proceeding of the Thirtieth Annual ACM Symposium on Theory of Computing, pp. 604-613 Jul. 21, 1999. |
| Marco La Cascia, “Combining Textual and Visual Cues for Content-based Image Retrieval on the World Wide Web,” Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries, pp. 24-29 (1998). |
| Atsuo Yoshitaka et al., “A Survey on Content-Based Retrieval for Multimedia Databases,” IEEE Transactions on Knowledge and Data Engineering, vol. 11, No. 1, pp. 81-93 (Jan./Feb. 1999). |
| Steve Lawrence et al., “Digital Libraries and Automonous Citation Indexing,” IEEE Computer, pp. 67-71 (Jun. 1999). |
| Akisato Kimura et al., “Very Quick Audio Searching: Introducing Global Pruning to the Time-Series Active Search,” IEEE Conf on Acoustics, Speech and Signal Processing, (ICASSP '01), vol. 3, pp. 1429-1432, 2001. |
| Edgar Chavez et al., “Searching in Metric Spaces,” ACM Computing Surveys, vol. 33, No. 3, pp. 273-321 (Sep. 2001). |
| Jaap Haitsma et al., “Robust Audio Hashing for Content Identification,” Workshop on Content Based Multimedia Indexing, Brescia, Italy (Sep. 19-21, 2001). |
| Jaap Haitsma et al., “A Highly Robust Audio Fingerprinting System,” Journal of New Music Research, 1744-5027, vol. 32, Issue 2, pp. 211-221 (2003). |
| Saul Schleimer et al., “Winnowing: Local Algorithms for Document Fingerprinting,” ACM SIGMOD, Jun. 9-12, 2003. |
| Edward Chang et al., “Searching Near-Replicas of Images via Clustering,” SPIE Symposium of Voice, Video and Data Communications, 1999. |
| Edward Y. Chang et al., “RIME: A Replicated Image Detector for the World-Wide Web,” SPIE, 1998. |
| Hector Garcia-Molina et al., “Safeguarding and Charging for Information on the Internet,” Proceedings of ICDE, 1998. |
| Sergey Brin et al., “Copy Detection Mechanisms for Digital Documents,” Proceedings of ACM SIG-MOD, May 1995. |
| Stefan Berchtold “The x-tree: An Index Structure for High-Dimensional Data,” Proceedings of the 22nd VLDB, Aug. 1996. |
| Norio Katayama et al., “The SR-tree: An Index Structure for High-Dimensional Nearest Neighbor Queries,” Proceedings of ACM SIGMOD, May 1997. |
| John T. Robinson, “The K-D-B-Tree: A Search Structure for Large Multidimensional Dynamic Indexes,” Proceedings of ACM SIGMOD, Apr. 1981. |
| Myron Flickner et al., “Query by Image and Video Content: The QBIC System,” IEEE Computer 28(9), pp. 23-32, 1995. |
| Amarnath Gupta et al., “Visual Information Retrieval,” Communications of the ACM, vol. 40, No. 5, pp. 69-79, May 1997. |
| John R. Smith et al., “VisualSEEk: A fully automated content-based image query system,” ACM Multimedia Conference, 1996. |
| David A. White et al., “Similarity Indexing: Algorithms and Performance,” Proc. SPIE, vol. 2670, San Diego, 1996. |
| Norbert Beckmann et al., “The R*-tree: An Efficient and Robust Access Method for Points and Rectangles,” Proceedings of ACM Sigmod, May 1990. |
| A. Guttman, “R-Trees: A Dynamic Index Structure for Spatial Searching,” Proceedings of ACM Sigmod, Jun. 1984. |
| David A. White et al., “Similarity Indexing with the SS-tree*,” Proceedings of the 12th ICDE, Feb. 1996. |
| King Lin et al., “The TV-Tree: An Index Structure for High-Dimensional Data,” VLDB, Journal 3, No. 4, 1994, pp. 517-542. |
| Paolo Ciaccia et al., “M-tree: An Efficient Access Method for Similarity Search in Metric Spaces,” Proceedings of the 23rd VLDB, Aug. 1997. |
| Nick Roussopoulos et al., Nearest Neighbor Queries, Proceedings of ACM Sigmod, May 1995. |
| C. Li et al., “An extensible hashing index for high-dimensional similarity search,” Stanford Techncial Report, Aug. 1998. |
| Jon M. Kleinberg, “Two Algorithms for Nearest-Neighbor Search in High Dimensions,” Proc 29th STOC, Feb. 7, 1997. |
| Opinion and Judgment entered (Affirmed), Google LLC v. Network-1 Technologies, Inc., Case No. 17-1379 in the U.S. Court of Appeals for the Federal Circuit, Doc. 45-2, Jan. 23, 2018, 2 pages. |
| Opening Brief of Appellant Google Inc., Google Inc. v. Network-1 Technologies, Inc., Case No. 16-2509 in the U.S. Court of Appeals for the Federal Circuit, Doc. 22, Nov. 28, 2016, 66 pages. |
| Opening Brief of Appellant Google Inc., Google Inc. v. Network-1 Technologies, Inc., Case No. 17-1379 in the U.S. Court of Appeals for the Federal Circuit, Doc. 16, Apr. 14, 2017, 112 pages. |
| Brief for Appellee Network-1 Technologies, Inc., Google Inc. v. Network-1 Technologies, Inc., Case No. 17-1379 in the U.S. Court of Appeals for the Federal Circuit, Doc. 24, Jun. 16, 2017, 72 pages. |
| Brief of Cross-Appellant and Appellee Network-1 Technologies, Inc., Google Inc. v. Network-1 Technologies, Inc., Case No. 16-2509 in the U.S. Court of Appeals for the Federal Circuit, Doc. 30, Feb. 8, 2017, 73 pages. |
| Reply Brief of Appellant Google Inc., Google Inc. v. Network-1 Technologies, Inc., Case No. 16-2509 in the U.S. Court of Appeals for the Federal Circuit, Doc. 36, Mar. 22, 2017, 34 pages. |
| Reply Brief for Appellant Google Inc., Google Inc. v. Network-1 Technologies, Inc., Case No. 17-1379 in the U.S. Court of Appeals for the Federal Circuit, Doc. 27, Jun. 30, 2017, 34 pages. |
| Corrected Opening Brief of Appellant Google Inc., Google Inc. v. Network-1 Technologies, Inc., Case No. 16-2509 in the U.S. Court of Appeals for the Federal Circuit, Doc. 59, Nov. 16, 2017, 277 pages. |
| Liu et al., “An Investigation of Practical Approximate Nearest Neighbor Algorithms”, Advances in Neural Information Processing Systems (NIPS) 2004. |
| K.L. Clarkson, “Nearest-Neighbor Searching and Metric Space Dimensions”, Nearest-Neighbor Methods for Learning and Vision: Theory and Practice:, Apr. 2005. |
| Swaminathan et al., “Robust an Secure Image Hashing”, IEEE Transactions on Information Forensics and Security, Jun. 2006, pp. 215-230, vol. 1, No. 2. |
| Burges et al., “Duplicate Detection and Audio Thumbnails with Audio Fingerprinting” [online]. 2004, [retrieved on Nov. 21, 2006]. Retrieved on the Internet: <URL: www.research.microsoft.coml-cburges/tech_reports/tr-2004-19.pdf>, 5 pages. |
| Cano et al., “A Review of Algorithms for Audio Fingerprinting” [online]. 2002, [retrieved on Nov. 21, 2006]. Retrieved from the Internet: <URL: www.iva.upf.es/mtg/publications/MMSP-2002-pcano.pdf>, 5 pages. |
| Haitsma and Kalker, “A Highly Robust Audio Fingerprinting System” [online]. 2002, [retrieved on Nov. 16, 2006]. Retrieved from the Internet: <url: www.ismir2002.ismir.netlproceedings/02-FP04-2.pdf>, 9 pages. |
| Jacobs et al., “Fast Multiresolution Image Querying” [online]. 1995, [retrieved on Nov. 21, 2006]. Retrieved from the Internet: <URL: www.grail.cs.washington.edu/projects/query.pdf>, 10 pages. |
| Ke et al., “Computer Vision for Music Identification” [online]. 2005, [retrieved on Nov. 21, 2006]. Retrieved from the Internet: <URL: www.cs.cmu.edu/-yke/musicretrieval/cvpr2005-rnr.pdf>, 8 pages. |
| Shazam, Shazam Entertainment Brings Music Recognition to Windows Mobile 5.0 Powered Smartphones [online]. 2006. [retrieved on Nov. 16, 2006J. Retrieved from the Internet: <URL: www.shazam.com/musicIportallsp/s/media-type/html/user/anon/page/defaullitemplate/pages/p/companLrelease30.html>. 1 page. |
| Stanford, “CS276 Information Retrieval and Web Mining” [online]. 2005, [retrieved on Nov. 16, 2006]. Retrieved from the Internet: <URL: www.stanford.edu/class/cs276/handouts/lecture19.pdf>, 8 pages. |
| Stanford, “Data Mining: Associations” [online]. 2002, [retrieved on Nov. 16, 2006]. Retrieved from the Internet: <URL: www.stanford.edu/class/cs206/cs206-2.pdf>, 11 pages. |
| Stollnitz et al., Wavelets for Computer Graphics: A Primer, Part 1: [online]. 1995, [retrieved on Nov. 21, 2006]. Retrieved from the Internet: <URL: www.graiLcs.washington.edu/pub/stolllwaveletl.pdf>, 8 pages. |
| Stollnitz et al., Wavelets for Computer Graphics: A Primer, Part 2: [online]. 1995, [retrieved on Nov. 21, 2006]. Retrieved from the Internet: <URL: www.graiLcs.washington.edu/pub/stolllwavelet2.pdf>, 9 pages. |
| Yang, “MACS: Music Audio Characteristic Sequence Indexing for Similarity Retrieval”, Oct. 21-24, 2001, New Paltz, New York. |
| Viola and Jones, Robust Real-Time Object Detection, Int. J. Computer Vision, 2002. |
| Burges et al., “Using Audio Fingerprinting for Duplicate Detection and Tthumbnail Generation, ” Mar. 2005, 4 pages. |
| Lin, et al., Input Data Representation for Self-Organizing Map in Software Classification, Knowledge Acquisition and Modeling, 2009. KAM '09. Second International Symposium on Volume: 2, Digital Object Identified: 10.11 09/KAM.2009.151 Publication Year: 2009, Page(s): 360-353. |
| Baluja et al., “Content Fingerprinting Using Wavelets”, 3rd European Conference on Visual Media Production, 2006, pp. 198-207. |
| Cohen et al., “Finding Interesting Associations without Support Pruning”, IEEE Transactions on Knowledge and Data Engineering, 2001, pp. 64-78, vol. 13, Issue 1. |
| Yang, Efficient Video Identification based on locality sensitive hashing and triangle inequality, National University of Singapore, 2005, pp. 1-64. |
| U.S. Appl. No. 60/304,647, filed Jul. 10, 2001. |
| U.S. Appl. No. 60/281,881, filed Apr. 5, 2001. |
| Sakata et al., Metadata Mediation: Representation and Protocol, Digital Vision Laboratories, Jul. 3, 1937 Akasaka, Minato, Tokyo, Japan. |
| U.S. Appl. No. 13/800,573, filed Mar. 13, 2013. |
| U.S. Appl. No. 13/800,890, filed Mar. 13, 2013. |
| U.S. Appl. No. 13/829,717, filed Mar. 14, 2013. |
| U.S. Appl. No. 13/830,447, filed Mar. 14, 2013. |
| U.S. Appl. No. 13/830,626, filed Mar. 14, 2013. |
| U.S. Appl. No. 13/830,986, filed Mar. 14, 2013. |
| U.S. Appl. No. 13/842,068, filed Mar. 15, 2013. |
| U.S. Appl. No. 13/338,079, filed Dec. 27, 2011. |
| U.S. Appl. No. 60/134,782, filed May 19, 1999. |
| U.S. Appl. No. 60/193,948, filed Mar. 31, 2000. |
| U.S. Appl. No. 60/195,535, filed Apr. 7, 2000. |
| U.S. Appl. No. 60/206,384, filed May 23, 2000. |
| U.S. Appl. No. 60/221,843, filed Jul. 28, 2000. |
| Fingerprint Identification Technology in Civil Applications, Sagem Morpho, Inc. (published 1998) Internet Archive (Aug. 4, 2001), http://web.archive.org/web/20010804045334/http://www.morpho.com/news_room/library/whitepapers/civil_afis_htm. |
| Google's Third Supplemental Interrogatory Responses, Nos. 2, 6-7, 9-11 and 13, Network-1 Technologies, v. Google, Inc . et al., No. 1:14-cv-02396 (S.D.N.Y. May 14, 2015). |
| Decision Instituting Inter Partes Review, Google Inc. v. Network-1 Technologies, Inc., No. IPR2015-00343 (PTAB Jun. 23, 2015), re U.S. Pat. No. 8,640,179. |
| Google Inc. and YouTube, LLC's Preliminary Invalidity Contentions, Network-1 Technologies, Inc. v. Google, Inc. and YouTube, LLC, No. 1:14-cv-09558 (S.D.N.Y. Jun. 18, 2015). |
| Decision Instituting Inter Partes Review, Google Inc. v. Network-1 Technologies, Inc., No. IPR2015-00345 (PTAB Jun. 23, 2015), re U.S. Pat. No. 8,205,237. |
| Decision Instituting Inter Partes Review, Google Inc. v. Network-1 Technologies, Inc., No. IPR2015-00347 (PTAB Jun. 23, 2015), re U.S. Pat. No. 8,010,988. |
| Decision Instituting Inter Partes Review, Google Inc. v. Network-1 Technologies, Inc., No. IPR2015-00348 (PTAB Jun. 23, 2015), re U.S. Pat. No. 8,656,441. |
| Transcript of Deposition of Pierre Moulin, PhD, vol. I (Aug. 19, 2015), Google Inc. v. Network-1 Technologies, Inc., No. IPR2015-00347 (PTAB filed Dec. 3, 2014), re U.S. Pat. No. 8,010,988. |
| Transcript of Deposition of Pierre Moulin, PhD, vol. II (Aug. 20, 2015), Google Inc. v. Network-1 Technologies, Inc., No. IPR2015-00347 (PTAB filed Dec. 3, 2014), re U.S. Pat. No. 8,010,988. |
| Declaration of Dr. George Karypis, Google Inc. v. Network-1 Technologies, Inc., No. IPR2015-00343 (PTAB Jun. 23, 2015), re U.S. Pat. No. 8,640,179. |
| Rudolf F. Graf, Modern Dictionary of Electronics, 425-426 (7th ed. 1999). |
| Nearest Neighbor Search, Wikipedia, available at http://en.wikipedia.org/wiki/Nearest_neighbor_search (last accessed Sep. 10, 2015). |
| Big O notation, Wikipedia, available at http://en.wikipedia.org/wiki/Big_O_notation (last accessed Sep. 15, 2015). |
| U.S. Appl. No. 60/147,029. |
| “Apple Unveils New QuickTime Version,” AP (Associated Press) Online, Apr. 20, 1999. |
| “Form-based Device Input and Upload in HTML,” W3C NOTE, Jul. 6, 1999, available at http://www.w3.org/1999/07/NOTE-device-upload -19990706. |
| “Novel Commercials in video Debut,” Broadcasting, p. 10, Jul. 7, 1941, available at http://www.americanradiohistory.com/Archive-BC/BC-1941/1941-07-07-BC.pdf. |
| “VDO expands Webcasting possibilities”, Broadcasting & Cable, Nov. 11, 1996. |
| Alex Bracetti, “The 80 Best Gadgets of the '80s,” Complex Media, Aug. 19, 2013, available at http://www.complex.com/pop-culture/2013/08/best-gadgets-of-the-80s/. |
| Bill Carter, “ABC's ‘Home Videos’ Pays Off Big,” NY Times, Feb. 19, 1990, available at http://www.nytimes.com/1990/02/19/business/abc-s-home-videos-pays-off-big.html. |
| Bruce and Marge Brown, “POPcast, Post and view audio on the Web—free,” PC Magazine, Dec. 29, 1999. |
| Business Wire, “Ivex Announces ViewOps Internet Video Service; Allows Business Managers to View and Manage Operations Online,” May 2, 2000. |
| E. Nebel & L. Masinter, “Form-based File Upload in HTML,” Nov. 1995, available at http://www.ietf.org/rfc/rfc1867.txt. |
| Early Television Museum, “Early Television Stations”, available at http://www.earlytelevision.org/w6xao.html. |
| Jose Alvear, “Web Developer.com Guide to Streaming Multimedia”, Chapters 9 and 11, 1998. |
| Leland, Jon, “Online Video Service Shortcuts,” Videography, Mar. 1999, 24, 3 (ABI/INFORM Professional Advanced), pp. 150-154. |
| RealServer Administration Guide, Version 5.0, 1997. |
| Trends, Digital Video, “Hollywood Goes to the Web,” PC Magazine, Jun. 22, 1999. |
| Videomaker.com, “Laundromat, Fotomat . . . Streaming Videomat?,” Apr. 15, 1999. |
| Network-1's Combined Petition for Panel Rehearing or Rehearing En Banc, Google LLC v. Network-1 Technologies, Inc., Case No. 16-2509 in the U.S. Court of Appeals for the Federal Circuit, Doc. 77, May 9, 2018, 43 pages. |
| Order, Google LLC v. Network-1 Technologies, Inc., Case No. 16-2509 in the U.S. Court of Appeals for the Federal Circuit, Doc. 94, Jul. 10, 2018, 2 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 60232618 | Sep 2000 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15365403 | Nov 2016 | US |
| Child | 15785171 | US | |
| Parent | 14981269 | Dec 2015 | US |
| Child | 15365403 | US | |
| Parent | 13842068 | Mar 2013 | US |
| Child | 14981269 | US | |
| Parent | 13800573 | Mar 2013 | US |
| Child | 13842068 | US | |
| Parent | 13338079 | Dec 2011 | US |
| Child | 13800573 | US | |
| Parent | 11977202 | Oct 2007 | US |
| Child | 13338079 | US | |
| Parent | 11445928 | Jun 2006 | US |
| Child | 11977202 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 09950972 | Sep 2001 | US |
| Child | 11445928 | US |
