Patent Application
20140032561
The described technology relates to the field of multimedia search techniques.
Searching refers to a process in which a user submits a query, such as a list of keywords, and receives in return a search result. The search result is a set of one or more resources in a search domain that are determined to be responsive to the query by a search algorithm. It is typical for queries to be processed against a search index that corresponds to a desired search domain. For example, a query may be applied against a search index corresponding to a search domain of web documents, such as web pages or other textual documents available via the Internet, to produce a result containing a list of links to web documents satisfying the query. A query may similarly be applied against a search index corresponding to a multimedia search domain, to produce a result containing a list of links to multimedia resources satisfying the query. For example, the search query “Santana” applied against a multimedia search index could identify a link to a multimedia resource corresponding to the song “Smooth” by the band Santana.
The conventional approach to indexing a search domain involves populating the search index using web crawling techniques. Such web crawling techniques typically involve automatically examining all or portions of the World Wide Web in a methodical manner to identify resources, and queuing information about visited resources for later processing to add these resources to an index. Depending on the scope of the crawl, such techniques can involve long update cycles—for example, the web crawl might run every three days and therefore the index would only be updated every three days. Because of the time required to complete the crawl, this method of indexing is better suited for indexing some types of resources over others. For example, this method of indexing is generally quite suitable for identifying multimedia resources that are available for relatively long periods of time. An example of a multimedia resource that may be accessible for an extended period is a link to a Santana song on Santana's homepage (santana.com). As long as the band Santana continues to enjoy consumer success, it is likely that a user will be able to select and play a Santana audio or video clip from the website at any time during the future (whether weeks, months, or years).
In contrast to such “static” multimedia resources that are available for long periods of time, some publishers make available transient multimedia resources as part of a data stream. For example, a publisher may offer programming from a radio station over the Internet as a digitized stream so that listeners do not have to be in geographic proximity to a broadcast tower in order to enjoy the station programming. When provided over the Internet as an audio stream, the programming is typically “real-time” in the sense that the station does not store or archive the transmitted content in a manner that can be easily accessed by users in the future. Instead, users are only able to receive and play the content that is presently being broadcast by the station. That is, a user accessing a station via the Internet halfway through the broadcast of a song is usually precluded from “rewinding” or otherwise listening to the song from the start. Instead, the user is only able to listen to the remaining portion of the song that hasn't been broadcast. In this sense, all of the multimedia resources contained in a real-time data stream—whether they are songs, interviews, traffic reports, weather updates, or any other content in audio, video, audio/video, and/or other form—may be considered to be “transient” since they are only accessible via the Internet for an instant of time. Unless stored by the user, each multimedia resource can only be accessed for the period that the resource is broadcast. Of course, real-time multimedia data streams vary significantly between publishers and may be produced in any of a variety of media types, in various encoding formats, and at various quality levels. Some multimedia resources broadcast in a real-time data stream may be “live” in the sense that the media content is being created contemporaneously with its broadcast, while other multimedia content may be preexisting content that is presented in the real-time data stream.
Because transient multimedia resources are only accessible for a brief instant of time, conventional indexing techniques used to identify static multimedia resources are not suitable for transient multimedia resources for at least two reasons. First, conventional indexing techniques that identify a real-time multimedia data stream are often only able to index the data stream based on general metadata associated with that stream. For example, a stream might be identified as “swing music from the 40s,” but the indexing would not be able to identify particular artists or song titles that are included in the stream. As a result, users searching for a particular artist may not know about the inclusion of the artist in a particular data stream because the metadata associated with the stream does not include such information. Second, even if the indexing technique was able to identify a particular artist that was being played on the stream at the time of indexing, because of the delay associated with most crawls such an index would quickly become out of date. Users searching for a particular artist would be unable to find the artist in a particular data stream because all or some of the results may be stale (meaning that a user selecting a search result and being redirected to a stream would not find the artist they were seeking in the stream) or no results may be returned even though the artist should be found in a stream if the stream were being indexed in a timely fashion. In each case, the user is unable to locate a desired result responsive to their search query.
In view of the above-discussed disadvantages of conventional approaches, a more effective approach to indexing transient multimedia resources would have substantial utility.
A software and/or hardware facility is described that enables searching for transient multimedia resources, either alone or in conjunction with searching for static multimedia resources. Transient multimedia resources are those audio, video, audio/video or other media resources that are typically delivered via a real-time data stream and accessible via a network for only a short period of time. Because transient multimedia resources are only available for brief periods, the facility uses special techniques to identify the transient multimedia resources, update a database of the transient resources, and quickly map the database to a search index for transient multimedia resources that may be accessed by users or other applications. Static multimedia resources are those audio, video, audio/video or other media resources that are available for access via a network for an extended period of time. In some embodiments, the facility also identifies static multimedia resources, stores them in a database, and indexes the static resources, but the timeframe for doing so may be longer because of the longevity of these resources. By maintaining the currency of the transient and static indices, end users are able to search for and accurately identify those media resources that are available at the time the search inquiry is made.
In some embodiments, the facility identifies a number of real-time multimedia streams that contain transient multimedia resources. Each real-time multimedia stream is analyzed to determine the characteristics of the stream, including, but not limited to: (i) the format of the stream (e.g., RealAudio®, Quicktime®, Windows® Media, MP3, Macromedia Flash®, etc.); (ii) the data rate of the stream (e.g., 48K, 128K, 256K, 750K, etc.); and (iii) any metadata associated with the stream that may characterize the stream (e.g., information related to the content of the stream, information related to the intellectual property embedded in the stream, information related to the technical aspects of the stream, etc.). Once a real-time multimedia stream has been accurately characterized, a stream listener may be configured to automatically monitor the stream and identify transient multimedia resources that are contained in the stream. Depending on the format of the stream, such listeners may monitor the stream continuously or may be able to monitor the stream on a periodic basis. In certain situations, not all identified real-time multimedia streams may need to be monitored. For example, sometimes data streams containing the same multimedia resources are available at different data rates, e.g., a low-bandwidth stream at 48K for dial-up users and higher bandwidth streams at 128K and 256K for broadband users, or in different formats, e.g., RealAudio and Quicktime. In such situations, only one of the data streams needs to be monitored to identify transient multimedia resources. Once identified, the information about the resources may be replicated for the other multimedia data streams having different rates or formats.
In some embodiments, after a stream listener has been configured for a real-time multimedia stream it may begin to monitor the stream in order to identify the transient multimedia resources that are contained in the stream. It will be appreciated that the content and length of transient resources in a stream may vary considerably from stream-to-stream. For example, an online radio may produce an audio multimedia stream with frequent transitions between songs, commercial breaks, live interviews, and news segments. The identity of the transient audio resource in the online radio stream may therefore change every few minutes. In contrast, an online video stream may contain transient video resources that last for an hour or more. The stream listeners are designed to monitor the multimedia stream to identify the transient resource that is currently being streamed, but more importantly to look for transitions in the stream that indicate the end of one media resource and the start of the next media resource. It is beneficial to identify transitions, since the sooner a new resource can be added to a database and indexed, the faster the resource can be located. When a transition is detected, the listener identifies information about the new transient resource. The facility may obtain such information from a variety of in-band and out-of-band sources. For example, the facility may extract embedded metadata from the data streams that identifies the content of the stream. Such metadata may be interspersed with or embedded in the resource, for example, in a header packet or in periodic packets containing metadata information that are in addition to or combined with packets containing the transient resource. Alternatively, some media publishers may offer a schedule on a web page that reflects the current content playing in a real-time multimedia steam, may offer an RSS or other feed to indicate the content in a real-time stream, or may offer a second metadata-only stream that is associated with a real-time stream. Regardless of the source of the information, when a listener detects that a new transient resource is contained in the stream it updates a transient multimedia database to reflect the current resource that is available if the stream were to be accessed. Note that the database record containing the immediately preceding resource in the stream is typically deleted or overwritten, as a user accessing the stream would no longer be able to access the preceding resource. A transient multimedia index is then quickly updated to map search terms to the transient multimedia database and enable searches to locate transient resources that are responsive to a search query.
In some embodiments, the facility maintains a pair of indices used to satisfy queries: a static multimedia index containing information about static media resources and a transient multimedia index containing information about transient media resources. The static multimedia index is updated using conventional techniques. In contrast, the transient multimedia index is typically updated on a frequent basis so that the index accurately reflects the media resources that are currently being streamed in a number of real-time multimedia streams. When a search query is received, the facility may perform a search of both indices and return static media results and transient media results that are responsive to the query. By providing multimedia search capability as described above, the facility provides an enhanced search experience by opening up a broader range of media sources that match a user's interests.
The multimedia resources may initially be found using a variety of techniques. One technique for identifying the multimedia resources is to use web crawlers 130 to crawl World Wide Web pages or other search domains 140. Another is to rely on human editors to search for and identify such resources. Still another technique is for publishers of multimedia resources to notify or otherwise register their multimedia resources with the facility. For example, a technique for locating multimedia resources is described in U.S. patent application Ser. No. 10/432,258 entitled “System and Process for Network Site Fragmented Search” (filed 21 May 2003) which is hereby incorporated by reference in its entirety. It will be appreciated that the technique use to locate multimedia resource may depend on the size of the search domain, the type of resources being located (e.g. static versus transient) or on other factors.
Media resources that are identified as part of a crawl or other technique are analyzed by characterization module 150 which sorts the identified resources into one of two types—static or transient. Certain multimedia resources that may be found are available for extended periods of time. That is, a user or other application may be able to access the identified resource now or in the future. Typically such resources are made available by a publisher until a time that the publisher determines that the resource should no longer be made available. Resources that have no known temporal restriction on when they can be accessed are characterized as “static” multimedia resources. The characterization module 150 stores a pointer to each identified static resource in the static multimedia database 155. In addition to a pointer to the resource, the characterization module stores other metadata that it identifies as being associated with the resource, and which may subsequently be useful to the facility to determine when a static resource is responsive to a user's search query. Such metadata may include, but not be limited to, information related to the content of the multimedia resource, such as the title of the resource, the author, the genre, reviews about the resource, or any other identifying information which would help characterize the content of the static multimedia resource. The metadata may also include any technical characteristics about the stored resource, such as the format of the resource or the data rate of the resource.
After characterizing information about each static multimedia resource is stored in the static multimedia database 155, a static multimedia index 120 is created. The static multimedia index is a searchable index that correlates search queries with records in the static multimedia database. Static multimedia index 120 may be created using any of a variety of techniques that are known in the art, such as those disclosed in U.S. patent application Ser. No. 10/432,306 entitled “System and Process for Searching a Network” (filed 21 May 2003) which is hereby incorporated by reference in its entirety. The static multimedia database 155 may be periodically modified or supplemented if better information about each resource becomes available, and static multimedia index 120 may be periodically rebuilt to optimize searching of the database 155.
In contrast to static multimedia resources which are always available, the characterization module 150 may also identify a number of real-time multimedia streams that contain transient multimedia resources. As described above, real-time multimedia streams are those streams that may be characterized as a “live” broadcast, meaning that the producer of the stream typically does not store or allow users to easily access historical portions of the stream. When the characterization module 150 identifies a real-time multimedia stream 155a-155n, it stores a pointer to the location of that stream in real-time multimedia stream database 160. Depending upon the size of the resource domain 140 that is being crawled by the web crawler 130, the number of static multimedia resources identified by the characterization module 150 typically exceeds the number of real-time multimedia streams 160 that the characterization module identifies.
In an embodiment, on a periodic basis a real-time multimedia stream analysis module 170 analyzes the format and content of each multimedia stream identified in the multimedia stream database 160. Specifically, for each stream identified in the database, multimedia stream analysis module 170 analyzes the stream to determine the characteristics of the stream including, but not limited to, the format of the stream (e.g., Real Audio, Quicktime, Windows Media, MP3, Macromedia Flash, etc.), the data rate of the stream (e.g., 48 k, 128 k, 256 k, 750 k, etc.), and any metadata associated with the stream that may characterize the stream. The metadata associated with the stream may include items such as a description of the genre of the stream (e.g., pop radio, news, classic videos, educational programming, etc.), information about the content producer and distributor (e.g., the New York Times, Real Networks, etc.), and any other information that may characterize the general format and delivery of the stream (e.g., this stream operates from 9 a.m. to 9 p.m., etc.). Metadata may include “hard” metadata that is inherent in the broadcast or distribution and includes the format of the stream, the data rate, the length of the stream or size of the associated file, the video resolution (height, width, and color depth), the sound characteristics (stereo or mono), etc., and “soft” metadata that is associated with the content of the stream and includes, title, performer, copyright, genre, topic, etc. Once the analysis module 170 has completed the characterization of multimedia streams in the database, the analysis module may configure one or more stream listeners 180 to monitor the streams.
It will be appreciated that stream listeners 180 can be configured in a variety of different ways to monitor real-time multimedia streams 155a-155n. The purpose of each stream listener 180 is to monitor a multimedia stream and identify any transient resource that is contained within the stream. Once a transient resource is identified in a multimedia stream, stream listener 180 updates a transient multimedia database 185 with the identity of the transient resource. As noted above, many real-time data streams will include an in-band or an out-of-band signal that provides metadata about the particular transient resource that is contained in the stream. Some media formats specify that a header be transmitted prior to any streaming content, the header containing metadata information about the streaming content that is to follow. For example, Windows Media format streams can include metadata in the metafile or in the header of the stream, whereas MP3 format streams can include the metadata in ID3 tags in the header or at the end of the file. Depending upon the format of the real-time stream, a stream listener 180 may be required to continuously monitor the stream in order to detect the presence of a header that precedes a transient resource in the stream. When a header is detected, the metadata associated with the transient resource is decoded and the information stored in the transient multimedia database 185. By continuously monitoring the real-time data streams, stream listeners 180 can detect all headers and insure that the transient multimedia database accurately reflects the current content of the streams.
While continuous monitoring insures that the identity of transient resources will be detected in real-time data streams, from a resource standpoint continuous monitoring can be a very inefficient process as it requires a dedicated listener for each multimedia stream and a significant amount of bandwidth in order to simultaneously monitor all streams. As a result, the stream analysis module 170 preferably attempts to identify a sampling methodology that conserves system resources while at the same time insuring that the transient multimedia database 185 is as accurate as possible. The sampling methodology used by a listener is contingent in part upon the metadata format in the associated real-time data stream. Certain formats provide that the metadata associated with a resource in a stream is periodically rebroadcast during the delivery of the resource. When a stream is delivered in a format that repeats metadata in the stream, a stream listener may be able to periodically sample the stream in order to detect the metadata and update the transient multimedia database. To insure the accuracy of the index, the sampling rate should preferably be more frequent than the length of the smallest typical resource in the stream. That is, if each resource in a particular stream is always greater than two minutes in length, the listener can sample the resource stream at a higher rate (e.g., once every minute) in order to be confident that the metadata for each resource is accurately captured. Of course, the sampling of a stream should preferably be done as near to the beginning of the resource as possible in order to insure that a user that is searching for that resource will be able to access the resource at the beginning of the resource rather than at the end of the resource. Those skilled in the art will recognize that sampling on a more frequent basis will increase the likelihood of sampling a resource at the beginning, and that tradeoffs can be made between sampling rate, resource usage, and probability of resource detection. Additionally, in some stream formats an associated metadata-only component of the stream may be subscribed to by the listener without having to monitor the portion of the stream containing the media resources. In this case, a different rate and type of sampling may be appropriate to monitor the contents of the stream. In still other formats, such as when metadata is provided in an out-of-band manner, alternative sampling methodologies may be adopted. Some producers of real-time multimedia streams may provide a feed (such as an RSS feed) or a schedule on a web page that indicates when a particular transient resource is starting in a multimedia stream. If such an out-of-band signal is provided by a stream producer, listener 180 can utilize the out-of-band signal to detect the start of the resource (such as by receipt of a message via the feed) and update the transient multimedia database 185 with the metadata information provided in the out-of-band signal. Those skilled in the art will appreciate that other stream formats and methods of delivering metadata about transient resources contained in real-time multimedia streams may dictate other forms of stream listeners as well.
Still further facility resources can be conserved when the same real-time multimedia stream content is found to be delivered in multiple formats or data rates. For example, it is not unusual for multimedia stream producers to provide access to the same real-time stream content in each of a dominant media format, e.g., in Real Audio format and in Windows Media format. Similarly, within each format the multimedia stream producer may have the stream delivered in a variety of data rates that are optimized for a user's available bandwidth. As an example, a producer may deliver a high bandwidth Real Audio stream for users with a broadband connection and a low bandwidth Real Audio stream for users with a dialup connection. When the stream analysis module 170 identifies that the same real-time multimedia stream is merely being delivered in different formats or data rates, and when all of the streams are being delivered in a roughly synchronous manner, it is not necessary to configure a stream listener 180 to monitor each of these streams. Since the content contained in each real-time stream is the same, a single listener 180 could be configured to monitor a stream having a preferred format and data rate, and any metadata that the listener identifies about the monitored stream can be replicated in the transient multimedia database 185 for the streams in other formats and data rates. Alternatively, the facility may also start monitoring two real-time streams and only later determine that the streams are the same based on the identified resources in the streams. When such a similarity between streams is detected, the facility may opt to monitor only the stream having a preferred format and data rate. By monitoring only a single stream, a significant amount of facility resources can be conserved.
Soon after characterizing information about a transient resource is stored in the transient multimedia database 185, the transient multimedia index 110 is updated. The transient multimedia index is a searchable index that correlates search queries with records in the transient multimedia database. Transient multimedia index 110 may be created using any of a variety of techniques that are known in the art, including an inverted file or other index construction. Because the transient resources contained in real-time multimedia streams 155a-155n are typically of short duration, the transient multimedia index 110 is frequently updated to insure that searches of the transient multimedia database will remain highly accurate for users.
An interface module 190 is provided to allow users or web services to access and search the transient and static databases in order to identify multimedia resources. Inquiries in the form of search queries are submitted by users, web services, or other applications and presented to a search module 195. Using standard techniques, the search module 190 applies a search algorithm to parse the search query and apply the query against both the transient multimedia index 110 and the static multimedia index 120. The search queries may be performed simultaneously or sequentially, and the results merged by the search module. An example of search technology that may be useful in this context may be found in U.S. patent application Ser. No. 10/886,946 entitled “Internet Streaming Media Workflow Architecture” (filed 7 Jul. 2004) which is hereby incorporated by reference in its entirety. The search module 195 generates results from the transient multimedia database 185 and the static multimedia database 120 that are responsive to the search query, and returns the results to the interface module 190 in accordance with various parameters set by the user, web service, or other application. By presenting search results that contains both static and transient multimedia resources, the facility offers a much more compelling search experience.
Once the real-time data stream is accessed, the stream is analyzed to validate that the listener is correctly configured to monitor the stream and to further characterize the stream beyond the overarching metadata previously collected. At block 230, metadata associated with the transient resources contained in the stream is collected. The collection of transient resource information in this preliminary analysis helps the facility to characterize the typical media resource contained within the stream. The real-time stream may therefore be monitored for a short period of time at block 230 until the facility achieves a reasonable degree of confidence as to the typical transient resource that may be embedded in the stream.
At block 235, sampling parameters are derived that optimize the sampling rate of the listener in order to monitor the real-time multimedia stream. Such sampling parameters may be based on a number of different factors, including but not limited to the rate at which metadata is provided in the stream and the availability of out-of-band metadata. Moreover, the sampling parameters may be tuned so that the sampling is no more frequent that the transient multimedia index can be updated.
At block 240, some of the derived information that characterizes the multimedia stream may be stored in the stream database 160, including, for example, a pointer or other index to where the stream may be accessed, the technical characteristics of the stream including format and rate, information about the content and publisher of the stream, and information about the sampling parameters necessary to monitor the stream. Such stored information is used to appropriately configure the stream listeners 180. Following block 240, the analysis process 200 of one of the multimedia streams terminates.
While
At block 520, the facility optionally estimates the lifespan of the search results. Since the transient search results reflect the content within real-time streams, and since resources within the real-time streams are constantly changing, the transient resource results produced by the search process 500 will only be valid for a certain period of time, (In contrast, the static resource results can be considered to have an indefinite lifespan.) Such lifespan can be estimated if the approximate length of the identified transient resources is known, as well as the time that each of the transient resources started streaming. For example, if a stream listener 180 is monitoring an audio stream and detects a transition to a new song, a timestamp can be recorded in the transient multimedia database 185 that indicates when the new song started. Once the start time of the song is known, and the length of the song is also known from metadata associated with the song, then the facility can estimate the remaining time left in the song whenever a search query is received. If such an analysis were performed for all transient resources contained in a search result, the facility would know exactly how long the search results would be valid before having to be refreshed. The refresh could be manually or automatically implemented, and may occur as soon as the resource having the smallest amount of remaining time ended. Refreshing the search results may take a variety of forms. New search results might be presented to the user, with expired transient resources being dropped from the list of search results. Alternatively, the same search results might be presented, but with a graphical indication to the user (such as an icon or a red font color) indicating that certain transient resources in the search results are no longer available. The estimate performed at block 520 may obviously be omitted if such estimated information is not going to be presented to a user or used in another fashion.
At block 525, the facility outputs the search results which may include both static multimedia resources and transient multimedia resources. The static and transient resources may be intermixed, or they may be separately identified. As discussed in greater detail below, the amount of detail and format of the results may also vary considerably. If the lifespan of the transient resources has been previously determined in block 520, at block 525 the facility may elect to omit certain transient results that are reaching the end of their availability. For example, if only 5 seconds of a song remains in an audio stream when the search results are being formatted for being output, the song may be omitted from the results so that a user is not presented with results that have such a short effective life as to not be useful.
At decision block 530, the results may be periodically updated. Such update may occur based on a manual refresh requested by the user. Alternatively, such update may occur automatically when the facility determines that the lifespan of some or all of the transient results has expired. Since the index for the static multimedia resources is not refreshed very frequently, updating the results may require only a new search of the transient multimedia database. As a result, if an update is to occur the facility loops to a decision block 535. At block 535, an analysis if made to determine whether a threshold period of time has elapsed since the results have been delivered. If a threshold period has elapsed, the facility returns to block 510 and an updated search of both the static and the transient databases are performed. If an extended period has not elapsed, the facility instead returns to block 515 and only an updated search of the transient multimedia database is performed. The threshold period of time may be set based on how frequently the static multimedia database and index is updated.
Search results area 620 present the results of the search after the user has specified the search query and clicked on the search button 625. As depicted in
It will be appreciated that the user interface depicted in
While various embodiments are described in terms of the environment described above, those skilled in the art will appreciate that the facility may be implemented in a variety of environments including a single, monolithic computer system, a distributed system, as well as various other combinations of computer systems or similar devices connected in various ways. Moreover, the facility may utilize third-party services and data to implement all or portions of the aforementioned functionality. Those skilled in the art will also appreciate that the steps shown in
While the foregoing description makes reference to particular embodiments, the scope of the invention is defined solely by the claims that follow and the elements recited therein.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11489273 | Jul 2006 | US |
| Child | 14048833 | US |
