Patent Application
20050193005
N/A
1. The Field of the Invention
The present invention relates to indexes for multimedia content. More specifically, the present invention relates to adding an arbitrary index to multimedia content such that the arbitrary index is capable of coexisting with one or more other arbitrary indexes.
2. Background and Related Art
Traditionally, indexing of multimedia content has been either time-based or frame-based. In other words, access to multimedia content (other than sequential access) requires specifying a particular time offset (e.g., 5 minutes and 18 seconds) or specifying a particular frame offset (e.g., frame 302010). With respect to content for which compression is linear, such as audio content, time offsets essentially correspond to byte offsets within the content, and therefore the content may appear to be indexed, when in fact it is not—seeking to a particular time offset is simply a matter of seeking to the corresponding byte offset within the content. Compression for other content, such as video content, is nonlinear, and therefore seeking to a particular time offset usually requires a time index that maps either directly to a byte offset within a file or indirectly to a byte offset through a frame index that maps directly to a byte offset.
One of the problems with traditional multimedia indexes is their lack of flexibility. Generally these indexes are produced by the content author and may not be subsequently altered or extended. In some cases, the indexing also may be linked to the layout or physical storage of the multimedia content on a medium. For example, DVDs often allow for navigation by scene. This navigation, however, is based on the multimedia content being divided into chapters on the DVD (each scene is a separate chapter). As a result, it would not be possible for a consumer to alter the DVD's scene navigation, by adding, deleting, or combining scenes. Other traditional indexes are stored in a file header and therefore are limited in size. When the index needs to expand, the multimedia content must be shifted down in the file, which in this context, may be prohibitive, given that the size of multimedia data often is measured in gigabytes.
Traditional indexes also tend to be brittle. For instance, returning to the DVD example above, a content author could alter the scene navigation, but in doing so, the original navigation is no longer possible. Because the scene indexing is based on chapter divisions, it is not possible for two different scene navigations to coexist on the same medium. Of course, altering scene navigation for a DVD would not be possible for anyone other than the DVD zauthor.
Similar indexing problems arise in other multimedia contexts. For example, the increasing popularity of personal video recorders (PVRs) offers an opportunity for a number of desirable indexing scenarios. PVRs generally are used to record multimedia content that is broadcast over a cable network, a satellite network, or publicly accessible frequency bands. Examples of desirable indexes might include an index for scene changes, voice changes, commercials, MPEG video frames (I, P, and B), viewer-defined events (e.g., favorite scenes, etc.), content protection information, and so forth. Traditional multimedia indexing schemes, however, lack the ability to generate arbitrary indexes that can coexist and not interfere or collide with each other. This is particularly so, where different indexes of the same type are present (e.g., scene changes, favorite scenes, favorite scenes of different consumers or users, etc.).
Accordingly, methods, systems, and computer program products are desired for arbitrarily indexing multimedia content such that the arbitrary indexes are capable of coexisting with each other.
The present invention relates to arbitrarily indexing multimedia content. In accordance with example embodiments, one or more multimedia content streams are received, each stream comprising one or more samples. At least one sample is passed to one or more multimedia filters for analysis based on criteria that define an index type for each filter. A determination is made that the sample meets the criteria for an arbitrary filter and index type. An index entry that identifies the sample and the index type of the arbitrary filter is created. The index entry is adapted in accordance with a common, extensible indexing scheme to coexist with other index entries of other or the same index type for the sample. The created index entry is stored for subsequent access.
The sample may have already been indexed, perhaps by the content author, with at least one other index, such as a time- or frame-based index, or a Society of Motion Picture and Television Engineers (SMPTE) timecode index. If multiple filters are present, it may be determined that the sample meets the criteria for another filter and index type. In this case, another index entry that identifies the sample and the index type of the other filter is created and stored for subsequent access. Multiple filters may be arranged in serial, in parallel, and/or a combination of serial and parallel. Index entries may be stored with the multimedia content, in a separate file, or in a temporary memory, such as an index cache. Types of filters and indexes are completely arbitrary, and may include new scenes, a user's favorite scenes, correlated external events (such a clocks, Internet pulses, instant messages, phone calls, and so forth), text criteria (such as closed captioning criteria, subtitle criteria, descriptive video criteria, and the like), voice change or other audio criteria, commercials, video frames, content protection information, etc. It should be recognized that indexes may be hierarchical (i.e., they may reference previously generated indexes to an arbitrary number of levels, which finally reference the actual content). Multimedia content should be interpreted broadly to encompass audio, video, graphics images, closed-captioning data, subtitles, Advanced Television Enhancement Forum (ATVEF) data, and the like.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention. The features and advantages of the invention may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth hereinafter.
In order to describe the manner in which the above-recited and other advantages and features of the invention can be obtained, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered as limiting its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
The present invention extends to methods, systems, and computer program products for adding an arbitrary index to multimedia content such that the arbitrary index is capable of coexisting and not interfering with one or more other arbitrary indexes regardless of index type. The embodiments of the present invention may comprise one or more special purpose and/or one or more general purpose computers including various computer hardware, as discussed in greater detail below.
When the analysis criteria of a filter is met or satisfied, the sample is flagged and an index entry is subsequently generated for the content under analysis. (An example multimedia index is described in greater detail below with respect to
The analysis of the filters may be applied to multimedia content in real-time as the content is received from a broadcaster or other source, or may applied as a post-processing operation. For example, as a personal video recorder (PVR), such as the one described below with respect to
PVRs frequently are used as time shifting devices. For a recorded program, it may be desirable to fast forward (or rewind), which may be implemented as a seek ahead/back a given amount of time or a given number of I-frames, requiring either a time-based index or a frame-based index. During playback, a user may be bored or uninterested with a particular scene and want to skip forward to the next scene, or may enjoy a particular scene and wish to play the scene back again, in which case a scene change index is needed. For content subject to conditional access (such as premium movie channels, pay-per-view, etc.), randomly seeking within the content may require locating content protection information in order to view the content, in which case a content protection index would be helpful.
A user may desire to locate a particular character within the multimedia content, in which case a voice change index may be needed, and depending on the sophistication of the filter, the filter may be able to index whenever the particular character speaks within the content. When closed captioning information is displayed, seeking to any of various indexes may start the closed caption in the middle of a word. Accordingly, a closed captioning index could be used as a fine adjustment to a more coarsely specified starting point within the content. For example, when seeking by time or frame, the actual time or frame may be altered using the closed captioning index to help assure that complete closed captioning information can be displayed when playback begins.
Similarly, PVRs may implement content jumps or skips of fixed amounts of time, such as 30 seconds. However, if a new scene begins in 25 seconds or 35 seconds, it is likely that the user would prefer jumping directly to the scene change, rather than rigidly holding to an exact 30 seconds. Here too, one index may be used as a fine adjustment to another index. The scene index can be used to alter a 30-second jump slightly, since it is most probable the user would prefer to navigate directly to the new scene, rather than having to rewind or fast forward manually after making the fixed 30-second jump.
Users also may wish to define their own indexes. For example, a user may wish to index favorite scenes within the multimedia content. The favorite scene index could be built on top of the scene index, or could be completely independent index. A favorite scenes index illustrates a filter and index with criteria based, at least in part, on receiving some form of input from the user. One of the problems that may be encountered with traditional indexes is the inability to keep similar indexes (e.g., scene-based indexes) from interfering with each other. In other words, keeping a separate index for all scenes, for one user's favorite scenes, and for another user's favorite scenes has not been possible. Indexing in accordance with the present invention, as described below with respect to the embodiment illustrated in
In another example, the scenes and VBI data within a newscast could be analyzed to generate two indexes for content: 1) scene changes on video, 2) programming type e.g. headlines, sports, weather, etc. The PVR could combine these two indexes so that one newscast could be replayed in a customized manner for two viewers from the same recording. Viewer A might prefer to watch only headlines and sports, in that order. Viewer B might prefer to watch weather first, then headlines, then sports.
Similarly, a voice-recognition index might allow a third viewer of the same newscast to watch only the content when the news anchor is speaking.
Another possible use of an index is a real-time analysis. For example, some analysis module might measure the average color of each frame and store a moving average once a second. Other modules might measure the video bit-rate, also sampled over a one-second duration. Several such measures might be combined to post-process the file and detect subtle correlations between the various measures to categorize passages of video.
Each entry (see
As indicated above,
Because the analysis performed by the filters requires some amount of processing time, filters generally introduce at least some delay when processing the content.
Although
The analysis performed by filters 110, 120, and 130 may be performed as the content is being received and/or may be conducted as part of a post-processing analysis. In either case, the multimedia content may already include existing indexes. As indicated above, the indexing may be saved or stored for subsequent access as either temporary or transitory indexing or as long-term indexing.
Index 210 include type 1 index entry 212, type 1 index entry 214, type 1 index entry 216, and type 1 index entry 218. Index 220 includes type 2 index entry 222, type 2 index entry 224, type 2 index entry 226, and type 2 index entry 228. Note that type 2 index entry 228 and type 1 index entry 218 reference the same multimedia content. This is simply an indication that the sample met the criteria for both filters. Index 230 includes type 3 begin index entry 232 and type 3 end index entry 234. Note that index 230 explicitly defines a span or sample of multimedia content using a “begin” and an “end” index entry. Other indexes may simply imply a sample or span based one or more index entries. Although not shown in
In
GUID 442 is intended to describe the type of index (e.g., scene, voice, commercial, etc.). Stream identifier 444 is used because the same index may be applied to different streams within the multimedia content (e.g., the audio stream, the video stream, etc.). It is also possible to have more than one index on a particular stream. For example, a user may wish to index commercials, scene changes, and frames, all on the video stream. Accordingly, each index will have a different GUID (commercials, scene changes, and frames), but will reference the same stream.
UDI data 450 includes various index entries, labeled Entry[0], Entry[1], and Entry[K] in the figure. Each entry includes a timestamp 452, a packet offset 454, and entry data 456. Timestamp 452 is a time offset into the multimedia content, packet offset 454 is a position offset into the multimedia content, and entry data 456 is a blob that can be used to hold application specific data for the index. Accordingly, at a high-level, a {GUID,blob} attribute exists for each indexed sample (e.g., GUID 442 and entry data 456), where the GUID distinguishes between different indexes and the blob holds application or index specific data. In
An example pseudo-code embodiment of a programming interface for indexing multimedia content may be found at the end of this specification.
The present invention also may be described in terms of methods comprising functional steps and/or non-functional acts.
A step for obtaining (810) one or more multimedia content streams, each comprising one or more samples may include an act of receiving (812) the one or more multimedia content streams. A step for sending (820) at least one sample to one or more multimedia filters for analysis based on one or more criteria that define an index type for each of the one or more multimedia filters may include an act of passing (822) the at least one sample to the one or more multimedia filters. A step for concluding (830) that the at least one sample meets the one or more criteria for an arbitrary filter and index type may include an act of determining (832) that the at least one sample meets the one or more criteria. A step for generating (840) an index entry that identifies the at least one sample and the index type of the arbitrary filter, the index entry being adapted in accordance with a common, extensible indexing scheme to coexist and not interfere with one or more other index entries of either a same or different type for the at least one sample may include an act of creating (842) the index entry. A step for saving (850) the index entry for subsequent access may include an act of storing (852) the index entry. As described above, index entries themselves may be further indexed to create an index hierarchy, which is illustrated by the dashed line from the act of storing (852) the index entry to the act of receiving (812) one or more multimedia content streams. It should also be noted here that for parallel filtering/analysis the multiplexing/synchronization of, for example,
Embodiments within the scope of the present invention also include computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of computer-readable media. Computer-executable instructions comprise, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions.
The processor 514 oversees the general video processing by dispatching instructions over the control bus 511 instructing the various components of the ASIC 510 to perform their specialized tasks. The processor 514 also monitors the progress of such tasks, thus controlling the various components of ASIC 510 in a coordinated fashion. The processor 514 may be any processor capable of performing such oversight functions including a MIPS or X86 architecture processor.
Typically, memory is required to perform such coordinated operations. Accordingly, the ASIC 510 has access to one or more memory subsystems 516 which provide volatile memory that is shared between the components of the ASIC 510. The memory subsystems 516 may be any memory subsystem that allows for rapid access to stored information. For example, the memory subsystems 516 may be SRAM or DRAM.
A memory unit 517 communicates directly with the memory subsystems 516. The memory unit 517 is more efficient if there are large, less frequent accesses to the memory subsystems 516. However, many of the components of the ASIC 510 may operate most efficiently when there are smaller, but more frequent memory transactions. The direct memory access (“DMA”) unit 518 acts as a buffering interface such that the components may have small, frequent transactions with the DMA unit 518, while leaving it up to the DMA unit 518 to bundle the smaller transactions into larger, less frequent transactions for the memory unit 517 to conduct with the memory subsystems 516. In this manner, when a component needs to access the memory subsystems 516, the component either communicates directly with the memory unit 517 or communicates through the DMA unit 518 depending on the nature of the transaction.
A universal serial bus (“USB”) interface 519 is capable of running a universal serial bus. The USB unit 519 may be any conventional USB interface that is capable of interfacing with the control bus 511 and the memory bus 512.
A device unit 521 includes interfaces for a number of miscellaneous devices. For example, the device unit 521 contains a bi-directional interface for an I2C bus 522 for communication with external components, a bi-directional interface for a smart card 523, a bi-directional infra red (“IR”) serial interface 524, and a bi-directional ISA/IDE bus 525 that interfaces with a read only memory 526, a hard disk drive 527, as well as a number of other devices such as a DVD-ROM drive.
A graphics unit 542 comprises a 3-D graphic rendering engine that may be, for example, an eight million polygon DirectX7 compatible 3-D graphics unit.
An audio unit 529 drives a PC audio interface 530 such as an AC'97 audio interface that may receive or transmit audio. The audio unit 529 may also drive other audio interfaces including a digital interface such as SPDIF digital audio interface 531.
A video unit 532 receives video data from the memory bus 512 and converts the video data into a digital display. The video unit 532 handles multiple windows of video data and may operate in RGB, YUV, or other color formats as needed. The video unit 532 provides the digital display data to the digital video encoder 533 which converts the digital display data into the desired format (e.g., NTSC or HDTV) and provides the digital video through a digital to analog converter (“DAC”) and filter 534 to a composite, S-Video or component output. The digital video encoder 533 also may output the video to a digital video interface (“DVI”) 535 using a DVI converter 536.
An MPEG decoder 538 is provided to decode MPEG streams. The MPEG decoder also performs subsample decoding by reducing the frame size of the resulting decoded frame.
A clock 541 synchronizes the components of the system with each other. Accurate clocks also can provide somewhat loose synchronization with external devices that have their own clocks, without any feedback or exchange of clock signals between the devices. In this way, indexes driven by clock times may be produced and stored, and then exchanged between different devices and/or multimedia content consumers.
Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination of hardwired or wireless links) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
With reference to
The computer 620 may also include a magnetic hard disk drive 627 for reading from and writing to a magnetic hard disk 639, a magnetic disk drive 628 for reading from or writing to a removable magnetic disk 629, and an optical disc drive 630 for reading from or writing to removable optical disc 631 such as a CD-ROM or other optical media. The magnetic hard disk drive 627, magnetic disk drive 628, and optical disc drive 630 are connected to the system bus 623 by a hard disk drive interface 632, a magnetic disk drive-interface 633, and an optical drive interface 634, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer-executable instructions, data structures, program modules and other data for the computer 620. Although the exemplary environment described herein employs a magnetic hard disk 639, a removable magnetic disk 629 and a removable optical disc 631, other types of computer readable media for storing data can be used, including magnetic cassettes, flash memory cards, digital versatile discs, Bernoulli cartridges, RAMs, ROMs, and the like.
Similar to clock 541 of
Program code means comprising one or more program modules may be stored on the magnetic hard disk 639, removable magnetic disk 629, removable optical disc 631, ROM 624 or RAM 625, including an operating system 635, one or more application programs 636, other program modules 637, and program data 638. A user may enter commands and information into the computer 620 through keyboard 640, pointing device 642, or other input devices (not shown), such as a microphone, joy stick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 621 through a serial port interface 646 coupled to system bus 623. Alternatively, the input devices may be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB). A monitor 647 or another display device is also connected to system bus 623 via an interface, such as video adapter 648. In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers.
The computer 620 may operate in a networked environment using logical connections to one or more remote computers, such as remote computers 649a and 649b. Remote computers 649a and 649b may each be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically include many or all of the elements described above relative to the computer 620, although only memory storage devices 650a and 650b and their associated application programs 636a and 636b have been illustrated in
When used in a LAN networking environment, the computer 620 is connected to the local network 651 through a network interface or adapter 653. When used in a WAN networking environment, the computer 620 may include a modem 654, a wireless link, or other means for establishing communications over the wide area network 652, such as the Internet. The modem 654, which may be internal or external, is connected to the system bus 623 via the serial port interface 646. In a networked environment, program modules depicted relative to the computer 620, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing communications over wide area network 652 may be used.
Notionally, a programming interface may be viewed generically, as shown in
Aspects of such a programming interface may include the method whereby the first code segment 710 transmits information (where “information” is used in its broadest sense and includes data, commands, requests, etc.) to the second code segment 720; the method whereby the second code segment 720 receives the information; and the structure, sequence, syntax, organization, schema, timing and content of the information. In this regard, the underlying transport medium itself may be unimportant to the operation of the interface, whether the medium be wired or wireless, or a combination of both, as long as the information is transported in the manner defined by the interface. In certain situations, information may not be passed in one or both directions in the conventional sense, as the information transfer may be either via another mechanism (e.g. information placed in a buffer, file, etc. separate from information flow between the code segments) or non-existent, as when one code segment simply accesses functionality performed by a second code segment. Any or all of these aspects may be important in a given situation, e.g., depending on whether the code segments are part of a system in a loosely coupled or tightly coupled configuration, and so this list should be considered illustrative and non-limiting.
This notion of a programming interface is known to those skilled in the art and is clear from the foregoing detailed description of the invention. There are, however, other ways to implement a programming interface, and, unless expressly excluded, these too are intended to be encompassed by the claims set forth at the end of this specification. Such other ways may appear to be more sophisticated or complex than the simplistic view of
Factoring
A communication from one code segment to another may be accomplished indirectly by breaking the communication into multiple discrete communications. This is depicted schematically in
As illustrated in
The factoring of interfaces may also follow associative, commutative, and other mathematical properties such that the factoring may be difficult to recognize. For instance, ordering of operations may be unimportant, and consequently, a function carried out by an interface may be carried out well in advance of reaching the interface, by another piece of code or interface, or performed by a separate component of the system. Moreover, one of ordinary skill in the programming arts can appreciate that there are a variety of ways of making different function calls that achieve the same result.
Redefinition
In some cases, it may be possible to ignore, add or redefine certain aspects (e.g., parameters) of a programming interface while still accomplishing the intended result. This is illustrated in
If a parameter, such as precision, is of no concern in a given scenario, as shown in
Similarly, as shown in
Inline Coding
It may also be feasible to merge some or all of the functionality of two separate code modules such that the “interface” between them changes form. For example, the functionality of
Similarly, shown in
Divorce
A communication from one code segment to another may be accomplished indirectly by breaking the communication into multiple discrete communications. This is depicted schematically in
Similarly, as shown in
Rewriting
Yet another possible variant is to dynamically rewrite the code to replace the interface functionality with something else but which achieves the same overall result. For example, there may be a system in which a code segment presented in an intermediate language (e.g. Microsoft IL, Java ByteCode, etc.) is provided to a just-in-time (JIT) compiler or interpreter 740 in an execution environment (such as that provided by the Net framework, the Java runtime environment, or other similar runtime type environments). The JIT compiler 740 may be written so as to dynamically convert the communications from the first code segment 710 to the second code segment 720, i.e., to conform them to a different interface, such as the interface shown in new first code segment 711, as may be required by the new second code segment 722 (either the original or a different second code segment). This is depicted in
As can be seen in
It is also noted that the above-described scenarios for achieving the same or similar result as an interface via alternative embodiments may also be combined in various ways, serially and/or in parallel, or with other intervening code. Thus, the alternative embodiments presented above are not mutually exclusive and may be mixed, matched and combined to produce the same or equivalent scenarios to the generic scenarios presented in
The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be
