Patent Application
20050123283
The present invention relates, in general, to data storage and archiving and, more specifically, to file formats for storing multiple tracks or streams of data.
Thanks to its fidelity, digital video and audio have become increasingly popular in entertainment and information recording. For example, digital versatile disc or digital video disc (DVD) provides a format used to store movies, music, or software programs. A DVD movie often has multiple audio tracks for multilingual presentation of the movie and/or multiple video tracks for including special features such as interviews with the movie producer, movie trailers, etc. A DVD has a memory capacity of approximately six gigabytes (GB). In the standard format, a single sided DVD generally can store approximately two to three hours of video.
It would be advantageous to have a file format for storing digital data with a high compression rate. It would be desirable for the file format to be capable of storing data in multiple streams or tracks. It would also be desirable for the file format to be able to encode and archive video, audio, and text data on easily accessible streams or tracks. It would be of further advantage for the file format to be able to provide copyright protection for the digitized content.
In accordance with preferred embodiments of the present invention, digitized data, e.g., digital video, audio, and/or text data, are encoded and stored in a multimedia file following a format that is compatible with a standard data coding and compression algorithm. The file is readable and/or executable by a processor, e.g., a specific or generic signal processor, a digital signal processor (DSP), a signal processor on an Application Specific Integrated Circuit (ASIC), an Advanced RISC Machine (ARM) microprocessor, etc.
In accordance with a specific embodiment of the present invention, the file format is based on the audio video interleave (AVI) multimedia format. The AVI file format is a Resource Interchange File Format (RIFF) file specification used with applications that capture, edit, and play back audio-video sequences.
RIFF, introduced in 1991 by Microsoft Corporation and IBM Corporation, is a format for storing tagged data structures. The structure and coding of RIFF can be found at the Microsoft Developer Network (http://msdn.microsoft.com/). The related information on the Microsoft Developer Network is incorporated herein by reference.
A RIFF file includes a RIFF header followed by zero or more lists and chunks. The RIFF header has the following form:
A chunk has the form:
A list is an ordered collection of other chunks, for example a collection of movie frames. In RIFF, a list has the form:
It is customary and efficient to imply the chunk size and adopt a simplified notation to represent a RIFF chunk:
An AVI file is identified by a FOURCC ‘AVI ’ in the RIFF header. All AVI files include two mandatory LIST chunks: the stream format and the stream data. An AVI file may also include an index chunk, which gives the location of the data chunks within the file. An AVI file with these components has the form:
The ‘hdrl’ list defines the format of the data and is the first required LIST chunk. The ‘movi’ list contains the data for the AVI sequence and is the second required LIST chunk. The ‘idx1’ list contains the index, which is optional. AVI files keep these three components in the proper sequence.
The ‘hdrl’ and ‘movi’ lists use subchunks for their data. The following example shows the AVI RIFF form expanded with the chunks needed to complete these lists:
The character set (CSET) chunk is typically used to define a character set and language information for a RIFF file, a LIST, or a stream. The CSET chunk is defined as follows:
In accordance with a preferred embodiment of the present invention, the functions and formats of the fields in the CSET chunk are defined as follows:
The information ‘INFO’ list is a registered global form type that can store information, e.g., copyright information and comments, that helps identify the contents of the chunk. This information, although useful, does not affect the way a program interprets the file. An ‘INFO’ list is a ‘LIST’ chunk with list type ‘INFO’.
In accordance with a preferred embodiment, an ‘INFO’ list may contains the chunks listed in Table 3. Additional chunks may be defined. Preferably, an application ignores any chunk it doesn't understand. Each chunk contains a null-terminated Unicode text string. The character set used in the string is specified by the global CSET chunk.
The AVI file header (‘hdrl’) list includes a main AVI header in an ‘avih’ chunk. One or more stream descriptor lists follow the main AVI header. Each stream descriptor is contained in an ‘strl’ list.
The main AVI header contains global information for the entire AVI file, such as the number of streams within the file and the width and height of the AVI sequence. The main header chunk includes an AVIMAINHEADER structure, whose syntax is defined as:
In accordance with a preferred embodiment, the members in the AVIMAINHEADER structure have the following variables:
By way of example, the flags in the member dwFlags include the following bitwise combinations:
One or more steam descriptor (‘strl’) lists follow the main header ‘hdrl’. Each ‘strl’ list corresponds to a data stream and includes information about the data stream in the file. A ‘strl’ list contains a stream header chunk (‘strh’) and a stream format chunk (‘strf’). In addition, a ‘strl’ list may contain a stream header data chunk (‘strd’) and a stream name chunk (‘strn’). The stream descriptors in the ‘hdrl’ list are associated with the stream data in the ‘movi’ list according to the order of the ‘strl’ lists. The first ‘strl’ list applies to stream 0, the second applies to stream 1, and so forth.
The stream header chunk (‘strh’) in the ‘strl’ list includes an AVISTREAMHEADER structure containing information about a stream in the AVI file. The AVISTREAMHEADER structure has the syntax:
In accordance with a preferred embodiment, the members in the AVISTREAMHEADER structure have following variables:
A stream format (‘strf’) chunk follows the stream header (‘strl’) chunk. The stream format chunk describes the format of the data in the stream. The data contained in this chunk depends on the stream type.
For video streams, the information is a BITMAPINFOHEADER structure, including palette information if appropriate. The structure of BITMAPINFOHEADER is defined as:
In accordance with a preferred embodiment, the members in the BITMAPINFOHEADER structure have following variables:
For audio streams, the information is a WAVEFORMATEX structure. For text streams, the information has a TEXTINFO structure:
where the meaning of all the fields (wCodePage, wCountryCode, wLanguageCode, and wDialect) is the same as those defined above with reference to CSET chunk. Different languages can be set for each of the text streams in a file having multiple text streams.
If the optional stream header data (‘strd’) chunk is present in an AVI file, it follows the stream format chunk. The format and content of the ‘strd’ chunk are defined by the codec driver. Typically, drivers use this information for configuration. Applications that read and write AVI files do not need to interpret this information, they simple transfer it to and from the driver as a memory block.
The information block for achieving the digital rights management (DRM) protection in the AVI file is presented in the ‘strd’ chunk associated with the main video stream. The format of the DRM information data for the video stream in the ‘strd’ should be as following:
where the two members in the structure DRMINFO are defined as:
The optional stream name ‘strn’ chunk includes a null terminated text string describing the stream. In accordance with an embodiment of the present invention, the string is “Video [—Description]” for a video stream, where optional [—Description] part is any string that describes the video stream, e.g., it can be “Video—Main”. For an audio stream, the string can be “Audio [—Description]”, where the optional [—Description] part is any string that describes the audio stream, e.g., it can be “Audio—English”, “Audio—French”, “Audio—Main”, or “Audio—Auxiliary”, etc. For a chapter stream, which is a text stream, the string can be “Chapter [—Description]”, where the optional [—Description] part is any string that describes the chapter stream. For a subtitle stream, which can be either a text stream or a video stream, the string can be “Subtitle [—Description]”, where the optional [—Description] part is any string that describes the subtitle stream, e.g., it can be “Subtitle—English”, or “Subtitle—Chinese”.
AVI stream data ‘movi’ list follows the header information in the AVI RIFF file format. The ‘movi’ list contains the actual data in the streams, e.g., the video frames and audio samples. The data chunks can reside directly in the ‘movi’ list, or be grouped together as subchunks within ‘rec’ lists. The ‘rec’ grouping implies that the grouped subchunks should be read from disk all at once, and is intended for files that are interleaved to play from CD-ROM.
Each data chunk in the ‘movi’ list is identified by a FOURCC that includes a two-digit stream number followed by a two-character code that defines the type of information in the chunk. In accordance with an embodiment of the present invention, the two-character codes for defining the data type are:
It should be noted that, in accordance with the present invention, additional two-character codes may be used to identify data streams not specified herein above.
By way of example, if stream 0 contains audio, the FOURCC for the stream would be ‘00wb’. If stream 1 contains video, the FOURCC for the stream would be ‘01db’ for uncompressed video or ‘01dc’ for compressed video. Video data chunks can also define new palette entries to update the palette during an AVI sequence. Each palette-change chunk (‘xxpc’) contains an AVIPALCHANGE structure. If a stream contains palette changes, the AVISF_VIDEO_PALCHANGES flag in the dwFlags member of the AVISTREAMHEADER structure for that stream is set accordingly.
The optional index list follows the ‘movi’ list in the AVI RIFF file format. The index contains a list of the data chunks and their location in the file. If the AVI file contains an index, the dwFlags member of the AVIMAINHEADER structure is set to AVIF_HASINDEX.
In version AVI 1.0, the index (‘idx1’) list includes an AVIOLDINDEX structure with entries for each data chunk, including ‘rec’ chunks. The AVIOLDINDEX structure has the syntax:
The members in the structure have following characters:
In accordance with a preferred embodiment, the AVIOLDINDEX structure includes the initial RIFF chunk (the fcc and cb members) followed by one index entry for each data chunk in the ‘movi’ list. The AVIOLDINDEX structure describes an AVI 1.0 index (‘idx1’ format). New AVI files should use an AVI 2.0 index (‘indx’ format).
Additional data can be aligned in an AVI file by inserting ‘JUNK’ chunks as needed. Applications will ignore the contents of a ‘JUNK’ chunk.
In accordance with the present invention, the video tracks of one or more movies are stored in an AVI file as AVI video streams or tracks. A single AVI file may include multiple video tracks. Preferably, the first of the multiple video tracks is the main video track.
The stream descriptor (‘strl’) list for a video stream should include a stream header (‘strh’) chunk, a stream format (‘strf’), a stream header data (‘strd’) chunk if the stream is DRM protected, and a stream name (‘strn’) chunk. In accordance with an embodiment of the present invention, the member fccType in the structure AVISTREAMHEADER the stream header (‘strh’) chunk for a video stream has the value ‘vids’. The stream header data (‘strd’) chunk of a video stream should exist only for DRM protected video. If the ‘strd’ chunks exists, the video stream is protected, and there will be ‘xxdd’ DRM chunks in the video stream. The stream name data (‘strn’) chunk for a video stream includes a string of the form “Video [—Descriptions]”.
The stream data (‘movi’) list of a video stream in includes an ‘nndb’ chunk for an uncompressed video data chunk or an ‘nndc’ for a compressed video data chunk, where ‘nn’ is a two digit data chunk index. If a video data chunk is DRM protected, the ‘movi’ list also includes a ‘nndd’ chunk preceding the corresponding ‘nndb’ or ‘nndc’ chunk of the protected video data chunk. In accordance with a specific embodiment of the present invention, the member dwFlags in the structure AVIOLDINDEX of the index entry for the ‘nndd’ chunk is set to AVIF_NO_TIME.
In one embodiment of the present invention, each video data chunk includes one video frame in variable bit rate coding. For video frames encoded in predicted frames (P frames) and bidirectional frames (B frames), a B frame is preferably placed in a chunk with the following P frame. In such cases, an uncoded dummy P frame (N in the following illustration) is preferably inserted by the codec to keep the timing. For example, a sequence of image frames (I frames), B frames, and P frames Im Bm+1 Pm+2 Bm+3 Pm+4 . . . is preferably arranged into the following video stream chunk sequence:
In accordance with the present invention, the audio tracks of one or more movies are stored in an AVI file as AVI audio streams or tracks. A single AVI file may include multiple audio tracks. Preferably, the first of the multiple audio tracks is the main audio track.
The stream descriptor (‘strl’) list for an audio stream should include a stream header (‘strh’) chunk, a stream format (‘strf’), and a stream name (‘strn’) chunk. In accordance with a specific embodiment the ‘strl’ list for an audio stream does not include the stream header data (‘strd’) chunk. In this embodiment, the application should ignore any data chunk with the ‘strd’ code in the steam descriptor (‘strl’) list of the AVI file.
In accordance with an embodiment of the present invention, the member fccType in the structure AVISTREAMHEADER the stream header (‘strh’) chunk for a video stream has the value ‘auds’. The stream name data (‘strn’) chunk for a video stream includes a string of the form “Audio [—Descriptions]”.
The stream data (‘movi’) list of an audio stream in includes an ‘nnwb’ chunk for identifying an audio data chunk, where ‘nn’ is a two digit data chunk index. In one embodiment of the present invention, each audio data chunk includes one audio frame in variable bit rate coding. In another embodiment of the present invention, each audio data chunk includes one or more audio frames in constant bit rate coding.
In accordance with the present invention, the chapter tracks are stored in an AVI file as AVI text streams or tracks. A single AVI file may include multiple chapter tracks. The stream descriptor (‘strl’) list for a chapter stream should include a stream header (‘strh’) chunk, a stream format (‘strf’), and a stream name (‘strn’) chunk. In accordance with a specific embodiment the ‘strl’ list for a chapter stream does not include the stream header data (‘strd’) chunk. In this embodiment, the application should ignore any data chunk with the ‘strd’ code in the steam descriptor (‘strl’) list of the AVI file.
In accordance with an embodiment of the present invention, the member fccType in the structure AVISTREANHEADER the stream header (‘strh’) chunk for a video stream has the value ‘txts’. The stream format (‘strf’) chunk for a chapter stream has the TEXTINFO structure. The stream name data (‘strn’) chunk for a video stream includes a string of the form “Chapter [—Descriptions]”.
The stream data (‘movi’) list of a chapter stream in includes an ‘nnch’ chunk for identifying a chapter data chunk, where ‘nn’ is a two digit data chunk index. In one embodiment of the present invention, each chapter data chunk has a CHAPTERCHUNK structure:
The members in the structure CHAPTERCHUNK are
The chapter stream is not a regular interval stream. In accordance with a specific embodiment of the present invention, the member dwFlags in the structure AVIOLDINDEX of the index entry for the ‘nnch’ chunk is set to AVIF_NO_TIME.
In accordance with one embodiment of the present invention, the subtitle tracks are stored in an AVI file as AVI text streams or tracks. In accordance with another embodiment of the present invention, the subtitle tracks are stored in an AVI file as AVI bitmap streams or tracks. A single AVI file may include multiple subtitle tracks. The stream descriptor (‘strl’) list for a subtitle stream should include a stream header (‘strh’) chunk, a stream format (‘strf’), and a stream name (‘strn’) chunk. In accordance with a specific embodiment the ‘strl’ list for a subtitle stream does not include the stream header data (‘strd’) chunk. In this embodiment, the application should ignore any data chunk with the ‘strd’ code in the steam descriptor (‘strl’) list of the AVI file.
In accordance with an embodiment of the present invention, the member fccType in the structure AVISTREAMHEADER the stream header (‘strh’) chunk for a video stream has the value ‘txts’ for text form subtitles or ‘vids’ for bitmap form subtitles. The stream format (‘strf’) chunk for a subtitle stream has the TEXTINFO structure for text form subtitles and the BITMAPINFOHEADER structure for bitmap form subtitles. The stream name data (‘strn’) chunk for a video stream includes a string of the form “Subtitle [—Descriptions]”.
The stream data (‘movi’) list of a subtitle stream includes an ‘nnst’ chunk for identifying a text form subtitle data chunk and/or an ‘nnsb’ chunk for identifying a bitmap form subtitle data chunk, where ‘nn’ is a two digit data chunk index. In one embodiment of the present invention, each subtitle data chunk has a SUBTITLECHUNK structure:
The members in the structure SUBTITLECHUNK are
The subtitle stream is not a regular interval stream. In accordance with a specific embodiment of the present invention, the member dwFlags in the structure AVIOLDINDEX of the index entry for the subtitle chunk is set to AVIF_NO_TIME.
For bitmap format subtitles, it is preferred to have compressed subtitle bitmaps in the subtitle field in the subtitle chunks. A compressed subtitle bitmap will have the following fields:
In accordance with a preferred embodiment, the subtitle bitmaps are four-level bitmaps with the following definition.
Compression of the subtitle bitmap uses a simple run-length coding according the rules in Table 4. In accordance with an embodiment of the present invention, the size of the run-length coded data within one line is 1440 bits or less.
In accordance with a preferred embodiment of the present invention, the streams in AVI files are interleaved. Audio stream chunks are interleaved ahead of corresponding video stream chunks in time. The amount of the audio stream that is interleaved ahead of corresponding points in the video stream should not exceed an predetermined upper limit, e.g., 2 seconds, 5 seconds, 10 seconds, 15 seconds, etc. The subtitle chunks are interleaved in the file ahead of the corresponding video chunk in time, with the amount of subtitle interleaved ahead of corresponding points in the video stream not exceeding a upper limit, e.g., 5 seconds, 10 seconds, 15 seconds, 20 seconds, etc. The interleaving of the chapter stream is not restricted. It could be all written in the beginning of the “movi” list, or interleaved with the other streams.
An AVI file typically does not contain a time stamp of the streams. Each stream has its frame rate specified in the stream descriptor (‘strl’) list in the AVI header. For variable bit rate streams such as video streams or variable bit rate audio streams, each chunk contains one and only one frame. Accessing the data of the variable bit rate stream at any given point is feasible with the known frame rate and the data chunk index. For constant bit rate streams, e.g., constant bit rate audio streams, each chunk may contain one or more frames. Because each frame has a known fixed size, locating data at any given point can be achieved by calculating the size of the stream data. Therefore, seeking an arbitrary location in an AVI file in accordance with the present invention can be achieved for either constant bit rate or variable bit rate streams by parsing and recording the index table for each frame.
Many playback devices, particularly consumer electronics devices such as DVD players, are not able to input pointers to arbitrary points as can a slider bar used in computer software. For such devices, it is beneficial to only record the chapter location, i.e., the starting point of audio, video, and subtitles, while parsing the index. For a memory restricted player, it may be preferred for the player to remember index records at the minute points to reduce memory usage, thereby saving limited memory space. The full index is not required during normal forward play because the chunk is self-contained.
In accordance with the present invention, the version of the video codec used in AVI files is signaled by the FourCC code in the fccHandler field or member of the AVISTREAMHEADER of the corresponding stream header ‘strh’ chunks, and the FourCC code bicompression field or member in the BITMAPINFOHEADER of the corresponding ‘strf’ chunks.
By way of example, for videos encoded according to a codec developed by DivX Networks, Inc., 10350 Science Center Drive, Building 14, Suite 140, San Diego, Calif. 92121, the FourCC codes fccHandler in the stream header (‘strh’) of the AVISTREAMHEADER is set to “divx” or “DIVX”. Furthermore, the FourCC (DWORD) code biCompression in the BITMAPINFOHEADER of the corresponding ‘strf’ chunks is set to signify the detailed codec version.
Specifically by way of example, for version DivX 3.11, ‘div3’ or ‘div4’ is used in AVISTREAMHEADER, and ‘div3’ or ‘div4’ is used in BITMAPINFOHEADER; for version DivX 4.x, ‘divx’ is used in AVISTREAMHEADER, and ‘divx’ is used in BITMAPINFOHEADER; and for version DivX 5.x, ‘divx’ is used in AVISTREAMHEADER, and ‘dx50’ is used in BITMAPINFOHEADER.
By now it should be appreciated that a file format for storing digital data with a high compression rate has been described. A file format in accordance with the present invention is compatible with high level data compressing algorithms, such as MPEG-4. Its data compression ratio is about six to ten times higher than a standard DVD format. In accordance with the present invention, the file format is capable of storing data in multiple streams or tracks. The file format is also able to encode and archive video, audio, and text data on easily accessible streams or tracks. Furthermore, the file format is able to provide protection of the copyright of the digitized content.
While the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention. The present invention is limited only by the claims that follow.
