Patent Application
20090157750
The present invention relates to an integrated multimedia file format structure, and a multimedia service system and method based on the integrated multimedia file format structure; and, more particularly, to an integrated multimedia file format structure and an integrated multimedia service system and method suitable for generating and providing integrated contents based on the integrated multimedia format structure.
As well known, in recent, user terminals have become capable of performing not only a function of communicating but also a function of multimedia data processing such as a picture taking a video recording a music reproduction and the like. In particular, it has been an ordinary task to listen to music files, e.g., MP3 files, by playing them on the user terminals.
Such music files are stored in small portable player devices having their own storage media (e.g., an MP3 player or the like), personal computers (PC) and the like, and a desired one among the music files is extracted therefrom for reproduction.
As for widely used music file formats, there are provided file formats such as MP3 (MPEG-1 Audio Layer IEI), MPEG-2/4 AAC (Advanced Audio Coding), WMA, AVI and the like. These music file formats include simply music data and metadata related to music contents (e.g., a song title, a singer, a music album title, a composer, a composition year, a copyright holder and the like). Accordingly, it is disadvantageous in that a user should listen the music without obtaining further detailed information about the music data.
Furthermore, such music file formats are used for compressing music files and thus utilized in applications such as listening comprehension aids for learning foreign languages, storytelling and the like. However, in those application environments, they are provided in file formats which are not capable of supplementing image files or contents-related text information with related contents in synchronization with music sources. That is, the music file formats have been limitedly used in applications that are dependent only on the music sources.
For example, the MP3 includes not only compressed audio data but also metadata related to limited music contents (e.g., a song title, a singer, an album title, a musical composer, a composition year, a copyright holder and the like). Since the metadata provides related information, it is possible to complement somewhat the problem of listening to the music files without obtaining detailed information thereof.
Furthermore, the players for reproducing music files have a limited battery lifespan, computing capability and storage capacity when they are portable players. Furthermore, they generally include a limiting display device (i.e., capable of providing only text information).
In order to overcome such problems for MP3 music files, an MP3 player is configured to connect with a server via a network and transmits related information such as a song title and the like while reproducing the Mp3 music files. In response thereto, the server provides lyrics of the related music files to the MP3 player and then reproduces music data while displaying the lyrics in synchronization therewith.
However, such a conventional method has a drawback in that it is not possible to receive and display the lyrics when the MP3 player is not in connection with the network. Moreover, in order to constantly receive the lyrics for displaying high costs are required for the network connection.
It is, therefore, a primary object of the present invention to provide an integrated multimedia file format structure capable of integrating (or combining) contents including at least one set of image data text data, video data relating to audio data, contents protection metadata and license information metadata.
It is another object of the present invention to provide a multimedia service system and method capable of generating and providing integrated contents based on the integrated multimedia file format structure.
In accordance with one aspect of the invention, there is provided an integrated multimedia file format structure for integrating contents containing at least one among compressed video data, image data, text data and contents information metadata, which correspond to compressed audio data, by using an ISO based file format, the integrated multimedia file format structure including: a moov box for indicating at least one data among the compressed video data, the image data, the text data, script data for applying animation effects to image, video and text data, the contents protection metadata and license protection metadata, which correspond to compressed audio data, using a multiplicity of trak boxes, and for specifying synchronized text data in an additional trak box; a mdat box containing the data indicated by the trak boxes of the moov box and the synchronized text data corresponding to the additional trak box; and a meta box for storing metadata where information on the contents is recorded.
In accordance with another aspect of the invention, there is provided a multimedia service system for providing integrated contents having an integrated multimedia file format structure, the system including: a content server system for assigning a rights to use the integrated contents, generating license information metadata according to the rights to use the contents, encrypting the contents according to the license information metadata and generating contents protection metadata related to the encryption, thereby generating the integrated contents having the integrated multimedia file format structure and transmitting the same via a network; and a user terminal for storing the integrated contents received via the network, decrypting and decoding the stored integrated contents according to the license information metadata and the contents protection metadata, displaying the decoded contents and executing the displayed contents according to an instruction for executing the integrated contents.
In accordance with still another aspect of the invention, there is provided a multimedia service method for providing integrated contents having an integrated multimedia file format structure, the method including the steps of: determining whether or not an integrated content request receiving unit of a content server system has received a content request signal transmitted via a network; performing a user authentication and checking the license of a user's rights to use contents if the integrated content request signal is received; requesting proper contents according to the user authentication and the license and then encrypting the contents as requested according to a content protection method; generating content protection metadata for recording information describing the content protection method and license metadata for recording license information of the contents; generating one integrated contents by using the encrypted contents, the contents protection metadata and the license metadata according to synchronization information and then transmitting the integrated contents to a user terminal via a network; and storing the integrated contents received from the user terminal.
In accordance with still another aspect of the invention, there is provided a multimedia service method for providing integrated contents having an integrated multimedia file format structure, the method including the steps of: determining whether or not integrated contents are requested from user terminals; determining whether or not the requested contents exist if the integrated contents are requested; displaying decrypted metadata of the integrated contents after checking a rights to contents and then displaying context and sources of the requested integrated contents if the requested contents exist, entire lists of pre-stored integrated contents; extracting and decrypting the corresponding contents if an instruction is inputted from the user terminals, according to the user's rights stored in the user terminals or a storage medium; and decoding and rendering the decrypted integrated contents according to the synchronization information and then executing the decoded integrated contents.
As described above, unlike the conventional multimedia service system which provides only audio data and simple text information related thereto only, the multimedia service system in accordance with the present invention produces integrated contents containing compressed audio data, image data, text data, script data for applying animation effects to the image, the text and the video data, contents protection metadata and license metadata. By providing such integrated contents, the compressed audio data is synchronized with the image and the text data related thereto. Consequently, the multimedia service system can effectively generate and provide integrated contents having an integrated multimedia file format structure.
And also, by providing the integrated contents having the integrated multimedia file format structure, the compressed audio data can be simultaneously executed in synchronization with the image data, the text data and the like related thereto. As a result, needs demands of the user using multimedia services can be satisfied.
The above and other objects and features of the present invention will become apparent from the following description of preferred embodiments, given in conjunction with the accompanying drawings, in which:
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
Referring to
As sown in
The script data for applying animation effects to the images, video and text data may be configured by using at least one among an MPEG-4 LASeR (Lightweight Application Scene Representation) standard, an SMIL (Synchronized Multimedia Integration Language) standard, an SVG (Scalable Vector Graphic) standard, an SVG Tiny standard and an XML standard.
The compressed audio data (e.g., MP3 audio data and the like) indicates MP3 data in the mdat box with its dedicated trak box in the moov box of the MPEG-4 file format and also indicates audio related text data (e.g., MPEG-7 metadata and the like) in the mdat box with another trak box. Moreover, at least one image data (e.g., JPEG image and the like) related to the compressed audio data indicates other trak boxes in the moov box.
When the MPEG-7 metadata contains text contents such as lyrics, story sentences, dialogues for learning (foreign) languages and the like, time information data (i.e., time stamp) is contained in a first sentence or a first syllable for synchronization when reproducing the music data (MP3). Furthermore, the synchronization text data such as lyrics, story sentences, dialogue sentences for language learning text related to the pronunciation of words and the like is formed as a separate timed text to be specified by one file, or one trak box in the moov box. Moreover, synchronization information, e.g., a time stamp, an offset, a chunk unit and the like, is contained in the synchronization text data which can be defined in the trak boxes of the MPEG-4 file format, so that the synchronization text data can be synchronized with audio data when reproducing the audio data.
On one hand, the trak boxes indicating image data (e.g., JPEG data) contain time information (e.g., a time stamp, an offset, a chunk unit and the like) for synchronizing each image with music when reproducing the music. Also, these trak boxes may contain the script data for applying animation effects to the image data in order to convert them into other images. If the data indicated by the trak boxes are protected (encrypted), the content protection metadata related to matters concerning which encryption method(s) is(are) used, which block(s) is(are) encrypted, the information about where key data is obtained for decryption, etc. and the license information metadata related to rights to use contents are recorded in an xml box in the meta box by using an MPEG-21 IPMP (Intellectual Property Management and Protection) components. At this time, the metadata may be encrypted with a hash function and include digital signature data to check the integrity of the MPEG-21 IPMP metadata.
In such an integrated multimedia file format structure, it is understood that it is possible to employ audio file formats other than MP3 for a music data file, and other additional file formats such as addition MPEG-2/4 AAC, WMA, AVI, G.723, PCM, MPEG-4 ALS (Audio Lossless Coding), MPEG-4 DST (Lossless Coding of oversampled audio) and the like. Furthermore, as for an image data file, there can be employed image file formats other than JPEG, such as PNG, BMP, TIFF, JPEG2000, MPEG I-frame, GIF, animated GIF, PGMYUV, PGM, YUV, SGI and the like. Furthermore, as for a video data file, it is possible to support at least one among H.264/AVC, SVC, MPEG-1, MPEG-2, MPEG-4, WMV, H.261, H.263, H.26L, AVI, RealVideo, DV, MJPEG, Apple Video file format and the like.
The asynchronous MPEG-7 metadata (e.g., a song title, a music album title, a singer and the like) may be contained in the MPEG-21 DID of the xml box for the meta box. Moreover, the text data that requires synchronization (e.g., text for song lyrics, story sentences, and dialogues for language learning text related to the pronunciation of words and the like) with audio or images may be contained in the mdat box, which is corresponding to an additional trak box. Each item information indicating each trak box of the MPEG-21 DID in the meta box may be mapped with info and iloc information in the meta box, respectively.
As particularly shown in
Referring to
In this regard, each item information indicating each trak box of the MPEG-21 DID in the meta box may be mapped with info and iloc information in the meta box, respectively.
As for another example of the integrated multimedia file format structure in accordance with the second preferred embodiment of
Referring to
Table 1 explains the semantics of the IPMP general information descriptor schema of the MPEG-21 IPMP which describes general information on the content protection tools in accordance with the present invention. Table 2 explains specific semantic of elements indicating tool description information in the general information descriptor schema of the MPEG-21 IPMP which describes general information on the contents protection tools in accordance with the present invention.
As shown in
Table 3 explains semantics of the IPMP information descriptor schema describing information on the contents protection tools for protecting specific contents in accordance with the present invention. And table 4 explains specific semantics of tool base description information components, initialization setting components and copyright description components in the IPMP information descriptor schema describing tool description information for protecting specific contents in accordance with the present invention.
Referring to the example of the file format structure in
Referring to
As shown in
Furthermore, the user terminal 100 executes the contents request through the integrated content requesting unit 114. If designated integrated contents exist in the content storage unit 104 of the user terminal 100, entire lists of stored integrated contents are displayed on the user terminal 100. Then, in order to display context and sources of the integrated contents as requested in the user interface unit 116, the integrated contents decryption/content usage control unit 112 interprets metadata of the integrated contents and then checks the rights to use the contents before transmitting the decrypted contents to the user interface unit 116.
If integrated contents desired by the user exist in the content storage unit 104 of the user terminal 100, the integrated contents as desired is selected via the integrated content metadata decryption/content usage control unit 112. Then, the integrated contents are extracted according to an execution instruction issued from the user interface unit 116 and then, protected contents are decrypted by the content decryptor 106.
The user's rights for decryption may be stored in the user terminal 100 or in the storage medium 400 and, also, the usage of the contents may be limited according to the user's rights obtained from the content decryptor 106.
The extracted and decrypted contents are respectively decoded by their corresponding contents decoders in the content decoding unit 108 of the user terminal 100 and then rendered through the contents rendering unit 110 followed by the user interface unit 116 to thereby render multiple contents in one screen.
On the other hand, if there exists no the integrated multimedia contents desired by the user in the integrated contents lists displayed at the user interface unit 116 of the user terminal 100, a signal requesting the integrated contents is transmitted from the integrated content requesting unit 114 to the content server system 300 via the network 200, i.e., a wire/wireless communication network.
The content server system 300 includes an integrated content request receiving unit 302, a contents protection/license management unit 304, a user authentication unit 306, a contents extracting unit 308, a content storage unit 310, a contents encrypting unit 312, a contents protection metadata generation unit 314, a license metadata generation unit 316, an integrated contents generation unit 318 and an integrated contents transmitting unit 320. The contents request receiving unit 302 of the content server system 300 receives an integrated contents request signal transmitted via the network 200 and then transmits it to the contents protection/license management unit 304. Furthermore, the user authentication unit 306 of the content server system 300 determines the limit of a user's rights to use contents.
The contents protection/license management unit 304 of the content server system 300 requests the contents extracting unit 308 to extract proper contents from the content storage unit 310 according to the user authentication and the user's rights to use the contents. Then, such provided contents are encrypted by the contents encrypting unit 312 according to a content protection method of the contents protection/license management unit 304.
The contents protection metadata generation unit 314 of the content server system 300 generates contents protection metadata to record information for describing the content protection method of the contents protection/license management unit 304. Also, the license metadata generation unit 316 generates license metadata to record license information in accordance with a decision of the contents protection/license management unit 304.
The integrated contents generation unit 318 of the content server system 300 generates one integrated contents by receiving the encrypted contents, the contents protection metadata, the license metadata, and file format configuration information such as synchronization information of those contents, which are outputted from the contents encoding unit 312, the contents protection metadata generation unit 314 and the license metadata generation unit 316, respectively. The integrated contents thus generated are transmitted to the integrated contents transmitting unit 320 and then, the integrated contents transmitting unit 320 transmits it to the user terminal 100 via the network 200.
In the user terminal 100, the integrated contents are received by the integrated content receiving unit 102 and then stored in the content storage unit 104. Moreover, in order to show that the integrated contents requested by the user have been received, the integrated content metadata decryption/content usage control unit 112 analyzes the received integrated content metadata and then transmits the context of the integrated contents and the rights to use the contents to the user interface unit 116.
Furthermore, in the user terminal 100, the integrated contents requested by the user are extracted according to an execution instruction from the user interface unit 316, and the protected contents are also respectively decrypted by the content decryptor 106 according to the user's rights stored in the user terminal 100 or in the storage medium 400. Such extracted and decrypted contents are decoded by the contents decoders in the content decoding unit 108 and also rendered through the user interface 116 followed by the contents rendering unit 110, thereby rendering multiple contents in one screen.
The storage medium 400 includes, e.g., an optical disk, a hard disk, a memory and the like and is connected with the user terminal 100. Accordingly, random integrated contents (e.g., MP3 data and the like) stored therein are extracted according to the user's request and then transmitted to the content decryptor 106 of the user terminal 100.
The following is a description of a process in which the contents service system configured as described above provides the integrated contents generated based on the integrated multimedia file format structure to the user terminal depending on the user's request and rights.
Referring to
If it is determined, at the step 1102 that a certain integrated content has been requested, the integrated content requesting unit 114 of the user terminal 100 checks whether or not the requested content exist in the content storage unit 104 (at step 1104).
If it is confirmed at the step 1104 that the requested contents exist, the entire list of the integrated contents including the requested contents stored in the content storage unit 104 is displayed. Next, the integrated content metadata decryption/content usage control unit 112 interprets metadata of the integrated contents and then transmits the decrypted contents to the user interface unit 116 after determining the user's rights. Accordingly, the context and the sources of the requested contents are displayed in the user interface unit 116 (at step 1106).
Thereafter, the user terminal 100 determines whether or not the user has inputted an instruction to render the integrated contents through the integrated content metadata decryption/content usage control unit 112 and the user interface unit 116 (at step 1108).
If it is checked at the step 1108 that the executive instruction for content rendering has been inputted through the user interface unit 116, the integrated contents are extracted from the content storage unit 104 and, then, the protected contents are then decrypted by the content decryptor 106 according to the user's rights stored in the user terminal 100 or the storage medium 400 (at step 1110).
Next, in the user terminal 100, the extracted and decrypted contents are decoded by the contents decoders in the content decoding unit 108, respectively, and then rendered through the user interface unit 116 followed by the contents rendering unit 110 by using the synchronized information recorded in the integrated multimedia file format to thereby render multiple contents in one screen (at step 1112).
Meanwhile, if it is determined at the step 1104 that the requested contents do not exist, the integrated content requesting unit 114 of the user terminal 100 transmits an integrated contents request signal to the content server system 300 via the network 200 (at step 1114).
As shown in
If it is determined at the step 1202 that the integrated contents request signal has been received, the contents protection/license management-unit 304 and the user authentication unit 306 perform an user authentication and determine the limit of a user's rights to use contents (at step 1204).
The contents protection/license management unit 304 requests the contents extracting unit 308 to extract the proper contents from the content storage unit 310 according to the user authentication and the user's rights to use the contents. Next, such provided contents are encrypted by the contents encrypting unit 312 according to the contents protecting method from the contents protection/license management unit 304 (at step 1206).
Thereafter, the contents protection metadata generation unit 114 generates contents protection metadata to record information for describing the contents protecting method of the contents protection/license management unit 304, and the license metadata generation unit 316 generates license metadata to record license information in accordance with an instruction issued from the contents protection/license management unit 304 (at step 1208). The integrated contents generation unit 318 produces one integrated contents by receiving the encrypted contents, the contents protection metadata and the license metadata provided from the contents encoding unit 312, the contents protection metadata generation unit 314 and the license metadata generation unit 316, respectively. Such generated integrated contents are transmitted to the integrated contents transmitting unit 320 and then, the integrated contents transmitting unit 320 transmits it therefrom to the user terminal 100 via the network 200 (at step 1210).
Next, in the user terminal 100, the integrated contents are received by the integrated content receiving unit 102 and then stored in the content storage unit 104 (at step 1212). Furthermore, in order to display that the integrated contents requested by the user has been received, the integrated content metadata decryption/content usage control unit 112 interprets the received integrated content metadata and then transmits the context of the integrated contents and the rights to use those contents to the user interface unit 116.
It will be understood that the processes shown in
Accordingly, the multimedia server system can extract contents having an integrated multimedia file format structure from the user terminal according to the user's request or receive contents having an integrated multimedia file format structure from the contents service system connected therewith, the integrated multimedia file format structure being generated by the content server system.
While the invention has been shown and described with respect to the preferred embodiments, it will be understood by those skilled in the art that various changes and modification may be made without departing from the scope of the invention as defined in the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2005-0080818 | Aug 2005 | KR | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/KR2006/003464 | 8/31/2006 | WO | 00 | 2/28/2008 |
