Patent Application
20120128334
1. Field of the Invention
The present invention relates to an apparatus and method for providing a mashup function. More particularly, the present invention relates to an apparatus and method for providing a language and a file format for mashup of multimedia content.
2. Description of the Related Art
The term “mashup” refers to the integration of content, obtained from disparate sources, to create new data that supports a new functionality. By the integration or hybridization of information, which includes the reuse and mixing of available content through associations and recombination, new and enhanced services can be provided to users. For example, an enterprise mashup may integrate real-time commodities performance data with an analytical tool to allow a security trader to track a large number of portfolios. As another example, a consumer mashup may combine user location information with a search tool that locates other users or desired locations within a certain radius of the user, or may combine user location information with weather information, train schedules, information from a Social Networking Site (SNS), and the like.
Referring to
Referring to
The mashup of information described above can be achieved using programming code as defined in the Enterprise Mashup Markup Language (EMML) specification, provided by the Open Mashup Alliance (OMA), and as defined by the Hypertext Markup Language (HTML) 5 specification, provided by the World Wide Web Consortium (W3C). The OMA is a non-profit consortium whose goal is to facilitate the successful adoption of mashup technologies as well as to standardize the EMML specification so as to increase interoperability and portability of mashup solutions. Key features of the EMML specification include the ability to integrate data from multiple sources, support database queries (e.g., search for a customer name with a particular customer identification, search for movies with specific titles, etc.), invoke services (e.g. Facebook, Twitter, Really Simple Syndication (RSS) feeds, etc.), and support operations for transforming and formulating data (e.g., filter, sort, join/merge, etc.). Table 1 provides a list of EMML operations/functions.
As can be seen in Table 1, the EMML specification supports database queries and provides a mechanism for acquiring and integrating data from multiple locations and for transforming and formulating the acquired results. However, while it is useful for data analysis, it is not designed to handle multimedia content such as video, audio, and the like.
The HTML5 specification supports a geo-location Application Programming Interface (API) and allows geographical location information to be queried and combined with other data from disparate sources to create a new location-aware mashup service. As in the above illustrated examples, HTML5 can be used to combine location information with a search made on an SNS to search for other users within a certain radius of a user's current location.
HTML 5 also provides for embedding of video information. However, it is limited in this capability because it lacks support for allowing precise synchronization and interactivity among multimedia components. Furthermore, there is not an existing mashup mechanism that provides a method to specify mashup behavior according to a transport system environment or terminal characteristics.
Therefore, there is a need for an advanced and flexible method to specify mashups for multimedia content and to support synchronization and interactivity among multimedia components so as to provide sophisticated multimedia services to users.
There is also a need for a file format for storing and distributing multimedia mashup content. In this case, the term “file format” refers to a mechanism for encoding information for storage in a file. For example, certain file formats are designed to store a particular type of data (e.g., .PNG for storing bitmapped images, .WAV for storing audio data, etc.). However, there is not currently a file format for storing and distributing multimedia mashup content.
The International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) has standardized the ISO Base Media File Format (ISOBMFF), which is a file format designed to contain timed multimedia information (video, audio, etc.) for presentation in a flexible, extensible format to facilitate interchange, and easy management of media content. In this regard, the Moving Picture Experts Group (MPEG)-4 Part 14 File Format (MP4 FF), which is derived from the ISOBMFF, is provided for storing different media types in an MPEG-4 presentation. The MPEG-4 presentation can be composed of video and audio streams multiplexed together as well as metadata (such as subtitles), images, synthetic objects, and the like. However, even though the MP4 file format is capable of storing multimedia content and objects, as well as the relationships and interactivity among these objects, it does not provide support for connecting to databases for retrieving external content and does not store descriptions of how the multimedia content could be mashed up/combined.
The MPEG-21 FF is another standardized file format derived from the ISOBMFF. The MPEG-21 standard provides an open framework for delivery and consumption of multimedia content and ensures interoperability among users to exchange, access, and consume digital items. A digital item can refer to, for instance, a music album, a video collection, a web page, etc. The MPEG-21 FF provides a standardized method of storing descriptions of content in a file as well as a method to reference external data from multiple sources, which is a desirable property of mashup content. However, the MPEG-21 FF does not provide a flexible method of specifying the use of mashup data (i.e., specify behavior of mashups), nor does it allow precise description of temporal and spatial composition of multimedia content, which is critical for mashup of multimedia content.
Thus, there is also a need for a file format that provides for storage of mashup related information and for specifying mashup behavior of the content.
Aspects of the present invention are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of the present invention is to provide an apparatus and method for mashup of multimedia content.
Another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that allows content to be accessed and queried from multiple sources, databases, and the like.
Yet another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that supports multiple services, applications, and the like.
Still another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that enables spatial and temporal composition of multimedia content components.
Another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that allows precise synchronization among multimedia elements.
Yet another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that provides flexibility when specifying how content should be mashed up (i.e., flexibility in controlling mashup behavior).
Still another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that provides for re-mix of available content to produce new content, thus alleviating the need to re-create content.
Another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that allows easy creation of customized content.
Yet another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that enables recombination of portions/fragments of an entire content (i.e., fine-granularity mashups).
Still another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that provides flexible operations of content (e.g., filter, merge, sort, etc.) to easily transform and formulate mashup results for presentation to a user.
Another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content that can specify mashup behavior according to a terminal profile.
Yet another aspect of the present invention is to provide an apparatus and method for mashup of multimedia content by which service providers can adaptively specify mashup behaviors according to transport system profiles.
In accordance with an aspect of the present invention, a method for combining multimedia data is provided. The method includes obtaining first multimedia content from a first source, obtaining second multimedia content from a second source, selectively combining the first multimedia content with the second multimedia content, and outputting the selectively combined multimedia content.
In accordance with another aspect of the present invention, an apparatus for combining multimedia data is provided. The apparatus includes a transceiver unit for obtaining first multimedia content from a first source and for obtaining second multimedia content from a second source, and a control unit for selectively combining the first multimedia content with the second multimedia content and for controlling output the selectively combined multimedia content by the transceiver.
Other aspects, advantages, and salient features of the invention will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses exemplary embodiments of the invention.
The above and other aspects, features, and advantages of certain exemplary embodiments of the present invention will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:
Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures.
The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of exemplary embodiments of the invention as defined by the claims and their equivalents. It includes various specific details to assist in that understanding but these are to be regarded as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. In addition, descriptions of well-known functions and constructions are omitted for clarity and conciseness.
The terms and words used in the following description and claims are not limited to the bibliographical meanings, but, are merely used by the inventor to enable a clear and consistent understanding of the invention. Accordingly, it should be apparent to those skilled in the art that the following description of exemplary embodiments of the present invention are provided for illustration purpose only and not for the purpose of limiting the invention as defined by the appended claims and their equivalents.
It is to be understood that the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to “a component surface” includes reference to one or more of such surfaces.
By the term “substantially” it is meant that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations or variations, including for example, tolerances, measurement error, measurement accuracy limitations and other factors known to those of skill in the art, may occur in amounts that do not preclude the effect the characteristic was intended to provide.
The term “mashup” refers to a technology that allows easy creation of new data and services by combining or mashing-up existing data and services. Exemplary embodiments of the present invention provide a mechanism to effectively create and deliver mashup of not just data, but of multimedia content in a flexible and cost-effective way. That is, exemplary embodiments allow for the re-use of already available content or services that involve more sophisticated/interactive media content, thus user experience is enhanced at a reduced cost. Also, exemplary embodiments of the present invention provide a mechanism to efficiently store and deliver multimedia mashup content as well as meta-information used for specifying how content could be mashed up (i.e., mashup behavior).
In more detail, exemplary embodiments of the present invention include an apparatus and method for seamless integration/mashup of multimedia content from multiple sources. To support this novel feature, exemplary embodiments of the present invention provide a new markup language, designated as Multimedia Mashup Markup Language (M3L), which allows multimedia content to be accessed and processed from multiple sources, and to be easily combined and formulated for presentation to a user. Moreover, exemplary embodiments of the present invention allow a user to specify how the multimedia content could be mashed up in a flexible manner (i.e., allow a user to more flexibly specify the behavior of a mashup). Furthermore, exemplary embodiments of the present invention allow for precise timing control among the multimedia elements to provide interactive multimedia mashup content and services. The description below first provides exemplary embodiments of the present invention and next provides a description and example of M3L.
In addition to the flexible hybridization of multimedia content from any source, an exemplary embodiment of the present invention provides a mechanism to allow spatial and temporal composition of multimedia elements. The exemplary mechanism disclosed in the present invention provides the mashup content and service authors with, among other things, the ability to precisely prescribe the synchronization and interactivity among multimedia components to create innovative and sophisticated multimedia services for the users.
Referring to
As illustrated in
Of course, although
Referring to
As illustrated in
Of course, the example provided in
Another exemplary embodiment of the present invention provides a mechanism for a mashup service provider or content creator to specify a different mashup behavior depending on the characteristics of a transport system. For example, the mashup provider may consider whether the transport system uses a hybrid delivery, uses a streaming system, uses a broadcast system, etc. This allows the mashup service provider to take advantage of different characteristics associated with each transport profile. For instance, if the mashup content is being delivered using a hybrid delivery that includes two delivery channels, such as a broadcast channel and a broadband channel that provides additional bandwidth and interactivity, a content creator can use the present invention to specify mashup behavior and content that can leverage the extra bandwidth and interactivity provided by hybrid delivery but, when the transport system is broadcast only, the mashup behavior can be limited to that supported by the smaller bandwidth and without interactivity.
In yet another exemplary embodiment, the present invention allows the service providers and content creators to specify mashup behavior according to the profile of a terminal on which the mashup will be received. For example, mashup content consumption on a smartphone can be specified differently than when the same content is consumed on a high definition television, which has a larger display screen. The mashup behavior specified for the latter can take advantage of the larger screen size to create a different mashup content consumption experience.
Table 2 provides a list of elements and attributes and detailed M3L syntax. These elements and attributes allow users to specify multiple content sources, as well as fragments of a source (i.e., less than the entire content). Multiple services can be invoked to derive relevant results that can be merged and presented to the users in a flexible manner. Users can specify how the sources should be mashed up (mashup behavior) to create new content/services as well as precise spatial and temporal layout of the mashed up multimedia elements.
In Table 2, the element <m31> represents the root element of an exemplary mashup script. In implementation, there should be one such element at the beginning of each script. This element has several child elements, including <content>, <service>, <invoke>, <query>, and <sql>.
The <content> element is used to declare various content sources. In an exemplary implementation, it includes attributes for referencing either the entire content or fragments/portions of the entire content to be mashed up. More specifically, the <content> element includes attributes of “id”, “url”, “offset”, “duration”, and “to”. The attributes “id” and “url” (i.e., identifier and uniform resource link) are used to uniquely identify the content sources and reference their locations. The attributes “offset”, “duration”, and “to” are timing attributes to allow flexible mashup of different segments of the content from different sources or multiple, disjointed segments from the same source.
The <service> element is used to specify one or more services and the <invoke> element is used to invoke or launch a specified service. The <invoke> element includes attributes of “url” and “result”. The “url” attribute is used to identify a location or a pointer of a service to be invoked, and the “result” attribute is used to facilitate the storing of a result of an invoked service.
The <query> element is used to execute a database query by specifying a database source and credentials. The <query> element includes attributes of “src”, “username”, “password”, and “db name”. The “src” attribute is used to designate a location/path to the database server, and the “db name” attribute is provided for designating a specific database. The “username” and “password” attributes are provided for supplying credentials used for identification and security purposes.
The <sql> element is provided to initiate a query command and includes a “string” attribute for containing the search command.
Referring to
In the example of
Similar to the first <content> element 401, the second <content> element 402 includes an “id” attribute to designate a second video fragment (i.e., “video1_seg2”), an “src” attribute to designate a specific link from which the second video fragment will be taken (i.e., http://www.videostore.com/news1.mpg), and an “offset” attribute to establish an offset from a starting time of the video at which the fragment will begin (i.e., offset=“500s”). However, the second <content> element 402 includes a “to” attribute to designate a stopping time of the second video fragment (i.e., to =“600s”) rather than a “duration” attribute. In this case, the second video fragment is identified as “video1_seg2” and consists of the 500th to 600th seconds of the “news1.mpg” file.
Similar again to the first <content> element 401, the third <content> element 403 includes an “id”, an “src”, an “offset”, and a “duration” attribute. However, the “duration” attribute of the third <content> element 403 designates a duration of the third video segment in terms of percentage, rather than time. That is, the third <content> element 403 designates that the third video fragment, identified as “video2_seg”, will begin from the 30th second of the news2.mpg file and have a duration equal to 60% of the news2.mpg file.
Of course, it is to be understood that the elements illustrated in
The remaining code 405 of the illustrated software script 400 includes Synchronized Multimedia Integration Language (SMIL) elements and attributes that are used for playback of the video fragments designated by the first through third <content> elements 401˜403. SMIL is standardized by the World Wide Web Consortium (W3C) and allows precise synchronization and timing control among multimedia elements. As illustrated in
Although not illustrated in
Referring to
Referring to
Referring to
The control unit 701 is provided to receive user input through the input unit 707, to receive multimedia content from distinct sources through the transceiver 705 in response to user input, to store and retrieve necessary information from the memory unit 709, and to output information to the display unit 703. More specifically, the control unit 701 is provided to receive user input through the input unit 707. According to an exemplary implementation, the user input may be a request to execute a program that includes the mashup of multimedia information. For example, as described above with reference to
The input unit 707 may include a key pad, a mouse ball, or other input means by which a user is able to select and control desired operations of the apparatus 700. The display unit 703 is provided for displaying a signal output by the control unit 701. For example, the display unit 703 may output a result of a mashup of multimedia content, such as the selected news topic as illustrated in
The transceiver 705 is controlled by the control unit 701 to transmit a request for and to receive desired information, such as multimedia content. That is, the transceiver 705 communicates with external devices such as an Internet provider, and may include a wired and/or wireless connection. The transceiver 705 may, for example, include a Radio Frequency (RF) transmitter and receiver for wireless communication. The transceiver 705 may also support communication via a variety of protocols, including Bluetooth, Wi-Fi, cellular service (e.g., 3rd Generation and 4th Generation services), Ethernet, Universal Serial Bus (USB), and the like. The memory unit 709 may include both a volatile and a non-volatile memory for storing various types of information. For example, the memory unit 709 may include a volatile memory for storing temporary data that is generated during the execution of various functions of the control unit 701. Additionally, the memory unit 709 may include a non-volatile memory for storing data such as a program for executing an M3L file, multimedia content that is received and configured by the control unit 701, multimedia data that is the result of a mashup of received multimedia data, and the like.
Certain aspects of the present invention can also be embodied as computer readable code on a computer readable recording medium. A computer readable recording medium is any data storage device that can store data which can be thereafter read by a computer system. Examples of the computer readable recording medium include Read-Only Memory (ROM), Random-Access Memory (RAM), Compact Disc (CD)-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. Also, functional programs, code, and code segments for accomplishing the present invention can be easily construed by programmers skilled in the art to which the present invention pertains.
While the invention has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents.
This application claims the benefit under 35 U.S.C. §119(e) of a U.S. Provisional application filed on Nov. 19, 2010 in the U.S. Patent and Trademark Office and assigned Ser. No. 61/415,708, and of a U.S. Provisional application filed on Nov. 22, 2010 in the U.S. Patent and Trademark Office and assigned Ser. No. 61/416,123, the entire disclosure of each of which is hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61415708 | Nov 2010 | US | |
| 61416123 | Nov 2010 | US |
