Patent Application
20080168516
The present disclosure relates to media processing devices, and to systems and methods for streaming media content to a media processing device and for performing random access during streaming.
Media processing devices can be configured to play back media files that contain audio and/or video content. Early implementations required a user to download an entire media file before playback could be initiated. The amount of time required to begin playback depended on the size of the media file and the speed at which the media file could be transferred. More recently, a remote media file that is to be played back can be progressively downloaded from a remote storage device. During progressive downloading, the portion of a media file downloaded to a local media processing device can be played while one or more remaining portions of the media file are being downloaded. For example, the website www.apple.com/trailers hosts movie trailers, the contents of which can be progressively downloaded to a local media processing device and viewed using the QuickTime media player distributed by Apple Computer, Cupertino, Calif. Nonetheless, through progressive downloading, the media file is downloaded sequentially. Alternatively, media can be streamed over a streaming media server and played back on a local media processing device. Streaming media over a network, however, requires the support of a streaming protocol.
At the media processing device, the media file 10 can be saved to a storage device. During a sequential download, with respect to the file structure shown in
Hyper Text Transfer Protocol (HTTP) provides a set of conventions that can be used to transfer or convey information within the world wide web. HTTP is a request/response protocol between clients and servers. In HTTP version 1.0, requests are issued sequentially, with the next request being issued only after the response to the current request has been completely received. HTTP version 1.1, however, allows multiple requests to be issued simultaneously without waiting to receive responses to one or more outstanding requests. The ability to issue multiple, co-pending requests is referred to as pipelining. Additionally, HTTP version 1.1 also supports requests to access specific byte ranges, which permits non-sequential retrieval from a remote server of any part of a file that can be requested as a particular range of bytes. Further, a file can include a table or map describing its organization with respect to byte addresses, such as in a header.
A media processing device, such as a media client, can be configured to play back media content received over a network, such as from a remote media server. Further, the media client can be configured to perform a variety of playback functions and operations at the direction of a user, such as skipping forward, skipping backward, and randomly accessing a point in a media timeline. Many of these techniques and methods rely on configuring the media client to utilize a particular communication protocol and to structure requests for media content in accordance with user commands. In order to permit more flexible playback of media content and to reduce the delay associated with accessing different portions of media content, the present inventors recognized that it was beneficial to permit a media client to access and play back portions of media content non-sequentially.
The present inventors also recognized the need to reduce the delay experienced by a user in starting playback of media content, particularly from a starting point other than the beginning of a file. Further, the need to permit downloading the portions of a media file in a non-sequential order also is recognized. Additionally, the present inventors also recognized the need to permit non-sequential downloading of media content using HTTP 1.1. Accordingly, the techniques and apparatus described here implement algorithms for accessing, downloading, and playing back media content in a non-sequential manner.
In general, in one aspect, the techniques can be implemented to include accessing one or more items of media data associated with a media file stored on a remote server, wherein the media data includes at least one index; receiving input from a user identifying a playback location associated with the media file; determining a plurality of data items required to play at least a portion of the media file from the identified playback location based on the at least one index; and transmitting one or more byte-range requests to the remote server using the hypertext transfer protocol to retrieve the plurality of data items.
The techniques also can be implemented such that the plurality of data items are non-contiguously ordered in the media file. The techniques further can be implemented to include initiating playback of the media file from the identified playback location before the entire media file has been downloaded. Further, the techniques can be implemented such that the media file is comprised of a plurality of atoms, each atom including a size field, a type field, and a data portion. Additionally, the techniques can be implemented such that accessing one or more items of media data further comprises reading the type field of an atom included in the media file to determine whether the atom includes media data; retrieving the data portion of the atom if the atom includes media data; and accessing the beginning of the next atom based on a size indicated by the size field of the atom.
The techniques also can be implemented such that the data portion of an atom can comprise audio content, video content, or media data. The techniques further can be implemented such that transmitting one or more byte-range requests further comprises transmitting the one or more byte-range requests substantially simultaneously. Additionally, the techniques can be implemented such that transmitting one or more byte-range requests further comprises transmitting the one or more byte-range before a response to at least one previously transmitted byte-range request is received.
The techniques also can be implemented to include progressively downloading the media file from the identified playback location until the limit of the media file is reached, playback of the media file is terminated, or input is received from the user identifying a new playback location. Further, the techniques can be implemented such that the index comprises one or more sample tables.
In general, in another aspect, the techniques can be implemented as a computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations comprising accessing one or more items of media data associated with a media file stored on a remote server, wherein the media data includes at least one sample table; receiving input from a user identifying a playback location associated with the media file; determining a plurality of data items required to play at least a portion of the media file from the identified playback location based on the at least one sample table; and transmitting one or more byte-range requests to the remote server using the hypertext transfer protocol to retrieve the plurality of data items.
The techniques also can be implemented such that the plurality of data items are non-contiguously ordered in the media file. Also, the techniques can be implemented to be further operable to cause data processing apparatus to perform operations comprising initiating playback of the media file from the identified playback location before the entire media file has been downloaded. Further, the techniques can be implemented such that the media file is comprised of a plurality of atoms, each atom including a size field, a type field, and a data portion. Additionally, the techniques can be implemented such that accessing one or more items of media data further comprises reading the type field of an atom included in the media file to determine whether the atom includes media data; retrieving the data portion of the atom if the atom includes media data; and accessing the beginning of the next atom based on a size indicated by the size field of the atom.
The techniques also can be implemented such that the data portion of an atom can comprise audio content, video content, or media data. Further, the techniques can be implemented such that transmitting one or more byte-range requests further comprises transmitting the one or more byte-range requests substantially simultaneously. Additionally, the techniques can be implemented such that transmitting one or more byte-range requests further comprises transmitting the one or more byte-range requests before a response to at least one previously transmitted byte-range request is received.
The techniques also can be implemented to be further operable to cause data processing apparatus to perform operations comprising progressively downloading the media file from the identified playback location until the limit of the media file is reached, playback of the media file is terminated, or input is received from the user identifying a new playback location. Further, the techniques can be implemented such that the index comprises one or more sample tables.
The techniques described in this specification can be implemented to realize one or more of the following advantages. For example, the techniques can be implemented such that any portion of a media file stored on a remote server can be accessed and downloaded to a media client using one or more requests that specify a specific byte range. The techniques also can be implemented to permit the use of a plurality of simultaneous byte-range requests. Additionally, the techniques can be implemented such that one or more byte-range requests are transmitted from the media client to the media server in response to an input received from a user. The techniques also can be implemented such that media content received by the media client can be played back before the entirety of the media content has been received. The techniques further can be implemented such that a plurality of non-sequentially downloaded portions corresponding to a single media file can be sequentially ordered.
The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims.
Like reference symbols indicate like elements throughout the specification and drawings.
The media client 100 also includes a storage device 110 that can be configured to store information including media, configuration data, and operating instructions. The storage device 110 can be any type of non-volatile storage, including a hard disk device or a solid-state drive. For example, media received from an external media server can be stored on the storage device 110. The received media thus can be locally accessed and processed. Further, configuration information, such as the resolution of a coupled display device or information identifying an associated media server, can be stored on the storage device 110. Additionally, the storage device 110 can include one or more sets of operating instructions that can be executed by the processor 105 to control operation of the media client 100. In an implementation, the storage device 110 further can be divided into a plurality of partitions, wherein each partition can be utilized to store one or more types of information. Additionally, each partition can have one or more access control provisions.
A communication bus 115 couples the processor 105 to the other components and interfaces included in the media client 100. The communication bus 115 can be configured to permit unidirectional and/or bidirectional communication between the components and interfaces. For example, the processor 105 can retrieve information from and transmit information to the storage device 110 over the communication bus 115. In an implementation, the communication bus 115 can be comprised of a plurality of busses, each of which couples at least one component or interface of the media client 100 with another component or interface.
The media client 100 also includes a plurality of input and output interfaces for communicating with other devices, including media servers and presentation devices. A wired network interface 120 and a wireless network interface 125 each can be configured to permit the media client 100 to transmit and receive information over a network, such as a local area network (LAN) or the Internet. Additionally, an input interface 130 can be configured to receive input from another device through a direct connection, such as a USB or an IEEE 1394 connection.
Further, an output interface 135 can be configured to couple the media client 100 to one or more external devices, including a television, a monitor, an audio receiver, and one or more speakers. For example, the output interface 135 can include one or more of an optical audio interface, an RCA connector interface, a component video interface, and a High-Definition Multimedia Interface (HDMI). The output interface 135 also can be configured to provide one signal, such as an audio stream, to a first device and another signal, such as a video stream, to a second device. Further, a non-volatile memory 140, such as a read-only memory (ROM) also can be included in the media client 100. The non-volatile memory 140 can be used to store configuration data, additional instructions, such as one or more operating instructions, and values, such as one or more flags and counters. In an implementation, a random access memory (RAM) also can be included in the media client 100. The RAM can be used to store media content received in the media client 100, such as during playback. Further, media content can be stored in the RAM whether or not the media content is stored on the storage device 110.
Additionally, the media client 100 can include a remote control interface 145 that can be configured to receive commands from one or more remote control devices (not pictured). The remote control interface 145 can receive the commands through wireless signals, such as infrared and radio frequency signals. The received commands can be utilized, such as by the processor 105, to control media playback or to configure the media client 100. In an implementation, the media client 100 can be configured to receive commands from a user through a touch screen interface. The media client 100 also can be configured to receive commands through one or more other input devices, including a keyboard, a keypad, a touch pad, a voice command system, and a mouse.
Further, the media client 100 and the local media server 215 can include network connections 235 and 240 respectively, which provide access to a network 245, such as the Internet. In an implementation, the media client 100 can communicate with a remote media server 250 and/or a media store 255 over the network 245. For example, a connection can be established between the media client 100 and the remote media server 250. The connection can be secure or unsecure. Thereafter, the media client 100 can receive media content from the remote media server 250, such as by streaming or downloading.
Similarly, the media client 100 can be configured to receive media content from a media store 255. For example, upon establishing a connection, the media client 100 can request a list of available media content from the media store 255. The list of available media content can include free content, such as trailers and pod casts, and for-purchase content, such as movies, television programs, and music. Additionally, the media client 100 can be configured to communicate with the media store 255 to validate media content, such as by verifying digital rights management information.
Media content can be transferred from any remote server, such as the remote media server 250 or the media store 255, via one or more transmission protocols, including Hyper Text Transfer Protocol (HTTP). For example, the media client 100 can request media content from a remote media server 250. In response, the remote media server 250 can transmit the requested media content to the media client 100 using HTTP. The media content can be transferred using a plurality of data packets, which are separately transmitted over the network 245. Further, by utilizing HTTP version 1.1, the media client 100 can transmit a plurality of requests for media content to the media server. The plurality of requests can be transmitted simultaneously or close in time, such that a request for media content is not delayed until a response to a previous request is received. As a result, latency in the transfer of requested media content can be reduced. Additionally, by utilizing HTTP version 1.1, one or more byte ranges can be identified to request specific portions of media content.
The size field 420 of an atom can be m bytes long, where m is an integer. In an implementation, the size field 420 can be 4 or 8 bytes long. The size field 420 can specify the total number of bytes comprising the atom. The type field 425 of an atom can be n bytes long and can include a code. In an implementation, the type field 425 can specify an ASCII code comprising one or more ASCII characters. For example, the type field 425 can be 4 bytes long and the ASCII code can be MOOV. The data field 430 represents the data portion of the atom. For example the data field 430 can contain one or more types of media content, including video, audio, images, text, or any combination thereof.
The media client 100 can be configured to access the size field 420 of an atom, such as the atom 405. Further, based on the size field 405 of the atom, the media client 100 can compute the beginning of the next atom contained in the media file 400, such as the atom 410. The media client 100 can repeat this process to sequentially access one or more additional atoms, such as the atom 415. In this manner, the media processing device 100 can traverse the media file 400 on an atom-by-atom basis.
In an implementation, one or more atoms included in the media file 400 can correspond to a type of media content, as depicted in
The media client 100 can employ the byte range request features of HTTP version 1.1 to first obtain the media data, such as a sample table, included in a media file 400. Although a media file 400 is frequently ordered such that the media data appears at the beginning of the file, the media data can be located at any position in the file. For example, if the media data is located in bytes 30-39 of the media file 400, the media client 100 can issue one or more byte-range requests from the media server for bytes 30-39. Further, bytes 30-39 can be requested before requests for bytes 1-29 are issued. Additionally, the pipelining feature of HTTP version 1.1 can permit the media client 100 to request a plurality of data items simultaneously, without having to issue individual, consecutive requests. For example, the media client 100 can transmit temporally overlapping byte-range requests for data stored in different portions of the media file 400.
Additionally, HTTP version 1.1 can be used to access a media file that is larger than the available storage and/or memory included in the media client 100. For example, a media file with a file size of 2.0 GB can be played back even though only 100 MB of storage are available in the media client 100. The media client 100 can be configured to maintain in storage a high percentage of data associated with forward playback of the media file. Once all of the available storage, e.g. 100 MB, has been filled, the media client 100 can evict data associated with played portions of the media file at the same rate additional portions of the media file are downloaded. If the user performs an operation terminating forward playback, such as reverse seek, the portions of the media file required to complete the operation can be downloaded.
Once the media data describing the media file 400 has been retrieved, the media processing device 100 can initiate playback of the media file 400. In an implementation, the media client 100 can buffer a predetermined amount of the media content, including audio content and video content, before beginning playback. The predetermined amount of media content can represent a fixed amount of playback time at a standard playback rate or can be a proportional amount based on an average download rate, such as an amount that is projected to allow uninterrupted playback of all or a portion of the media file. Further, one or more parameters, such as the average download rate, can be periodically updated during playback. The media client 100 can be configured to begin playback in a default state, such as from the beginning of the media file, unless otherwise instructed by a user. Further, the media client 100 can commence progressive download of the media file 400 from the beginning of the file. In an implementation, the media client 100 also can be configured not to download information, such as the media data, that already has been downloaded to and/or stored on the media client 100.
The media client 100 can receive an input from a user, such as through a remote control or touch screen interface, requesting playback of the media file 400 at a location removed from the current playhead position, such that playback does not proceed sequentially. The location in the media file can represent a time with respect to the media file. Alternatively, the location can represent an offset of data within the media file. For example, indexing information can map media time to corresponding file offsets. In an implementation, the input can specify any playback position within the media file 400. Alternatively, the requested playback position can be specified relative to the current playback position, such as a specific increment forward or backward in the media file 400 For example, a remote control device can include a plurality of buttons, wherein each button can be configured to perform one or more functions. A command associated with one or more buttons can cause the media client 100 to skip (or “jump”) from the current playback position to a new playback location in the media file 400. When the skip occurs, a new start location 520 is identified corresponding to the beginning of the new playback location, as is shown in
In an implementation, the new start location 520 can be a function of the duration of the media file 400, which can be expressed in the media data associated with the media file 400. For example, the media client 100 can logically divide the media file 400 into a plurality of segments for playback. Further, the segments can be designated such that all of the segments have substantially the same duration. Upon sensing the input signal to commence playback at a new location, the media client 100 can be configured to skip forward or backward from the current playback location by a predetermined amount. The new playback location can then be used to identify the new start location 520. As content is downloaded from the new start location 520, the status bar 505 can be progressively filled with a download indicator 525, such as with a color or pattern. When sufficient data has been downloaded starting from the new download location 520, the media client 100 can resume playback of the media file 400.
In an implementation, the progressive download can be indicated on a display device and the download location can be selected using a pointer displayed on the display device. The pointer can be operated using a keyboard or a suitable pointing device (e.g., mouse, track ball, stylus, touch screen) to interact with the display device. The pointing device also can be operated by a near contact screen that employs a regional sensing field to detect objects in proximity with screen. In another implementation, the remote control device can be configured to include a touch screen interface. The touch screen interface of the remote control device can display the status bar 505 as it is displayed on the display device 500. The media client 100 also can be configured to display a pointer on the display device 500, the location of which can be controlled by the user through input provided to the remote control device. The pointer can be used to select a playback location. If a new playback location is selected, the media client 100 can commence downloading the corresponding portion of the media file 400.
Upon receiving input from a user to play back a different portion of the media file 400, the media client 100 can commence downloading the portion of the media file 400 associated with the requested portion. In
Subsequently, input received from a user can indicate that playback of a second portion of the media file 400 is desired. If the second portion of the media file 400 is not contiguous with the portion of the media file 400 represented by the first filled portion 605, content corresponding to the second portion can be downloaded. Thus, the media client 100 commences downloading the portion of the media file 400 corresponding to the second portion. As downloading of the second portion commences, the media client 100 can fill the status bar 505 from the second start location 610, generating a second filled portion 615. In an implementation, all other filled portions of the status bar 505 can be hidden and only the download indicator corresponding to the current playback location, i.e. the second filled portion 615, can be displayed. Although a different, non-contiguous portion of the media file 400 is being downloaded, the media client 100 can store the downloaded portion of the media file 400 represented by the first filled portion 605.
Additionally, input can be received from the user to indicate that playback of a third portion of the media file 400 is desired. If the third portion of the media file 400 is not contiguous with the portion of the media file 400 represented by the second filled portion 615, the media client 100 can begin downloading content corresponding to the third portion of the media file 400. As downloading of the third portion commences, the media client 100 can fill the status bar 505 from the third start location 620, generating a third filled portion 625. In an implementation, all other filled portions of the status bar 505 can be hidden and only the filled portion corresponding to the current playback location, i.e. the third filled portion 625, can be displayed. Although a different, non-contiguous portion of the media file 400 is being downloaded, the media client 100 can store the downloaded portion of the media file 400 represented by the second filled portion 615. Further, the portion of the media file 400 corresponding to the second filled portion 615 can be stored separately from the portion of the media file 400 corresponding to the first filled portion 605.
In this manner, the media client 100 can download one or more separate portions of the media file 400 based on input received from a user. The status bar 505 can be progressively updated to indicate the status of the portion of the media file 400 being downloaded. Each separate portion of the media file 400 downloaded by the media client 100 can be stored on the media client 100. Further, the relationship between the separate portions of the media file 400 and/or the relationship between a separate portion of the media file and the complete media file 400 can be maintained by the media client 100. In an implementation, the status bar 505 can persistently display each of the portions of the media file 400 that have been downloaded. For example, the first filled portion 605 and the second filled portion 615 can be reflected on the status bar 505 while the third filled portion 625 is being downloaded.
The media client 100 also can be configured to download as much of the media file 400 as possible during the time the media file 400 is being accessed by a user. For example, if downloading of the third portion of the media file 400 is completed before playback ceases and before a command is received from the user to access a different portion of the media file 400, the media client 100 can begin downloading any other portion of the media file 400 that has not yet been downloaded.
The media client 100 can be configured to select the next portion to download based on a variety of factors. For example, the media client 100 can be configured to downloading the remaining portions of the media file 400 sequentially. Alternatively, the media client 100 can be configured to prioritize the download sequence for the remaining portions of the media file 400 based on proximity to the portion of the media file 400 being played. For example, if the media client 100 is playing back the third portion of the media file 400, the fifth portion 635 could be selected for download before the fourth portion 630. In another implementation, the remaining portions can be prioritized for download based on one or more other factors, including size, the projected time remaining in the playback, and the ability to logically form a contiguous portion of the media file 400.
Further, the media processing device 100 can be configured to indicate when an additional portion of the media file 400 has been downloaded to form a logically contiguous portion. For example, once all portions have been downloaded, the status bar 505 can be filled in to reflect that the entire media file 400 has been stored locally.
In an implementation, the media client 100 also can be configured to download bi-directionally. For example, when a point in the media file 400 is selected, the media client 100 can commence sequentially downloading the content ordered after that point. Once the end of the media file 400 has been reached, the media client 100 can commence sequentially downloading the content ordered before that point, such that the media client 100 is downloading content in the direction of the beginning of the media file 400.
A number of implementations have been disclosed herein. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the claims. Accordingly, other implementations are within the scope of the following claims.
