Patent Application
20150350714
The present disclosure relates to caching and streaming of video content and, in particular, to techniques for predictively delivering and/or acquiring content for instantaneous viewing.
Currently, network-based media delivery services are available that support delivery of coded video. Those media delivery services typically code an input video sequence (“media stream”) as coded video data that has been parsed into a plurality of separately deliverable segments. Each segment may represent a portion of the source media stream, for example, a five or ten seconds increment of the media stream. The media delivery service may include an HTTP server that responds to requests from a client, and furnishes the coded segments in response to those requests. The requests may identify requested segments by an address, such as a uniform resource locator (commonly, “URL”). A common server may respond to service requests from a number of different client devices. The request may also be a single request from a gateway or representative of a group of client devices, for example the request may be made by a router connecting several client devices in a local area network (“LAN”).
Media, such as video, can be delivered in various manners. One form of media delivery is streaming and downloading media content substantially simultaneously, which allows for immediate playback but has compromised video quality. For example, if download speeds do not meet playback demands, the playback may stall. Alternatively, the media stream can be entirely downloaded, then played, which provides high quality playback, but typically involves a delay in beginning playback of the video, because sufficient frames must be downloaded before the video can be played.
Furthermore, each requesting client is typically provided with its own copy of a coded segment, and thus, support of multiple requests for the same coded segment can consume unnecessary bandwidth within a network between the server and the client(s). These challenges are exacerbated by the rising popularity of high-definition streaming video.
Embodiments of the present invention provide techniques to distribute video content among a plurality of associated devices by identifying segments associated with the video content. For each identified segment, if it is determined that the segment is stored by one of the associated devices, other associated devices requesting the segment may retrieve the segment from the device storing the segment. Otherwise, the segment may be delivered directly to the device requesting the video content.
By perceiving a need in the art for a media delivery system that adapts to the bandwidth of recipient clients, the inventors have developed a method and system for predictively selecting, delivering, downloading, and caching content, which can be played in real time without (or with a minimal) initial delay in playback, while maintaining the quality of the content. The media delivery system balances a trade-off between the quantity and quality of content delivered, by selecting a quality level of content that best matches the bandwidth available for transmitting the content. The system may also efficiently utilized a given bandwidth to deliver content more likely to played by a client, so that bandwidth is not wasted on downloading videos that will not be played by the client or is not immediately needed by the client. For example, if download speeds are relatively low, the media delivery system can prudently select a more relevant or popular video clip or deliver a version of content that is of lower quality, so that a larger portion of a video clip may be downloaded in a given period of time compared with a higher quality version of the video clip. The media delivery system may also exploit connections between client devices to send a video segment to a client device, which the client device may then share with neighboring client devices.
The media source 110 may include a server or network of servers (not shown) 112 and a database 114 that stores a manifest file 114.1, an optional client log 114.2 and coded video files 116, 118. The manifest file 114.1 may store data representing the video files that are available for retrieval. The video files 116, 118 may represent individual programs that are stored by the media source, for example, movies, television programs, music videos and the like. The coded video files 116, 118 may include a manifest file 116.1, 118.1 (which may be part of or separate from manifest file 114.1) and a plurality of addressable segments 116.2-116.m, 118.2-118.n. The manifest files 116.1, 118.1 may contain a list of the segments for the associated video file 116, 118 and network addresses (for example, uniform resource locators) where those segments may be requested. Each segment 116.2-116.m, 118.2-118.n may represent a corresponding portion of the media stream, for example a five minute increment of the media stream.
Although not illustrated in
The source server(s) 112 may field requests from the clients 130.1, 130.2 and may furnish data in response to those requests. A client device, for example, may request data that describes the video programs stored by the media source 110; in response, the server 112 may furnish data from manifest file 114.1, which may describe topical information the video files 116, 118, for example, title, length. Alternatively, the client may request data regarding a single video file 116; in response, the server 112 may furnish data from manifest file 116.1, which provides additional data regarding the video file 116, for example, coding parameters and addresses of individual segments 116.2-116.m. A client may request an individual segment (say, segment 116.3), in which case the server 112 may furnish the requested segment 116.3. In doing so, operation of the media source 110 may be subject to other control processes that are not relevant to the present discussion, such as authentication of client devices, processing of payments for content, parental controls and the like.
The clients 130.1, 130.2 may receive, decode, and/or play media content.
The receiver 132.1 may receive a coded video segment and may store it in the cache 134.1 and/or in the coded picture buffer 136.1. The cache 134.1 may store video segments which, when decoded, may form at least part of a playable video clip. The coded picture buffer 136.1 may also store video segments. In an embodiment, the coded picture buffer stores video samples immediately usable for decoding, while the cache 134.1 stores pre-fetched data before decoding or overflow segments.
The video decoder 138.1 may operate according to any of a number of different coding protocols, including, for example, MPEG-4, H.263, H.264, and/or H.265 (HEVC). Each protocol defines its own basis for defining pixel blocks and the principles of the present invention may be used cooperatively with these approaches. The video sink 144.1 may consume the coded video data, typically by rendering the data on a display or perhaps storing it for later use.
The subscription log 146.1 may include settings associated with the client 130.1, for example, a listing of video content to which a client (or a user of the client) has subscribed. The neighbor log 148.1 may store a list of neighbor clients (say, client device 130.2) that are associated with the client 130.1, including, for example, attributes such as cache sizes and downloading capabilities of the associated neighbor client. The controller 142.1 may control overall operation of the client 130.1.
According to an embodiment, clients 130.1, 130.2 may accelerate download and rendering of video files by exchanging segments among themselves prior to delivery from a media source 110. For example, when a client device 130.1 is engaged to begin rendering of a newly selected video file 116, the client device 130.1 may determine, with reference to its subscription log 146.1 and neighbor log 148.1, whether segments of the video file can be retrieved from another client 130.2 with which it is associated. The other client 130.2 may have prefetched a segment 116.2 of the video file and may be positioned to deliver the segment 116.2 to the client device 130.1 faster than could the media source 110.
In an embodiment, which clients share a cache may be determined by a shared user account or linked user accounts. For example, members living in the same household each having their own device may share a user account for a software application for downloading and streaming video. In a further embodiment, which clients share a cache may be defined by the neighbor log of each client. For example, if a client is in communication with a neighbor client and meets a threshold downloading suitability, the client may share a cache with the neighbor client. Two members of the household that watch the same television show may share a cache, so that the video is downloaded when the first member watches the television show, and is then readily available in the cache and easily retrievable by the second member over a LAN instead of from the media source when the second member desires to watches the television show later.
In another embodiment, the neighbor log 148.1 may be periodically updated to reflect changes in connectivity with neighbor devices. For example, devices may leave a LAN, and such a change may be reflected in the neighbor log. The subscription log 146.1 and the neighbor log 148.1 may each be in communication with the controller 142.1, and provide data for determining queries that the client sends to media sources to reflect the preferences of the client 130.1 and to optimize downloading of content as described further herein.
In a further embodiment, each client 130.1, 130.2 may have its own cache 134.1, 134.2. Caches may collaborate to store material and may communicate with each other to optimize downloading of material that may be shared among clients. In an alternative embodiment, one or more clients 130.1, 130.2 may access a single cache. Sharing cache, whether by using a single cache between multiple devices or accessing caches corresponding to different devices, reduces the consumption of bandwidth between clients and a server, because a single copy of a video segment may be downloaded to the cache, and the clients may then locally retrieve the content from neighboring devices.
For example, in operation, client 130.1, 130.2 may share a cache. Client 130.1 may have stored in its cache segments 116.3 and 116.5, but not segments 116.2 and 116.4. Client 130.2 may have stored in its cache segments 116.2 and 116.4, but not segments 116.3 and 116.5. Thus, the clients would benefit from sharing their respective cached segments to form a complete media stream. To play an entire media stream, client 130.1 may obtain segments 116.2 and 116.4 from client 130.2 so that it has all of the segments 116.2 to 116.4 for playback. In another example, client 130.1 may have stored in its cache the first half (segments 118.1 and 118.2) of a media stream 118, while client 130.2 may have stored in its cache the second half (segments 118.3 and 118.4) of a media stream 118. The clients may collaborate to form a complete media stream by sharing cache.
In
The methods described herein may also find application in service infrastructures that uses multicast or hierarchical server caching. For example, a local server that provides content to a number of users may determine which movies to cache based on common interests of the users it serves.
If the segment is not in the cache, the method 200 may determine whether there are any other clients associated with the first client (box 210). If there are no nearby devices, the method 200 may query a media source 110 for the segment (box 216). If there is at least one nearby device, the method 200 may determine whether the requested segment is available for download from another client associated with the first client (box 212). If the requested segment is available for download from another client, the method 200 may request the segment from the other client (box 214). The method 200 may advance to box 208 and begin decode and playback of the downloaded segment. If the method 200 determines in step 212 that the segment is not available from another device, the method 200 may query a media source 110 for the segment (box 216), and upon receipt of the segment from the server, decode and play the segment (box 208).
The method 200 may repeat steps 204-216 to continue downloading additional video segments until an entire coded video file is downloaded. For example, an entire coded video file 116 may be made up of segments, for example, segments 116.2 to 116.5. In an example, each media stream may represent a complete movie and each segment may represent a portion of the media stream, for example a thirty-second increment of the media stream. Client 130.1 may have stored in its cache segments 116.2 and 116.4, but not segments 116.3 and 116.5. Client 130.2 may have stored in its cache segments 116.3 and 116.5, but not segments 116.2 and 116.4. Clients 130.1 and 130.2 may be determined to be neighboring devices with a connection that is suitable for sharing cache in step 208 according to the techniques further described herein. Thus, while the video is being played back by client 130.1, steps 204-214 may be performed to continue downloading segments 116.3 and 116.5 from client 130.2 without delaying (or with minimal delays to) playback. The client may also pre-fetch segments before they are needed for decoding to stay ahead of playback. Examples of the order in which the client may pre-fetch the segments is further discussed herein with respect to prioritization of segment download.
According to method 200, although client 130.1 does not have segments 116.2 and 116.4, client 130.1 may obtain segments 116.2 and 116.4 from nearby client 130.2 so that at least a contiguous portion of the segments making up video 116 can be assembled, decoded, and played. In another example, client 130.1 may have stored in its cache the first half (segments 118.1 and 118.2) of a coded video file 118, while client 130.2 may have stored in its cache the second half (segments 118.3 and 118.4) of the coded video file 118. Client 130.1 may retrieve segments 118.3 and 118.4 from nearby device 130.2 to form at least a portion of a media stream needed to begin playback. Similarly, while the video is being played back by client 130.1, steps 204-216 may be performed to continue downloading segments 118.3 and 118.4 from client 130.2 without delaying (or with minimal delays to) playback.
The determination of whether a video segment is available for download from another client can be done in a variety of ways. In a first embodiment, the method 200 may consult a neighbor log 148.1 to determine whether there are any active nearby neighbors. If there are active neighbors, the method 200 may then request data from the neighbors. If there are no active neighbors, the method 200 may download a new neighbor log or updates to a neighbor log responsive to a query to the server (box 216).
The neighbor log 148.1 may store a list of neighbor clients (also “nearby clients”) in communication with the client 130.1, including, for example, attributes such as accessibility, range, and downloading capabilities of the associated neighbor client. The neighbor log 148.1 may be periodically updated to reflect changes in connectivity with neighbor devices. For example, devices may leave a LAN, and such a change may be reflected in the neighbor log.
In a further embodiment, the video segment may be downloaded based on evaluations of the suitability of nearby devices for providing the segment. In such an embodiment, a requested segment may be downloaded if it is both available in the cache of a nearby device, and the nearby device is accessible, within range, and capable. For example, the method 200 may determine whether a nearby device is accessible, within range, and capable. The accessibility of a nearby device may be determined based on recent communications between the nearby device and the client device 130.1. For example, the client device 130.1 may periodically ping a nearby device and log the time of a response of the nearby device indicating that the nearby device was still active. A nearby device may be considered accessible if it was in communication with the client device 130.1 within a pre-definable time period before the present time. The accessibility of the nearby device may also be based on whether a nearby device is communicatively coupled to the client device 130.1. For example, in a BLUETOOTH network, the client device 130.1 may ping a nearby device to determine whether the two devices are paired.
The range of a nearby device 130.2 may be a quantification of a download and/or upload speed between the device 130.1 and the nearby device 130.2. For example, a range can be an amount by which the download and/or upload speed exceeds a threshold value. A bandwidth between the clients may be considered sufficient if an estimated time to download a segment is below a threshold.
The capability of the nearby device 130.2 may be a measure of the connection between the nearby device 130.2 and the media source 110. For example, it may measure download and upload speeds between the nearby device 130.2 and the media source 110. Because devices may be communicatively coupled to the media source over different types of connections and each may use different communications standards and have different hardware configurations, the capabilities of one device may be different from the capabilities of another device, even if they are both in the same LAN. The nearby devices may be ranked according to their accessibilities, capabilities, and ranges relative to the device.
In an embodiment, if multiple devices associated with a client are suitable candidates from which the requesting device can download a segment, the requesting device may download the segment over the best connection. For example, a ranking of the nearby devices in the neighbor log may be used to determine the best connection. The selection may alternatively be based on the nearby device that is, on average, the most accessible, within range, and capable. The selection may alternatively be based on a weighting of each of the factors.
It is also possible that a segment is available in nearby devices, but it would be more efficient to obtain a segment directly from a server instead of obtaining the segment from a nearby device. For instance, the bandwidth between the devices 130.1 and 130.2 may be poorer than the bandwidth between a device 130.1 and the media source 110. For example, two nearby devices may be coupled via a weak BLUETOOTH connection, which may not be as fast as an Internet connection by which each device is connected to the server. In this situation, the segment may be requested directly from the server. A method by which a server may select and push a segment to the client device for downloading is described in further detail in relation to
In a first step 302, the method 300 may receive a query for video data. For example, the request may be received from a client device. In an embodiment, the request may be for a specific video segment, and the source may find the requested segment in its coded video files 114. In an alternative embodiment, the request may be for a media stream or a portion of a media stream, and the server may determine the appropriate segment to provide to the requesting client. For instance, a media stream may be formed by several segments, and the source may determine the next segment that is needed by the media stream. The source may also queue a several segments to push to the client, for example, according to the prioritization techniques described further herein.
When the appropriate segment is identified by the server, the method may proceed to push the segment to the client, either directly or through a nearby device. In an embodiment, the method 300 may push the requested segment to the requesting client, along with a neighbor or an update to a neighbor log (box 310). The contents of the neighbor log are further discussed herein, and may be used by the client for obtaining other segments from its nearby devices, for example, according to method 200.
In an alternative embodiment, method 300 may determine whether there are any devices associated with the requesting device (box 304), and if so, whether any nearby devices already have the requested segment (box 306). If any nearby devices already have the segment, then the method 300 client may direct the requesting client to retrieve the data from its neighbor (box 310). Otherwise, if the segment is not in the cache of any associated devices, the method 308 may proceed to provide the segment and a neighbor log (or update of the neighbor log) to the requesting client (box 308).
A group of client devices may be considered “nearby,” if the devices are communicatively coupled to each other, for example all in the same LAN or together form a BLUETOOTH piconet or scatternet. The method 300 may provide updates to the neighbor log regarding the nearby devices, e.g., nearby devices that have left or joined the LAN and their respective attributes. A list of the nearby devices may be maintained in the client log 114.2 of the source 110. The method 300 may also periodically determine whether client devices have joined or left a local network of client devices and log this information in the client log 114.2. The characteristics of each client, e.g., such as accessibility, range, and downloading capabilities, may be provided to the server 110 over a network, such as the one illustrated in
Information about the proximity of client devices to each other and their performance specifications may be used by the method 300 to determine an order in which to provide video segments and which segments are provided to which clients. This download optimization allows a bit stream to be built in a scalable way such that, even given bandwidth constraints, a client device can smoothly stream a video. Given a bandwidth of a group of client devices, e.g., a LAN, a resolution of the video segments may be selected to enhance the viewing pleasure of the user.
The source server may split the requested content into several video segments and send various segments to different nearby clients to decrease the amount of time it takes to download the requested content. For instance, for a coded video file 116, the method 300 may send segments 116.2 and 116.4 to client 130.1 and segments 116.3 and 116.5 to client 130.2. The requesting client may then be directed to each of its nearby clients to piece together the requested content as described further herein.
In yet another embodiment, method 300 may be performed to provide content to client(s) absent a request from the client(s). For example, the server may queue up and push video segments to clients, and the video segments may be cached locally over time as they become available from the source server. For example, content may be pushed to clients for download while the client is inactive or when playback is paused. Alternatively, content may be pushed to clients while another video segment is playing.
The queuing up of video segments for download may be predictively performed based on popularity among all or a sub-set of users of a software application, a viewing history of a user of the client device, interests of the user of the client device, and/or a user's active choices, for example, as indicated by selections in a subscription log. For example, while a user is navigating a UI, user behavior may trigger caching. Content associated with a synopsis or trailer may be cached, e.g., while a user is watching a trailer or while a synopsis is displayed on the UI (the user is presumably reading the synopsis) or watching the trailer. In an alternative example, the predictive downloading is performed based on a combination of a user's interests and subscriptions. Information regarding a user's preference may be associated with a user account such that the preferences for a particular user may be associated with more than one client device. Users associated with one another, e.g., in physical proximity to each other, may be inferred to have similar interests. The user account may include information regarding all of the client devices that are used by the user.
Because a local cache size may be limited or available bandwidth to the cache may be limited, the order and/or duration of content loaded to the cache may be prioritized. Content may also be at least partly randomly cached. Additionally, the basis and/or manner for the prioritization for caching may be one or more of the following:
Partial download (e.g., one or a few video segments) of each item on a user's subscription list;
The beginning (e.g., the first minute) of each item on a user's subscription list may be cached, which may provide a quick start-up when viewing the items on the list;
A lower resolution version of each item in a user's subscription list may be downloaded before downloading a higher resolution version;
Higher resolution layers of a video sample may be downloaded at a lower priority than a lower resolution version: for example, download of higher resolution layers may begin once sufficient bandwidth and/or cache is available or may begin after a lower resolution version is partially or completely downloaded. For instance, when the method 300 determines that there is sufficient bandwidth during playback, enhancement layer representations can be downloaded and decoded on top of the cached base representation. The enhancement layers may relate to spatial, temporal (e.g., frame rate), and dynamic range aspects of the video sample;
The segments of video with higher bits that consume more bandwidth may be prioritized for caching to avoid pausing videos during playback in more bandwidth intensive segments. For example, a video decoder of the client (such as the decoder 138.1, 138.2 shown in
A usage pattern of the user, which may be learned over time: for example, videos that are in a similar genre to videos previously viewed by a user may have a higher priority for download;
Splash screens that may appear at the beginning, end, or elsewhere in a video: for example, a production logo, an opening logo, a closing logo, and/or copyright warnings;
A summary of a movie or a video: for example, a movie may be divided into chapters and one or a few frames from each chapter may be cached, and, when played, provide a summary of the entire movie. In an example, the summary may be made up of video snippets, the video snippets being frames shown during fast-forwarding through a video selection. In an alternative example, the summary may be made up of key frames or I-frames. The chapters may be provided by a movie distributor. For example, this information may be provided by the media source as metadata to a client. A viewer may view the summary or view an entire movie by viewing the cached summary while fetching the missing parts; and
A selection of advertisements, which may be shared among the clients of the LAN. Thus, a delay in starting playback may be eliminated by playing the commercials, and while the advertisements are playing, additional video segments may be downloaded.
The cache may be periodically cleared, for example by evicting segments for least recently watched and least frequently watched video clips. The replacement policies may provide for some of the prioritized cached content to remain in the cache while other content is evicted. In an example embodiment, while video playback is paused, the cache may be updated to clear out content that has already been played and content subsequent to the resume point may be downloaded.
Information regarding what material is available for immediate viewing may be provided on a UI of a client device. For example, the UI may display and categorize content that is available for immediate viewing at full resolution, various intermediate resolutions, and low resolution streaming quality. The determination of what content is available for immediate viewing may be based on what is available in the cache of the client devices in a LAN. Caching may be performed in an interactive and/or supervised way. For example, a user may filter and indicate which ones of the displayed system recommendations to carry out. As a further example, notifications may be sent to users and user feedback may be used to carry out the caching. Users may specify types of predictive caching, for example full, partial, summarization, and quality level. Users may also specify starting times of video content viewing, and the system may adaptively select video tiers based on a bandwidth between caches, a bandwidth for communications between the source server and the clients.
Caching may also be adapted to the attributes of the content being cached. For example, if the content is live, the caching may be performed without ads. In another example, content providers may indicate the types of video segments being provided (e.g., via metadata), and a client may adapt caching based on the information provided. In a further example, events of interest may be automatically detected on the client side, and caching may be performed based on these automatically detected events of interest.
A source server or client may provide suggestions on a UI based on the contents of cache. For example, the server or client may suggest movies that have been at least partially cached in a LAN. For example, an episode of a television series not yet viewed, but at least partially cached by the user may be recommended. The recommendation may be provided on the UI of the client device.
The foregoing discussion has described operation of the embodiments of the present invention in the context of client devices that functional units (
The foregoing description has been presented for purposes of illustration and description. It is not exhaustive and does not limit embodiments of the invention to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from the practicing embodiments consistent with the invention. Unless described otherwise herein, any of the methods may be practiced in any combination. For example, the methods of prioritizing video segments for download and pushing segments to various client devices may be practiced in any combination.
