Patent Application
20120143994
The present invention is related generally to data-delivery systems and, more particularly, to systems that send or receive media presentations.
More and more users are downloading more and more media presentations to more and more devices. (Here, “media presentations” generally include just about any kind of digital content, and, more specifically, sound, video, and interactive files.) These media presentations are often enormous, and downloading them can consume a significant amount of available bandwidth and battery power on the user's device.
In order to manage download requests, download servers often divide a large media presentation into consecutive “chunks” where each chunk represents, for example, a few seconds of video. When a user wishes to consume a media presentation, his device begins by requesting a “playlist” for the presentation from the download server. (Note that here “consume” is meant as a general term for any type of human interaction with a medium. It can include watching television, listening to radio, playing a computer game, talking or texting on a telephone, interacting with a web site, and the like. To simplify the present discussion, a media consumer is generally called a “user” or a “viewer,” even when his medium of choice does not have a visual portion.) The playlist includes a list of descriptions of the chunks into which the presentation is segmented on that server (including alternative resolutions). With the playlist in hand, the user's device asks the server to download the first chunk of the presentation. While the user is viewing the first chunk, his device attempts to “keep ahead” of the user's viewing (and thus avoid “video freeze”) by requesting subsequent chunks of the presentation. The chunks are received and buffered on the user's device so that the user can continue to view the media presentation while subsequent chunks are still being delivered.
It is, however, very common for a user to request a media presentation, begin viewing it, and then decide not to view the entire file. This wastes bandwidth and battery power on the user's device as chunks are sent that are never viewed. Also, the user may fast-forward (or skip) through parts of a media presentation looking for scenes of interest. (For example, the user may fast-forward through much of a soccer game looking for an interesting goal.) This fast-forwarding can also waste bandwidth because the presentation is often downloaded at a maximum possible resolution (unless otherwise specified) even though it would be perfectly acceptable to display to the user the fast-forwarded parts at a much lower resolution. (Of course, downloading a media presentation at low resolution saves significant bandwidth and battery power compared to downloading the same presentation at a higher resolution.)
The above considerations, and others, are addressed by the present invention, which can be understood by referring to the specification, drawings, and claims. According to aspects of the present invention, “importance” information is associated with each chunk (or at least with some chunks) of a media presentation. An end-user device, a download server, or a third-party server can use this importance information to more intelligently manage resources when downloading the media presentation.
Many different types of importance information may be used. For example, a human (or maybe electronic) editor can tag a chunk of a soccer game as important because that chunk includes a goal. The importance information given to the end-user device can then include a recommendation that this chunk be rendered at a higher-than-usual resolution and that the end-user device start downloading this high-resolution chunk out of order. Receiving this recommendation, the end-user device looks at its current (or predicts its future) bandwidth availability, and starts downloading the chunk before its natural place in the media presentation. That way, the high-resolution chunk is available when needed.
Similar techniques can be used in a very different situation. Because some download services are supported by advertising revenue, the service providers are very sensitive to the advertiser's needs. The advertiser may request that an advertisement be rendered at a resolution high enough for the end user to view it appropriately. The server can include a recommended minimum resolution for the chunk(s) of the advertisement as importance information for a media presentation during which the advertisement should appear. Instead of a minimum-resolution recommendation, the advertiser can request that the advertisement be seen in a resolution matching that of the surrounding media presentation, to prevent visual discontinuities. The end-user device receives the recommendations (along with other importance information) and responds accordingly by requesting the advertisement chunk(s) at a suitable resolution.
Unlike the chunks in a “normal” media presentation, an advertisement chunk does not have a “natural” temporal location at which it should be rendered. Thus, the importance information can include one or more recommended points at which to display the advertisement (e.g., between scenes in the media presentation).
If the advertisement consists of a number of chunks, then the importance information can note that fact, so that the end-user device does not begin rendering the advertisement until it has received at least a minimum number of the chunks in the advertisement.
Finally, the end-user device can take advantage of otherwise-unused bandwidth to download advertisements long before they are needed. Later, when the user requests the download of a media presentation, an already downloaded advertisement can be rendered to the user during the time that the initial chunks of the user-requested media presentation are being downloaded, thus preventing the boring “buffering” screen. The importance information of each advertisement can include characterizing information so that only appropriate advertisements are rendered before the beginning of the user-requested media presentation.
While the appended claims set forth the features of the present invention with particularity, the invention, together with its objects and advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which:
a and 3b together form a flowchart of a method for an end-user device to use (and, in some embodiments, to gather) importance information;
a and 4b together form a flowchart of a method for a server to provide media content and importance information;
a and 8b are graphs that show how intelligent use of chunk-size information can reduce video freeze;
a and 9b together form a flowchart of a method for a server to provide media content and importance information; and
a and 10b together form a flowchart of a method for an end-user device to use importance information.
Turning to the drawings, wherein like reference numerals refer to like elements, the invention is illustrated as being implemented in a suitable environment. The following description is based on embodiments of the invention and should not be taken as limiting the invention with regard to alternative embodiments that are not explicitly described herein.
Aspects of the present invention may be practiced in the representative communications environment 100 of
The servers 104, 106, 108 provide, via the networking technologies 102, media-download and related services to end-user devices. One example of an end-user device is a cellular telephone 110. This telephone 110 communicates wirelessly to a wireless base station (not shown but known in the art) to access the public switched telephone network, the Internet, or other networks to access the services provided by the servers 104, 106, 108.
Non-wireless end-user devices are supported by “wireline” network technologies (e.g., fiber, wire, and cable) 112. For example, a set-top box 114 generally receives television programs and provides a user interface (e.g., an interactive program guide) for selecting and viewing content from the cable provider. A digital video recorder (not shown) can store programming for later viewing. Video content may be viewed on a television monitor 116. In some situations, a laptop computer 118 accesses web-based services either wirelessly or via the wireline network 112. A home gateway, kiosk, digital sign, or media-restreaming device (not shown) are other possible end-user devices.
(A media-restreaming device transfers content between disparate types of networks. For example, it receives content from a cable system 112 and then transmits that content over a local radio link such as WiFi to the cellular telephone 110. The media-restreaming device usually operates in both directions to carry messages between the networks. In some embodiments, aspects of the present invention are practiced by a media-restreaming device.)
Wireless and wireline network technologies generally support two-way traffic: Media content and related information are delivered to the end-user devices 110, 114, 116, 118, and download requests go “up” to the servers 104, 106, 108.
The method of
(Note that all of the flowcharts are primarily intended to support the following discussion. The “steps” in the flowcharts are, in some embodiments and in some situations, optional and may be performed in a different order, if at all.)
In step 300 of
Statistics can be gathered about how many people actually watch which portions of a media presentation. If, for example, a large percentage of users stop requesting chunks of a music video after the first few seconds, then it can be inferred that at least the remainder (and possible the entirety) of the music video should be tagged as “unimportant.” Of course, different tags can specify in great detail exactly what is meant by the importance tag. In this scenario, the tag could give the demographic statistics of viewership, and each chunk can be tagged with the estimated or conditional probability that a viewer from a certain demographic population will be interested in and will watch this chunk.
“Importance” is meant to be broadly defined and can include just about any information that the end-user device 110 may use (in step 308, discussed below) to decide whether or not to download this chunk or to decide how to handle or render the chunk (in steps 312 through 316 of
Other types of importance information are possible and are contemplated. (See, in particular, the discussion accompanying steps 302 through 306.)
It should be noted that although in the present discussion, “importance” information is usually associated with a given chunk, that need not always be strictly true. A chunk might contain ten seconds of video, and a rating tag may only apply to a few seconds within that chunk. The tag can tell the user the exact scope of the importance information.
The end-user device 110 may receive the importance information from a number of sources. In one embodiment, the end-user device 110 receives a “playlist” from the download server 104. (The playlist may also be called a “manifest” or a “media-presentation description.”) The playlist contains information (such as the number of chunks, playing time duration of each chunk, supported resolutions, and the like) about a media presentation. The playlist can include the importance information or can include links to other sources for importance information. Instead of, or in addition to, the playlist, the end-user device 110 may receive importance information from a third-party server 106. (Here, the server 106 is a “third party” whenever it is not the download server 104 or an edge server 108.) For example, the user may only trust ratings information provided by a certain “kid-friendly” source.
The example of the “kid-friendly” ratings source brings up a more general topic: Not all users will receive the same importance information for a given media presentation. The playlist sent by the download server 104 can be customized for a particular user or for a particular device. As above, demographic information can be gathered about how a media presentation is actually viewed. If possible, this information can be carefully compared to what is known about a particular user (based, for example, on a profile stored on the end-user device 110), and the importance information tailored appropriately. If the end-user device 110 requesting the chunks only has a low resolution screen, then the playlist can be tailored for lower-resolution versions of the media presentation. (Note that in the present discussion, “resolution” is used as a shorthand for any measure of a quality of presentation.) If the user profile indicates a rating limit, then chunks that do not fall within that limit may be sent in censored form or in an alternate form that removes the objectionable content. In some embodiments, the importance information is accompanied by information stating the group for which the importance information is appropriate. The end-user device 110 can then decide whether or not this particular importance information is of interest to it.
Steps 302 through 306 of
Many other types of local behavior can be observed and remembered or used in real time. A portion of the media presentation that is fast-forwarded through or skipped can be deemed to be of little importance to this user. Conversely, rewind and slow-motion playback mark a portion as being of special importance. If the user highlights or saves a scene, then it is even clearer that the user finds the scene to be important. Other interactions with the user interface 204 can be used to infer importance. For example, if the user brings up a menu of playback controls, that might indicate that the portion of the media presentation currently being viewed is of greater or lesser importance. In response, the current portion may be marked to be cached locally or a future portion may be downloaded at a lower resolution. Again, if the user increases the volume of playback, that might indicate that the current portion is of greater importance to the user. The potential for “real-time” use of these types of behavioral observations is discussed below in reference to steps 308 of
In step 306, the end-user device 110 can, with the permission of its user, report its behavioral observations to a download server 104 or to a third-party server 106. These observations generated by the end-user device 110 are especially important because they can show what portions within a given chunk are deemed to be important and which are not. (Observations collected by the servers 104, 106 themselves are generally made on a chunk-by-chunk basis and cannot look “within” a chunk. See the discussion accompanying step 406 of
In step 308, the end-user device 110 uses the importance information to decide whether or not to download the chunk. For example, based on either demographic information received from a server 104, 106, 108 or on observations of the local user, the end-user device 110 may decide that it can safely skip over this chunk and then either stop downloading or request an alternative chunk. (In some embodiments, the end-user device 110 presents its decision to skip a chunk to the local user. The local user is given the option of accepting or overriding the decision made by the end-user device 110.) If this chunk is desired, then the end-user device 110 requests it of a server 104, 108, and the server 104, 108 sends the requested chunk. Note that criteria other than importance may be used in the decision of step 308. For example, the end-user device 110 may note that its cache is running low, and thus to avoid a video freeze, it might request a subsequent chunk in low resolution (in order to get that chunk more quickly) even though that chunk is tagged as important and would normally be requested in high resolution. As another example, the end-user device 110 may use the importance information to download a first chunk with low importance at a low resolution so that there is enough time to download a second chunk with high importance at a high resolution without causing a video freeze.
(Note: There is some confusion in the art about the meaning of a “chunk” that is relevant here. Sometimes, a “chunk” is equated with a given time segment of a video presentation, regardless of the coding resolution of that time segment. That is to say, the first two-second segment is a “chunk” that can be encoded at different resolutions. Other times, each resolution of that first two-second segment is considered to be a different “chunk.” The present discussion uses both meanings (the meaning is always clear from the context), but the latter is used when precision is required. Therefore, the decision in step 308 can be to not download this “chunk,” but instead to download a different resolution version of the same segment of the media presentation.)
In some embodiments, the end-user device 110 can, in step 308, work directly with its local user. If the local user wants just the highlights of a media presentation, then the end-user device 110 can review the importance information for the entire presentation, set an importance threshold, make a highlights video containing only those chunks whose importance exceeds the threshold, and offer the highlights video to its local user. At the given importance threshold, the highlights video will run, say, for ten minutes. The local user can then adjust the threshold (possibly without knowing that a threshold is being used) to set the highlights video to a desired length. Thus, simply by applying the importance information, each user can create a highlights video according to his own specifications. A similar service can be provided by the download server 104.
Step 312 of
If the end-user device 110 knows that its user is usually interested only in the goals of a soccer game, then the end-user device 110 can, in step 314, request the chunks tagged as goal scenes, even requesting them in high resolution and out-of-order with respect to other chunks (e.g., non-goal scenes that the user is fast-forwarding through). The end-user device 110 can also delay requesting a chunk, waiting for more behavioral information from its user that will help the end-user device 110 to know whether or not that chunk should be requested. For example, if demographic statistics received from a server 104, 106, 108 indicate that the last N chunks of a presentation are not commonly viewed (i.e., viewers usually abort the presentation before the last N chunks are viewed), then the end-user device 110 can delay requesting a download of these chunks while observing the behavior of its local user. If that user does not abort the presentation but continues to watch beyond a certain point, then the end-user device 110 can request the remaining chunks. Alternatively, the end-user device 110 can download the N-th chunk at the lowest resolution possible and delay the download of further chunks until and if the local user starts and continues watching after a certain point of the N-th chunk.
Often, the end-user device 110 will have limited memory and cannot store the entire media presentation. The importance information can then be used by the end-user device 110 to know which chunks to cache because its user may go back and review them (e.g., goals) and which chunks can be discarded immediately after viewing (e.g., the rest of the game).
In step 316, the end-user device 110 renders the chunk to its user via the user interface 204. (In some situations the user interface 204 is used to actually render the chunk on another device, such as when the set-top box 114 renders to the television monitor 116.) Here, the end-user device 110 can use the importance information (often along with local user-interface settings) when deciding how to render this chunk. For example, the end-user device 110 can “pixelate” (a method of obscuring a digital image) to censor scenes tagged as visually offensive or can blur the audio to make offensive language unintelligible. Or, the end-user device 110 can clarify a scene normally obscured. (E.g., the chunk can be encoded to satisfy FCC broadcast standards, standards which need not be followed by the local user, and the end-user device 110 can remove the obscurities, possibly by consulting a third-party server 106 for additional information.) The end-user device 110 might also choose to anticipate its user's wishes by fast-forwarding or skipping to a scene presumably of interest to that user.
Note that the steps of
The method of
Some embodiments of the present invention provide benefits even if the servers 104, 106, 108 are not enhanced in any way over the known art. (That is, the end-user device 110 only has access to the importance information that it can infer from observations of its user's behavior in step 302 of
a and 4b provide an example of such an enhanced server 104. In step 400 of
In some embodiments of step 408, the server 104 sends at least some importance information (or links to importance information stored elsewhere) to a client device. (The end-user device 110 is one type of client device, but there are others, as discussed below.) The importance information may be included in a playlist, either generic or customized, as discussed above. In other embodiments of step 408, the server 104 does not actually send the importance information but instead creates and sends a customized playlist based on the importance information. A customized playlist might include only those chunks that meet the appropriateness criteria of a user profile stored on the end-user device 110 or might include substitute, non-objectionable, chunks for those chunks deemed objectionable. Note that step 408 can be repeated during the download of a media presentation as updated importance information becomes available.
In some embodiments, an alternative step 408 can be used with legacy end-user devices 110. These are devices that do not know about importance information. The server 104, knowing the limitations of this particular end-user device 110, can, instead of sending out importance information that will simply be ignored, use the importance information to tailor a version of the playlist for this particular end-user device 110. The results as perceived by the user of the end-user device 110 will roughly approximate the results obtainable by an end-user device 110 that is fully cognizant of the importance information.
In steps 410 and 412, the server 104 receives a request for a chunk from a client device and fulfills that request by downloading the requested chunk. Most systems today are “pull” systems where the client device actually makes the decision about what to download (in step 308 of
In some situations, the gathered importance information can lead the server 104 to decide that the present chunking is not the most efficient. For example, it may be discovered that half of a ten-second chunk is very important, but the other half is rarely viewed. This leads to inefficiencies because most (but not all) current systems can only download on a chunk-by-chunk basis and cannot deliver only part of a chunk. To alleviate this inefficiency, the server 104 can, in step 414 of
Similar to the situation in step 414, the server 104 may, in step 416, decide that a whole new version of the media presentation (or parts of the media presentation) should be provided at a new resolution. That is, scenes often subject to extensive fast-forwarding or skipping may be recoded to make them available at a low resolution, while oft-viewed scenes may be provided at a high resolution.
As with the method of
For the sake of clarity, the discussion of the method of
In reference to step 408 of
In accordance with aspects of the present invention,
On the other hand, step 502 summarizes the role of the edge server 108 with respect to download servers 104 (and, in some embodiments, with respect to third-party servers 106). That is, the edge server 108 can perform the steps of the end-user device method as illustrated in
The edge server 108 does not perform entirely at the whim of the servers 104, 106 and of the end-user device 110. In step 504, the edge server 108 can use importance information (either given to it or inferred by it) to decide which chunks to “pre-cache,” that is, which chunks to request from the download server 104 and store even before they are requested by an end-user device 110. For example, it can be decided up front that the highlights of a championship game are going to be pretty popular download targets. Then, rather than waiting for the first requests from end-user devices 110 to come in, the edge server 108 can store these highlights immediately, thus making its response to the first requests quicker than if it had to retrieve the highlights only upon the first request.
Similarly, in step 506, the edge server 108 can use importance information and can also observe the download behavior it is seeing and decide which chunks are popular enough to keep in its somewhat limited cache (and, conversely, which chunks can be deleted to make room for others). Note that this decision can be made independent of, and even counter to, the demographic statistics gathered by the download server 104 and third-party server 106. That is because the edge server 108 is seeing a more localized population whose tastes may differ from those of the more general population seen by the servers 104 and 106.
Some embodiments of the present invention use chunk-size information in addition to, or instead of, importance information to increase the efficiency of downloads. Because the chunks that make up a media presentation are generally all of the same play length (e.g., each chunk represents two seconds of the presentation), one might think that all of the chunks contain the same number of bytes (for a given resolution, of course). That assumption is, however, often not true because the encoding efficiency can vary throughout the presentation due to changes in the complexity of the scene being viewed and how rapidly the scene is changing.
While this variance in encoding efficiency has long been known in the art, end-user devices have not been able to intelligently handle the variance. Prior-art end-user devices had to assume that all of the chunks in one media presentation are of the same size (for a given resolution). It is quite possible that when an upcoming chunk is much larger than the assumed size (e.g., chunk 7 of
In step 702, the end-user device 110 reviews the chunk-size information. For example, the end-user device 110 can continuously analyze the performance of its network link. Based on that analysis, the end-user device 110 can estimate how long it should take to download the next chunk, given the size of that chunk. The end-user device 110 can decide that it is unlikely that the next chunk can be downloaded in time. Then, to avoid the possibility of a video freeze, the end-user device 110 could, in step 704, request the next chunk at a lower resolution (that is, with a smaller chunk-size). In some situations, the end-user device 110 may decide to request a completely different chunk or not request any chunk at all.
In some situations, the chunk-size information and the importance information are both available to the end-user device 110 which can use both types of information to decide what to do in step 702.
If in step 704, the end-user device 110 requests a chunk, then the server 104, 106, 108 provides that chunk in step 706.
a and 8b present experimental results. In
Other examples of the use of importance information are illustrated in
(Note that the methods of
a and 9b present a method for the server 104. In step 900 of
Because it takes longer to download a high-resolution version of a chunk than a low-resolution version, the server 104 may include, in the importance information for this chunk, an explicit recommendation that the end-user device 110 start downloading this chunk out-of-order, using any available bandwidth. This helps ensure that the high-resolution chunk is available when its time for viewing arrives.
In an example very different from the soccer goal discussed above, an advertisement can be downloaded within a user-requested media presentation. (But also see the discussion of step 908 of
(It should be noted that in some embodiments, the end-user device 110 is not told which particular chunks are advertisements and which are part of the requested media presentation. This is a consequence of the “pull” nature of these embodiments.)
Although advertisement chunks are downloaded in the same fashion as are the chunks of the user-requested media presentation, the advertisements are clearly not a part of that media presentation. A consequence of this is that, unlike a soccer goal, an advertisement does not have a “natural” place within the media presentation. Thus, the advertiser (or the server 104 working independently) can add to the importance information for the advertisement chunk a recommendation for when to play the advertisement (step 904). For example, scene changes in the media presentation can be noted (by a human or an electronic editor), and the recommendation made that the advertisement be played right after the chunk ending the current scene. A time range can also be given. For example: “Play this advertisement at a time decided on by the end-user device 110, but certainly within the next five minutes.”
Another consequence of the fact that the advertisement is not a part of the user-requested media presentation is illustrated in step 906. The advertisement is really a media presentation unto itself and may include a number of chunks. To prevent a video freeze at the end-user device 110 when playing the advertisement, the importance information notes how many chunks are in this advertisement and includes a recommendation of a minimum number of chunks to download before beginning the rendering of the advertisement.
Step 908 of
The remainder of the method of
In the method of
The decision process for determining when to download a chunk is complicated by the many different types of importance information that may be available. In a very simple case, a soccer goal chunk is marked with a recommended resolution. The goal chunk has a definite time within the media presentation at which it should be rendered; the end-user device 110 knows the size of the goal chunk at the recommended resolution and has determined its available bandwidth. The end-user device 110 may simply put these numbers together to decide when it should request the goal chunk. (A potential complication comes when the available bandwidth is so low that downloading the intervening chunks before the goal chunk may prevent the end-user device 110 from downloading the goal chunk in a timely manner.)
When the chunk is an advertisement, the importance information might include a number of recommended times at which to render the chunk (e.g., at any one of the next few scene changes in the media presentation). Thus, the end-user device 110 can balance current download needs and schedule the download of the advertisement chunk a little further in the future, knowing that the advertisement should be received before at least one of the recommended playing times.
If the recommendation is made to render the advertisement in a resolution that matches that of the chunks of the user-requested media presentation that will surround it, then the end-user device 110 has some flexibility: It might change the resolution it is requesting for the media-presentation chunks if that ensures that the advertisement chunks will arrive in time.
Step 1010 considers the case when chunks are “tied” together, e.g., an advertisement includes multiple chunks. The end-user device 110 now may need to schedule the download not of one chunk, but of the minimum recommended number of tied-together chunks, making sure that they arrive before a recommended time for rendering the advertisement.
Based on the above considerations, the end-user device 110 requests the chunk (step 1006) at an appropriate resolution (step 1008) (see also the discussion above in reference to steps 902 and 904 of
With the chunk in hand, the end-user device 110 decides when to render it (step 1012 of
Step 1016 considers the case where a recommendation is made to render the downloaded advertisement before a user-requested media presentation is ready to start playing. That is, the advertisement is downloaded early so that it can be played while the initial chunks of the user-requested media presentation are still being downloaded. The advertisement might have been downloaded hours earlier during periods of low bandwidth use (e.g., when the user is on a telephone call or browsing the web or when the end-user device 110 is idle) and held in the memory of the end-user device 110 until needed.
Step 1016 also applies while the final chunks of a user-requested media presentation are being rendered. Because the media presentation has already been fully downloaded, the end-user device 110 can take advantage of the unused bandwidth to download another advertisement to be played in the future.
Because the advertisement is not downloaded during the download of the media presentation, care should be taken to make sure that the advertisement is displayed during an appropriate media presentation. The advertisement's importance information can include a characterization of the advertisement, and the end-user device 110 can compare that characterization to a characterization of the media presentation. If they match, then the advertisement can be rendered. Otherwise, another advertisement may be rendered while this one is held ready for a more appropriate time. Known techniques such as data mining, user profiling, and analysis of past user activity can be used to match advertisements with suitable media presentations. These techniques can also be used to determine which advertisements to download. The importance information can include a time setting stating how long the advertisement should be retained while it is waiting to be played.
In view of the many possible embodiments to which the principles of the present invention may be applied, it should be recognized that the embodiments described herein with respect to the drawing figures are meant to be illustrative only and should not be taken as limiting the scope of the invention. For example, aspects of the present invention may be particularly useful in adaptive-streaming environments, but the invention is not limited to these environments. Aspects of the present invention are not limited to any particular implementing data-networking protocols or to particular server and end-user device deployments. Therefore, the invention as described herein contemplates all such embodiments as may come within the scope of the following claims and equivalents thereof.
The present application is related to U.S. patent application Ser. Nos. 12/891,348 and 12/891,421, both filed on Sep. 27, 2010, and incorporated herein by reference in their entireties.
