Patent Application
20190069036
Consumer devices such as tablet computers and smartphones are increasingly being used to receive and view video programs. However, it is still cumbersome for a user to use the user's device to browse through a selection of available video programs.
Popularity of the Internet as the network for video delivery to users continues to grow. Internet Protocol (IP) packets allow seamless video delivery over heterogeneous networks such as wired and wireless networks. As the computational power of user devices such as mobile phones and tablet computers grows, so does a user's ability to receive and view multiple and/or higher bandwidth video programs.
In a first aspect a system for distributed control of segmented media includes, a control server that may be in communication with a content delivery network (CDN) node and a client device. The control server may include a client controller configured to receive a manifest from the CDN defining a plurality of content segments of a master content available at the CDN, and receive client information from the client device. The client controller may be configured to generate, based on the client information, one player manifest for each of a plurality of client device media players of the client device to control access of each client device media player to appropriate ones of the plurality of content segments based on the client information.
In embodiments of the first aspect, the manifest may include a top-level manifest identifying the master content, and at least two variant manifests identifying variants of the master content and respective ones of the plurality of content segments associated with each variant. In embodiments of the first aspect, the control server may select the appropriate ones of the plurality of content segments based on the top-level manifest and the at least two variant manifests.
In embodiments of the first aspect, each player manifest may include some, but not all, of the plurality of variant manifests corresponding to the appropriate ones of the plurality of content segments.
In embodiments of the first aspect, each variant manifest may correspond to a different resolution.
In embodiments of the first aspect, the client information may include a window size communicated by the client device.
In embodiments of the first aspect, each player manifest identifying the variant manifest for a peak resolution corresponding to the window size.
In embodiments of the first aspect, each player manifest may identify the variant manifest corresponding to a peak resolution determined based on a total bandwidth and a bandwidth used by a plurality of client devices on the same network and in communication with the control server.
In embodiments of the first aspect, each player manifest may identify the variant manifest selected based on a total bandwidth and a bandwidth used by a plurality of client devices on the same network and in communication with the control server.
In embodiments of the first aspect, the client information may include a primary media selection and at least one secondary media selection.
In embodiments of the first aspect, the control server may use the primary media selection and the secondary media selection to select the appropriate ones of the plurality of content segments.
In embodiments of the first aspect, the client information may include total bandwidth and bandwidth used by the client device. In embodiments of the first aspect, the control server may use the total bandwidth and the used bandwidth to select the appropriate ones of the plurality of content segments.
In embodiments of the first aspect, the client information may include a player rate buffer level for each of the client media players.
In embodiments of the first aspect, the control server may use the player rate buffer levels to select the appropriate ones of the plurality of content segments.
In embodiments of the first aspect, each player manifest may be configured as a live manifest and generated periodically.
In embodiments of the first aspect, the player manifest may identify the appropriate ones of the plurality of content segments with different resolution when determined by the control server based upon one or more of the total bandwidth, the bandwidth used, and a network bandwidth usage.
In a second aspect, a method for distributed control of segmented media, may comprise: receiving a manifest from a content delivery network (CDN) node defining a plurality of content segments of a master content available at the CDN, receiving client information from a client device, and generating, based on the client information, one player manifest for each of a plurality of client device media players of the client device to control access of each client device media player to appropriate ones of the plurality of content segments based on the client information.
In embodiments of the second aspect, the method may comprise selecting the appropriate ones of the plurality of content segments based on a top-level manifest and at least two variant manifests included within the manifest, the top-level manifest identifying the master content, and the at least two variant manifests identifying variants of the master content and respective ones of the plurality of content segments associated with each variant.
In embodiments of the second aspect, each player manifest may include some, but not all, of the plurality of variant manifests corresponding to the appropriate ones of the plurality of content segments.
In embodiments of the second aspect, each variant manifest may correspond to a different resolution.
In embodiments of the second aspect, the client information may include a window size communicated by the client device.
In embodiments of the second aspect, each player manifest may identify the variant manifest for a peak resolution corresponding to the window size.
In embodiments of the second aspect, each player manifest may identify the variant manifest corresponding to a peak resolution determined based a total bandwidth and a bandwidth used by a plurality of client devices on the same network and in communication with the control server.
In embodiments of the second aspect, the method further includes selecting the appropriate ones of the plurality of content segments based upon a primary media selection and at least one secondary media selection included within the client information.
In embodiments of the second aspect, the method further includes selecting the appropriate ones of the plurality of content segments based upon a total bandwidth and a bandwidth used of the client device as included within the client information.
In embodiments of the second aspect, the method further includes selecting the appropriate ones of the plurality of content segments based upon a player rate buffer level for each of the client media players as included within the client information.
In embodiments of the second aspect, the method further includes selecting the appropriate ones of the plurality of content segments based upon total memory and memory used by the client device media players as included within the client information.
In embodiments of the second aspect, the method further includes periodically generating each player manifest as a live manifest.
In embodiments of the second aspect, the method further includes generating subsequent ones of the player manifest to identify appropriate ones of the plurality of content segments with a different resolution based upon one or more of the total bandwidth, the bandwidth used, and a network bandwidth usage.
The foregoing and other features and advantages of the disclosure will be apparent from the more particular description of the embodiments, as illustrated in the accompanying drawings, in which like reference characters refer to the same parts throughout the different figures. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure.
Content delivery to user devices is becoming more and more popular with ever increasing data transmission speeds and evolution of “cloud” based internet streaming services. Moreover, the user devices themselves have increasing capabilities, such as the ability to use multiple media windows at the same time on the same device. The problem arises that each video window (e.g. an active player instance) competes with the other video windows for available bandwidth with respect to (1) the given device, but also (2) other devices on the same network. The resulting situation is that one device (or a given player instance in a single device) uses high-quality content while other devices (or player instance(s) in the single device) suffer poor quality of service due to lack of timely content delivery caused by bandwidth issues. In other words, each player instance is unaware of its peers, and thus is unable to share the network in a way that allows all instances (or devices) to function subject to an overall bandwidth limit.
Embodiments herein resolve the above described, and other, problems and limitations by using a content control server to monitor the user client configuration (for example, but not limited to, bandwidth, memory occupancy, content requested, etc.). Based on the client configuration the control server may select one of the available content variants (e.g. based on bit rate and resolution) for each player instance in that device (or in other devices). The client may have no choice in the variant available for play because the control server controls the client (or specific player instance therein) by providing the one and only content variant advertised to each client instance by constructing a unique player manifest for each player instance in each client. One or more items of client configuration information may be periodically reported to the control server, and based on these reports, the particular variant selected for a client instance may be changed. All content is considered to be “live” (as opposed to recorded or video on demand (“VoD”)), and only a small amount of content is made available to each client instance with each player manifest request. As such, repeated interaction between each client and the content control server enables the server to change content variants frequently, thereby improving the overall bandwidth management of the system (i.e., for all player instances on all client devices).
Embodiments herein may use any one or more of the following constraints. These constraints may be optional, and are not limiting in scope unless specifically claimed, stated, or otherwise indicated as required.
Embodiments may use segmented multi-bitrate video delivered via HTTP. This video may be delivered over TCP/IP using standards such as, but not limited to, dynamic adaptive streaming over HTTP (“DASH”), HTTP Live Streaming (“HLS”), or HTTP Dynamic Streaming (“HDS”). The TCP/IP may be advantageous, in embodiments, to provide reliable point-to-point delivery of the content, and the adaptivity may allow a choice of different effective bitrates based on network performance (e.g., time varying throughput and congestion) and client device characteristics (e.g., window resolution, memory available, etc.). In all of the above standards, the client is typically provided a choice of different variants of the content (differing by bitrate and resolution) and has the ability to switch between content variants at segment boundaries. The available variants are advertised to the client by means of a manifest object which specifies the bitrate and resolutions (among other attributes) available.
The embodiments herein may interface with existing media players (e.g., using JavaScript wrappers) and decoders used on and by the client device for a given player instance. As such, embodiments herein may provide the advantage of improving the user's investment in client software based on these existing player packages by providing solutions which not only coexist but build upon the features provided by these existing player packages.
Embodiments herein may use, for wide area distribution of audiovisual content on the public internet, and the value provided by content delivery networks (“CDNs”). CDNs typically use specialized domain name servers to connect to an edge server, which is both close (in network topology) to the client device and lightly loaded in order to provide a high quality of service. The CDNs may provide HTTP caching and streaming (continuous one-way video distribution between the origin server and the various geographically dispersed edges) services.
Embodiments herein may use HTTP or HTTPS communication protocol (over standard TCP ports (e.g., ports 80 and 443)) between individual components of the system (e.g. between the control server and the client device, the control server and a CDN, and/or the client device and the CDN). As such, the system herein may avoid issues arising due to network firewalls.
System 100 includes at least one content delivery network (“CDN”) node 102, a control server 104, and at least one client device 106 used by a respective user 108. It should be appreciated that only two CDN nodes 102(A), 102(B), and only two user devices 106(1), 106(2), are illustrated in
As shown in
Processor 202 may be one or more computing devices capable of executing computer readable instructions to implement functionality of the CDN node 102 as discussed herein.
Network interface 204 may be 1) a wired communication protocol, such as telephone, Ethernet, fiber optics, Cable, USB, lightning cable, or other wired communication protocols, 2) a wireless communication protocol, such as Bluetooth, Bluetooth low-energy (BLE), WiFi, cellular 2G, 3G, 4G, 5G, LTE, or any other wireless communication protocol, or 3) a combination of wired and wireless communication protocols. Network interface 204 may be used to facilitate one or more of first communication path 110 and third communication path 114 according to the above described wired and/or wireless communication protocols.
Memory 206 may include one or both of volatile (e.g. RAM, DRAM, SRAM, etc.) or non-volatile (e.g. ROM, PROM, EEPROM, NVRAM, flash memory, solid-state storage, optical or hard disk drives, etc.) memory. Memory 206 may store a media manager 208, which may include transitory and/or non-transitory computer readable instructions that when executed by the processor 202 implement the following functionality.
Media manager 208 may receive master content 210 and segment said master content 210 into one or more content segments 212. Although only a single master content 210 is shown, it should be appreciated there may be additional master contents 210 for a given CDN node 102 without departing from the scope hereof. As shown in
Master content 210 need not be the same format as any of the segments 212. For example master content 210 may be in the format of any one or more of FLV, MPEG2 transport stream, MP4, while any given segment may be in the format of any one or more of MPEG2-TS or fragmented MP4 media segments. As such, master content 210 may be decoded (or transcoded) via a media manager 208, or an external decoding/transcoding device prior to storage within memory 206 as one or more content segments 212. Moreover, each variant may be configured to store a different media format.
As each master content 210 is segmented into variant(s), media manager 208 stores a top-level manifest 214. In embodiments using HLS, the top-level manifest 214 may be a “Master Playlist” and a variant manifest 216 may be a “Media Playlist”. In embodiments using MPEG-DASH, there may only be a top-level manifest 214 as a single manifest called a “Media Presentation Description (MPD)” document. Each type of delivery (e.g., HLS, DASH, HDS) may include a different manifest format, and has a separate top-level manifest 214. In the embodiment of
Turning now to
Processor 302 may be one or more computing devices capable of executing computer readable instructions to implement functionality of the control server 104 as discussed herein.
Network interface 304 may be 1) a wired communication protocol, such as telephone, Ethernet, fiber optics, Cable, USB, lightning cable, or other wired communication protocols, 2) a wireless communication protocol, such as Bluetooth, Bluetooth low-energy (BLE), WiFi, cellular 2G, 3G, 4G, 5G, LTE, or any other wireless communication protocol, or 3) a combination of wired and wireless communication protocols. Network interface 304 may be used to facilitate one or more of first communication path 110 and second communication path 112 according to the above described wired and/or wireless communication protocols.
Memory 306 may include one or both of volatile (e.g. RAM, DRAM, SRAM, etc.) or non-volatile (e.g. ROM, PROM, EEPROM, NVRAM, flash memory, solid-state storage, optical or hard disk drives, etc.) memory. Memory 306 may store a client controller 308, which may include transitory and/or non-transitory computer readable instructions that when executed by the processor 302 implement the following functionality.
Control server 104 may receive the top-level manifest 214 and associated variant manifests 216 from one or more of the CDN nodes 102 via each respective first communication paths 110. As such, although only a single top-level manifest 214 is illustrated in
Client controller 308 may analyze one or more of the top-level manifests 214 and associated variant manifests 216 to identify a list of available media 310 for streaming to one or more of the client devices 106. Available media 310 may be transferred to each client device 106 via respective ones of second communication paths 112. Available media 310 may provide a guide to the user 108 via interaction with an interface of the client device 106. The guide may be a list, or may be interactive such as the user interface(s) depicted in FIGS. 11-27 of U.S. Pat. No. 9,582,157, entitled “User Interface and Program Guide for a Multi-Program Video Viewing Apparatus” which is incorporated herewith in its entirety.
Client controller 308 may receive, via respective second communication paths 112 and from each respective client device 106, client information 312. Client information 312 may include any control message parameter, such as one or more of: total bandwidth 314, bandwidth used 316, total memory 318, memory used 320, primary media selection 322, secondary media selection(s) 324, a channel change indication 326, and player rate buffer levels 327, and a player window size 329. Player rate buffer levels 327 provided by the client device 106 is used for bandwidth management, for example (e.g., a player rate buffer level is provided for each client media player). Client Information 312 may further include one or more of screen width and height, browser type, device name and version, protocol type (e.g. HLS, HDS, DASH), window sizes, content URLs, seek timestamps, and change timestamps.
Based on client information 312, and top-level manifest 214 indicating various variant manifests 216, client controller 308 may generate a player manifest 328 for each client media player (e.g., client media player 412,
Turning now to
Processor 402 may be one or more computing devices capable of executing computer readable instructions to implement functionality of the client device 106 as discussed herein.
Network interface 404 may be 1) a wired communication protocol, such as telephone, Ethernet, fiber optics, Cable, USB, lightning cable, or other wired communication protocols, 2) a wireless communication protocol, such as Bluetooth, Bluetooth low-energy (BLE), WiFi, cellular 2G, 3G, 4G, 5G, LTE, or any other wireless communication protocol, or 3) a combination of wired and wireless communication protocols. Network interface 404 may be used to facilitate one or more of second communication path 112 and third communication path 114 according to the above described wired and/or wireless communication protocols.
User interface 408 may be any interface, such as a touch screen, display screen with mouse and/or keyboard, voice interface, video interface, or any other device capable of interacting with user 108 via client device 106.
Memory 406 may include one or both of volatile (e.g. RAM, DRAM, SRAM, etc.) or non-volatile (e.g. ROM, PROM, EEPROM, NVRAM, flash memory, solid-state storage, optical or hard disk drives, etc.) memory. Memory 406 may store a client content manager 410, which may include transitory and/or non-transitory computer readable instructions that when executed by the processor 402 implement the following functionality.
Client content manager 410 may be implemented from a software development kit (SDK) and used on the client as an interface between CDN nodes 102, control server 104, and one or more media players 412 located on the client device. Media players 412 may be any client software package used for playing media on client device. One or more media players 412 may be configured to provide multiple streaming media (e.g. video, audio, or other media) windows at the same time. Each client media player 412 will “play” one or more media according to a corresponding player manifest 328. That is, each client media player 412 uses a separate player manifest 328 that is generated by client controller 308 of control server 104. For example, each player manifest 328 includes a URL where the streaming media content is located on CDN node 102 and the corresponding client media player 412 contacts the associated CDN node 102, using the URL, and accesses the media file thereat for play by the client media player 412.
Multiple variables affect the ability of media to stream properly on client device 106, and multiple client devices 106 when implemented. Accordingly, client content manager 410 may monitor the total bandwidth 314 and the bandwidth being used 316 by the client device, and particularly by each client media player 412 while playing media from each of CDN nodes 102 according to player manifest 328. Client content manager 410 may further monitor the total memory 318 and the memory being used 320 within the memory buffers 414 of the client device 106 to determine player rate buffer levels 327. Client content manager 410 may further identify the required resolution (and identify said resolution in the primary media selection 322 and/or secondary media selection(s) 324) based on the window size that the media is to be displayed on client device 106. Client content manager 410 then transfers this information 312 to control server 104 such that control server 104 may determine the appropriate content segments 212 (including the appropriate variant of the master content 210) such that all media streaming is satisfactory for all client media players 412 on all client devices 106 by minimizing bandwidth errors during playback. The control server 104 then controls each client media player 412 of the client devices 106 by generating the player manifest 328 to include only information of the appropriate content segments 212 and then transferring the player manifests 328 to the client devices 106.
Transfer of information 312 to control server 104, and generation of player manifests 328 (and identification of manifests 214, 216) is repeated periodically or aperiodically by control server 104 such that control server 104 is able to effectively buffer streaming media to the client device 106. By managing multiple client devices 106 at the same time, control server 104 also manages network bandwidth usage by the client devices and CDN nodes 102.
Bandwidth and memory management control by control server 104 also provides seamless interaction with client device 106 by user 108 while watching multiple media instances on one or more client media players 412. As shown in, for example, FIGS. 3 and 4 of U.S. Pat. No. 9,582,157 (incorporated by reference in its entirety), a client device 106 may display a primary video and one or more secondary videos. The media selection of user 108 for the primary video is stored in client information 312 as primary media selection 322, and the secondary media selection(s) are stored as secondary media selection(s) 324. In embodiments, the player manifest 328 may identify a first variant having a first resolution for displaying the primary media selection 322, and a second variant having a second resolution lower than the first resolution for the one or more second media selection(s) 324. A user, at any time, may “change channels” by interacting with user interface 304 to select one of the secondary media selection(s) 324 (or any other media content not previously identified in information 312 that is available) to be shown as the primary media selection 322. At such time, the primary media selection 322 and the secondary media selection(s) 324 are changed and updated client information 312 is transmitted to control server 104 from client device 106. Control server 104 then analyzes the updated client information 312 to generate new player manifests 328 (one per client media player 412) which are transmitted to client device 106. Within client device 106, each client media player 412 then continues to operate by pulling the appropriate media content segments 212 from the CDN nodes 102 as defined by the new player manifests 328. This process allows for content switching at the client device 106 within the flow of the same player manifest 328 (e.g., without requiring the media player to restart and begin pulling a new player manifest representing the new content). This advantageously allows for seamless content switching by eliminating the need for the player to restart and pull a new player manifests, which disrupts the presentation of content at the client device 106.
Player manifest 328 may allow the client media player 412 some variability in selecting the variant. For example, in some embodiments, the player manifest 328 may include some, but not all, of the variants 216 identified in the top-level manifest 214. For example, the player manifest 328 may include some of the variants 216 corresponding to different resolutions for selection by the client media player 412.
Player manifest 328 may select the variant 216 to transmit to the client media player 412 based upon the client information 312, such as based on the player window size 329. For example, the player window size 329 may dictate a peak resolution available for that window. As such, the player manifest 328 may identify the variant 216 corresponding to the peak resolution. In addition, or alternatively, the peak resolution may be identified based on bandwidth available over the network. For example, a peak resolution may be determined based a total bandwidth and a bandwidth used by a plurality of client devices 106 on the same network and in communication with the control server 104.
In one example, system 100 is used in a live sporting event. Each master content 210 may be a different camera, or video stream, of the live sporting event. The user 108 may include a desired camera angle as the primary media selection 322, and secondary camera angles as the secondary media selection(s) 324. The user 108 then may seamlessly shift between various master contents 210. Where the user 108 is at a specific venue, and multiple users 108 are connected to system 100, the bandwidth management becomes more important such that each user 108 is capable of receiving the live media streams.
Each user 108 may further include subscription access to various master contents 210, or to specific variant levels. For example, user 108 may buy into a “premium” subscription model where the control server 104 prioritizes the bandwidth for that given user (as well as others in that subscription model). Not only may the users 108 be part of a subscription model, but the content providers as well. For example, content providers may pay a premium such that their master content 210 is distributed at better variants (e.g. better resolution, higher bandwidth, etc.) than other content providers.
Specific control of client device 106 via control server 104 differs from prior systems that allow the client device to control what variant of the master content 210 the client device desires. Allowing the client device control of the desired variants causes bottlenecks in video content delivery because the client is unaware of the available bandwidth, and/or what other devices are attempting to stream. Also, allowing each client media player independent control over the variant pulled does not account for what other client media players are attempting to stream. Controlling video delivery through the network to multiple client media players and client device allows scalability that was previously unachievable.
Control server 104, in embodiments, also allows for consolidation of different CDN nodes 102 into a single system. For example, control server 104 may ingest manifests 214 from different CDN nodes 102. The centralized control server 104 may then manipulate the different constraints of each CDN node 102 and distribute a single player manifest 328 that accommodates these different constrains.
In step 502, method 500 receives manifests from a CDN node. In one example of step 502, control server 104 receives a top-level manifest 214 including one or more variant manifests 216 from a CDN node 102. Step 502 may be repeated for multiple top-level manifests 214 and variant manifests 216. The received manifests may then be stored within control server 104 for further processing.
In step 504, method 500 transmits available media to a client device. In one example of step 504, control server 104 transmits available media 310 to client device 106.
In step 506, method 500 receives client information from a client device. In one example of step 506, control server 104 receives client information 312 from client device 106. In embodiments, client information 312 may include total bandwidth 314 and bandwidth used 316 associated with client device 106. In embodiments, client information 312 may include (additionally or alternatively) total memory 318 and total memory used 320 by client media player 412. In embodiments, client information 312 may include (additionally or alternatively) primary media selection 322 and secondary media selection(s) 324. In embodiments, client information 312 may include (additionally or alternatively), channel change information 326 identifying user interaction with user interface 408 to update one or both of primary media selection 322 and secondary media selection(s) 324. Step 506 may be repeated for any number of client devices 106 within system 100.
In step 508, method 500 generates one player manifest for each client media player. In one example of step 508, control server 104 generates one player manifest 328 for each client media player 412. For example, control server 104 may use the manifests received in step 502, and the client information received in step 506 to generate a satisfactory player manifest 328 for each client media player 412. Step 508 may allow an individual client device and/or the entire network to be optimized such that multiple clients are capable of accessing content on CDN nodes 102.
In step 510, method 500 accesses the CDN node(s) according to the player manifest generated in step 508 to stream media content. In one example of step 510, each client media player 412 within client device 106 initiates contact between the client device 106 and the CDN node 102 according to the corresponding player manifest 328.
Changes may be made in the above methods and systems without departing from the scope hereof. It should thus be noted that the matter contained in the above description or shown in the accompanying drawings should be interpreted as illustrative and not in a limiting sense. The following claims are intended to cover all generic and specific features described herein, as well as all statements of the scope of the present method and system, which, as a matter of language, might be said to fall therebetween.
