Patent Application
20250150655
The present principles relate to video and/or audio processing, and more specifically, to Just-In-Time-Transcoding (JITT) video and/or audio systems and methods.
The surge in video streaming services and the increasing reliance on video in all web-related content necessitates the need for affordable, high-quality streaming technology. The varying usage scenarios and deployment situations greatly influence the streaming system costs. For instance, cloud DVR systems can place a substantial financial strain on storage, but the expenses associated with the Content Distribution Network (CDN)/edge distribution architecture should not be overlooked either. Factors such as the application (e.g., DVR, VOD, Live), the quantity of channels or number of media assets stored, the user count, and the CDN or distribution network's architecture/bandwidth all contribute to shaping the system's design and cost centers.
While advancements in various technologies have boosted streaming performance and cost-efficiency, the deployment of streaming systems remains a considerable investment. Content-aware transcoding and just-in-time packaging (JITP) are two technologies that have positively impacted streaming deployments.
In this disclosure, we will introduce and discuss “Just-in-Time Transcoding/Transcoder” (JITT). We will explain how a JITT product sets a new standard for affordable video streaming deployment, eases generational codec transitions and enhances video quality and performance for the end-users in the process. An example of such a JITT product is a Blazar Just-in-Time Transcoder being designed by igolgi Inc. of Princeton, NJ.
In addition, another benefit of the Blazar JITT is that it reduces energy consumption per stream, helping to reduce carbon emissions.
A video streaming system employs a technique known as adaptive bitrate (ABR) encoding. The audio and video streams are first chopped up into small time chunks known as segments. These segments may last anywhere from 2 to 10 seconds (this is only a guideline, not a requirement). The file segments are then encoded into a variety of resolutions and bitrates, referred to as a “bitrate ladder”. The audio signal is similarly divided into several audio channels and bitrates. The goal of ABR is to deliver the appropriate version of video and audio segments based on the capabilities of the receiving device and the available bandwidth. The entire streaming operation is thus converted to a series of file transfers (that can be supported by any Internet server) from the headend to the client devices. Rate selection is done by each individual client based on a manifest of rates provided by the ABR encoding head end.
As alluded to earlier, another key feature of this system 100 is the division of content into brief time segments. The advantage of this is that the device receiving the content can easily switch between profiles by just requesting the next small segment from a different profile. Common packaging formats in use are HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH). The process of packaging these segments can happen at various stages in the network: before the storage, after storage in the Origin Server, or at the edge of the network. Each of these packaging architecture options has tradeoffs with storage size, core network bandwidth, and edge cache complexity and edge storage.
Just-In-Time Packaging (JITP) is an enhanced version of packaging that's particularly efficient when various digital rights management (DRM) or packaging formats are required for different types of devices. In this setup, the original video and audio signals are stored just once, but they are packaged in real time to satisfy the specific needs of the receiving device. This method significantly cuts down on storage and bandwidth costs.
Streaming systems typically have many different resolutions and bitrate profiles available so that each client can stream the content efficiently. In
In the Content Aware Ladder system, a smaller number of profiles are generated using “content-aware” encoding, which creates unique profiles based on the specific requirements of the content. For instance, a high-action movie may need different combinations of resolutions and bitrates than a news program featuring people simply talking. Video-on-Demand (VOD) systems are good candidates to employ a content-aware ladder. While Digital Video Recorder (DVR) systems might also use content-aware encoding, they may require re-encoding from the live signals.
The advantage of an Automated (content-aware) ladder is that it uses fewer profiles. This means a DVR system would use less storage, and the bandwidth needed by the Content Delivery Network (CDN) to deliver a certain video quality to a specific client could also be marginally reduced compared to the static ladder. In the provided example shown in Table 200, the content-aware ladder results in a 60% reduction in storage requirements.
In both the content aware ladder or static ladder, neither provides an optimal profile based on the end-client screen size or dynamically available bandwidth. Since both systems require transcoding to the ladder profiles and storing ahead of time in the DVR, the end client can only choose from a limited set of operating points. The static ladder provides more operating points so gives more choices for bandwidth options but requires more storage. The content aware ladder uses less storage but does not give as many options for bandwidth which creates less efficient streaming for certain end point devices.
For example, if an end client had available 1400 Kbps, the static ladder could provide the 540p profile at 1200 Kbps, while the content aware could only provide 480p at 811 Kbps, so in this case the content aware system actually produced worse quality than the static system.
The result is that neither of these solutions are ideal.
Just-in-Time Transcoding (JITT) is a system and a method where compressed video and audio signals are converted from one format to another at the precise moment when a request for the signal is made by a video or audio player (client). This necessitates that the transcoding system functions at a speed much faster than real time. Blazar, a JITT transcoder, possesses several unique characteristics, primarily its ability to transcode at a speed up to 100 times faster than real time. Blazar JITT is also dynamically scalable which allows for an array of new applications and use cases.
Blazar JITT fully leverages the latest developments in CPU and GPU technologies and makes optimal utilization of the various resources in both computational platforms. While JITT has been contemplated in the past, neither the computational power nor the SW architecture could achieve the types of consistent gains in speed for all the hybrid operations that Blazar is able to accomplish.
The most ideal bitrate ladder would be an “infinite” ladder, offering every possible profile. This would deliver the exact resolution and bitrate that aligns with the client device, the available bandwidth from the Content Delivery Network (CDN), and the type of content, thereby ensuring the highest video quality. This is precisely what the Blazar solution delivers.
In the Blazar JTTT system, only the top most profile—the one with the highest resolution and bitrate—needs to be stored on the DVR storage system. When a client device requests the content, Blazar's Just-In-Time Transcoding (JITT) is capable of immediately converting the content to the best bitrate/resolution profile that suits the client's device type, available last-mile bandwidth, CDN bandwidth, and content type. The system architecture is shown in
When the end device has sufficient bandwidth for the top profile, then the Origin can serve that profile directly from the DVR storage. When the endpoint needs to use less bandwidth, then the Origin re-directs the request to Blazar JITT which retrieves the top profile from storage and instantly transcodes it to the lower profile needed.
In practice an infinite ladder would lead to an infinitely large manifest file that the client would need to parse. Therefore, a practical approximation of an infinite ladder is to construct a ladder with 20-50 profiles, yet only the top profile media is preserved in the DVR storage. The current client can then select one from these 20 or more profiles and request it from the Origin server. Blazar is able to create the specific profile faster than real time. To the end device it appears as if the file was served from the storage server in the same amount of latency. In the future, the concept of static multiple bit rates with different profiles can be replaced by an intelligent client that can make segment by segment requests
To put this into perspective for some use cases. A 500-channel cloud HD DVR storing all the channels for 1 year would require nearly 30 PetaBytes with the static ladder, 11.3 PetaBytes with the Content Aware Ladder, and 7.7 PetaBytes with the Blazar system that only stores the top profile.
This example demonstrates both advantages of Blazar JITT:
Another benefit Blazar JITT provides is an adaptable solution for the ever-evolving landscape of audio and video compression standards. During the codec transition phase, which typically lasts several years, both codec ladders need to be supported by the DVR for streaming to legacy and newer devices. In the past, this has always slowed the introduction of newer codecs in the marketplace. With Blazar JITT, the newer codec can be supported by Blazar JITT. When the number of client devices supporting the newer standard (which is usually more bandwidth efficient) reaches a critical mass, then the entire DVR library can be turned over to the newer codec and legacy devices are now supported with the same Blazar JITT. For instance, today H.264 is the most widely adopted video compression standard in client devices and set-top boxes, with newer devices increasingly integrating H.265 and AV1. The transcode conversion by Blazar would initially be from H.264 to HEVC/AV1. When the client population is more mature with HEVC or AV1, the library could be converted to that technology and all other devices would be supported by the Blazar JITT. This way, there is no requirement to pick technology winners early in the game as well.
H.265 and AV1 demonstrate bitrate efficiency surpassing H.264 by 50% or more. Consequently, employing these more efficient codecs when the end device supports them can result in significant savings in network bandwidth and edge cache storage.
Blazar broadens the concept of a bitrate ladder to encompass the video or audio compression format, effectively multiplying the effective ladder size by the number of codecs supported. A full Blazar ladder for every codec that is needed can be supported with still just a single codec and one top profile stored on the DVR storage.
This is not feasible with static or content aware ladders unless all the codec versions and associated ladders are stored which is an expensive proposition.
Blazar's dynamic ability to switch video codec formats offers several substantial benefits in terms of architecture and cost:
In
In another example, the client may be constrained by computational power or ability to decode specific codecs. In this case, it can request a stream that is most suitable for its capabilities. For example, if the stored profile is a HD HEVC stream 1280×720 at 60 fps encoded at 4 Mbps, the client may request a HD H.264 segment at 60 fps and encoded at 8 Mbps because it is not capable of decoding HEVC streams. Such client requests may also be determined by the power constraints. For example, if the battery level on a mobile device falls below a threshold, the mobile client may request lower level profiles with codec choices that are more power efficient. The client may even request lower resolution profiles at lower bit rates for the same reason. The power of Blazar JITT architecture is that such decisions made by the client can be done instantaneously on a segment by segment basis based on the each client's constraints. Traditional ABR systems have not supported different codec selection on a segment by segment basis. They also only support a limited set of pre-determined profiles with a single codec. With arbitrary profile requests, the Blazar JITT system guarantees the optimal profile delivery tailored to the bandwidth constraint and the computational power constraints of each client.
Currently, adaptive bit rate (ABR) system client starts out with making an asset request. It gets returned, from the streaming server, a manifest file that lays out the available profiles that can be requested. The profiles contain the bit rates and other relevant information on the media segments. In current systems, the number of profiles in the manifest is predetermined and determines the granularity with which client requests can be made.
In our contemplated use of Blazar JITT to support the legacy client, when the asset request is first made, the server returns a manifest with a fixed ABR profile set. The Client calculates the best ABR profile to receive based on the current bandwidth. An extension to this calculation could take into account the available power in the client since a lower profile will generally consume less battery power. When the client sends back a profile request for a segment (from the list of profiles in the manifest), the server determines if the top profile was requested. With the Blazar JITT architecture, this is the only profile that is available (either file or live source). If the top profile is requested, it is returned from the live or file source to the client. If a different profile has been requested, then the JITT is invoked to create the requested profile from the live or file source. The transcoded segment (per the requested profile) is then sent to the client.
An advantage with this architecture is that the master manifest can include a large set of available fixed profiles since the Blazar JITT will make them available when requested. Since all those profiles are not created/stored unless requested, the master manifest can have many more of them compared to a system without Blazar JITT which would have to create and store all the profiles ahead of time.
In the new architecture, there is no predetermined set of profiles except a single top profile.
An alternative architecture to deliver optimal media based on the constraints of network bandwidth and client compute resources is to move the calculation of the segment characteristics to the server side.
In
In both the smart client and smart server architectures discussed above, there are semantic changes in the client or server workflows that are needed in order to leverage the arbitrary profile creation and optimal use of the system. We now discuss a legacy mode support architecture that accommodates legacy devices to still operate with the JITT and take advantage of near-optimal bandwidth use to deliver the best quality.
In this architecture, we revert back to creating a content aware architecture where the server provides the list of possible profiles to the client. The client, based on network bandwidth (and possibly power available) make a profile selection from what has been presented to it. This requires none to very minimal changes in the legacy client software, The server will now have to decide if it needs to invoke the JITT (if the top profile has not been requested) and deliver the appropriately requested profile from the client.
Other variations of JITT could include delivery of a profile that is very similar to one requested by a different end client. The server determines that the profile requested by the current client matches very closely (but not exactly) by a different client and may choose to send the transcoded version of the different client. The threshold of difference in profile request can be set at the server. This choice obviates the need for an additional JITT of the segment.
It must be mentioned that all of the architectures for smart client and smart server extend in the same manner to JIT audio transcoding. Whether it is conversion between mono, stereo and 5.1 or higher channel audio formats or conversions between different audio codecs, the same principles of recreating the requested audio profile on the fly would apply.
The storage and core network bandwidth savings are even higher when the edge caches are considered. In an Adaptive Bitrate (ABR) video streaming system, a Content Delivery Network (CDN) and edge caches work together to deliver a smooth, high-quality viewing experience.
The edge caches store the different video profiles (different bitrates and resolutions) that make up the ABR ladder for a particular video. When a client device requests a video, the edge cache can quickly provide the version of the video that's most appropriate for the device's current network conditions, thus minimizing buffering and maximizing quality.
The device doesn't just request and stick with one profile but dynamically adapts to changing network conditions. When network conditions change, the client device may switch to a different profile (either higher or lower quality), which the edge cache should also have available.
This is why it is beneficial for edge caches to store all the profiles of the ABR ladder rather than just the one that the client device initially requests. By storing all profiles, the edge cache can quickly respond to changes in the client's network conditions and provide the most appropriate video profile.
Of course, this might not be the case for all systems. In some systems with severe storage limitations, the edge cache might store only the most requested profiles. But such a setup could lead to more cache misses and poorer performance if the network conditions change, and the client device needs to switch to a profile that the cache doesn't have.
Blazar JITT at the edge can greatly optimize such a system.
Therefore, the same storage savings as described before can be achieved at each edge point. Depending on the number of edge cache devices this can be a significant multiplying effect of edge cache storage savings. In addition, the network bandwidth from the origin to the edge has the equivalent percent savings since only the top profile needs to be pushed to the edge.
The benefits of this JITT edge approach are multifold
As with the Smart Server or Smart Client architectures referred to in the previous discussion, the edge cache system can have a similar Edge Smart Server or a Smart Client to support the on-the-fly decision making.
In some cases, different clients might request similar but slightly different profiles. In this case, the cache or origin could decide to only generate 1 profile that would be used for both clients. This will reduce the load on the Blazar system and the cache/origin system as well. For example, if 2 clients request 2 bitrates that are within a small threshold different, then only the lower bitrate could be generated by Blazar and used for both.
Blazar JITT transcoder is a software solution that runs on standard Linux servers with GPU compute accelerator hardware. Blazar is available as a software component that can run on any cloud provider such as Google Cloud, AWS, Azure, OCI etc., or is available as an on premise solution (sold as a software license or full turnkey server appliance).
The Segment Processing API can also be used natively or invoked from a real-time transcoding SW stack that supports transcoding/processing applications via an external application API.
The Blazar Software stack is controlled from a simple but powerful API that can be integrated into any workflow The API can be customized for certain use cases or workflows if needed. The hardware can also be selected for optimum cost/performance tradeoffs depending on the system workload.
In some use cases, it makes sense to have Blazar JITT nodes be dynamically spun up or down depending on workloads (peak viewing times for example). This can most easily be done in cloud deployments, but Blazar nodes can also be used as VOD transcoding resources in an on premises deployment when not needed for Blazar functions.
For JITT operation, Blazar takes segmented default profile media files as input (usually fMP4 or MPEG-TS segments), and transcodes them fast enough for the end device playback to not be disturbed any more than if that end device was retrieving content from a storage server. A typical use case is 2 second media segments transcoded 10 times faster than real time which achieves this goal.
In some use cases more than 10× faster than real-time or greater transcode speed are needed and in those cases the Blazar hardware architecture must be scaled to support that. This can be done statically with the hardware deployment up front, or dynamically by combining Blazar JITT nodes together.
This patent application claims the priority and benefits of U.S. Provisional Application No. 63/629,540, filed on Nov. 6, 2023, the disclosure of which is incorporated by reference herein in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63629540 | Nov 2023 | US |
