TRIM-PASS CORRECTION FOR CLOUD-BASED CODING OF HDR VIDEO

Abstract

In a cloud-based system for encoding high dynamic range (HDR) video, each node receives a video segment and bumper frames. Each segment is subdivided into primary scenes and secondary scenes to derive scene-based forward reshaping functions that minimize the amount of reshaping-related metadata when coding the video segment. When a parent scene of a secondary scene is processed by two or more neighboring nodes, initial forward reshaping functions and trim-pass correction parameters are adjusted using reference tone-mapping functions and updated scene-based trim-pass correction parameters.

Description

TECHNOLOGY

The present disclosure relates generally to images. More particularly, an embodiment of the present invention relates to trim-pass correction for processing HDR video in cloud-based coding architectures.

BACKGROUND

As used herein, the term ‘dynamic range’ (DR) may relate to a capability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, e.g., from darkest grays (blacks) to brightest whites (highlights). In this sense, DR relates to a ‘scene-referred’ intensity. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular breadth. In this sense, DR relates to a ‘display-referred’ intensity. Unless a particular sense is explicitly specified to have particular significance at any point in the description herein, it should be inferred that the term may be used in either sense, e.g. interchangeably.

As used herein, the term high dynamic range (HDR) relates to a DR breadth that spans the 14-15 orders of magnitude of the human visual system (HVS). In practice, the DR over which a human may simultaneously perceive an extensive breadth in intensity range may be somewhat truncated, in relation to HDR. As used herein, the terms visual dynamic range (VDR) or enhanced dynamic range (EDR) may individually or interchangeably relate to the DR that is perceivable within a scene or image by a human visual system (HVS) that includes eye movements, allowing for some light adaptation changes across the scene or image. As used herein, VDR may relate to a DR that spans 5 to 6 orders of magnitude. Thus, while perhaps somewhat narrower in relation to true scene referred HDR, VDR or EDR nonetheless represents a wide DR breadth and may also be referred to as HDR.

In practice, images comprise one or more color components (e.g., luma Y and chroma Cb and Cr) wherein each color component is represented by a precision of n-bits per pixel (e.g., n=8). For example, using gamma luminance coding, images where n≤8 (e.g., color 24-bit JPEG images) are considered images of standard dynamic range, while images where n≥10 may be considered images of enhanced dynamic range. HDR images may also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

Most consumer desktop displays currently support luminance of 200 to 300 cd/m² or nits. Most consumer HDTVs range from 300 to 500 nits with new models reaching 1,000 nits (cd/m²). Such conventional displays thus typify a lower dynamic range (LDR), also referred to as a standard dynamic range (SDR), in relation to HDR. As the availability of HDR content grows due to advances in both capture equipment (e.g., cameras) and HDR displays (e.g., the PRM-4200 professional reference monitor from Dolby Laboratories), HDR content may be color graded and displayed on HDR displays that support higher dynamic ranges (e.g., from 1,000 nits to 5,000 nits or more).

As used herein, the term “forward reshaping” denotes a process of sample-to-sample or codeword-to-codeword mapping of a digital image from its original bit depth and original codewords distribution or representation (e.g., gamma, PQ, HLG, and the like) to an image of the same or different bit depth and a different codewords distribution or representation. Reshaping allows for improved compressibility or improved image quality at a fixed bit rate. For example, without limitation, reshaping may be applied to 10-bit or 12-bit PQ-coded HDR video to improve coding efficiency in a 10-bit video coding architecture. In a receiver, after decompressing the received signal (which may or may not be reshaped), the receiver may apply an “inverse (or backward) reshaping function” to restore the signal to its original codeword distribution and/or to achieve a higher dynamic range.

In many video-distribution scenarios, HDR video may be coded in a multi-processor environment, typically referred to as a “cloud computing server.” In such an environment, trade-offs among ease of computing, workload balance among the computing nodes, and video quality, may force reshaping-related metadata to be updated on a frame-by-frame basis, which may result in unacceptable overhead, especially when transmitting video at low bit rates. Splitting of a scene into multiple computing nodes may create temporal inconsistencies in sub-scenes, which may affect how trim-pass data affect the output video. As appreciated by the inventors here, improved techniques for trim-pass correction in a cloud-based environment are desired.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not assume to have been recognized in any prior art on the basis of this section, unless otherwise indicated.

BRIEF DESCRIPTION OF THE DRAWINGS

An embodiment of the present invention is illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1A depicts an example single-layer encoder for HDR data using a reshaping function according to prior art;

FIG. 1B depicts an example HDR decoder corresponding to the encoder of FIG. 1A, according to prior art;

FIG. 2 depicts an example architecture and processing pipeline for cloud-based encoding of HDR video according to an embodiment;

FIG. 3A depicts an example division of a video input into segments and bumper frames to be assigned into three nodes;

FIG. 3B depicts an example of merging scene cuts to generate a list of primary scenes;

FIG. 3C depicts an example of a primary scene split into two computing nodes;

FIG. 3D and FIG. 3E depict examples of trim-pass windows used to derive updated trim-pass correction parameters according to embodiments;

FIG. 4A depicts an example of an iterative segment to scene segmentation process according to an embodiment;

FIG. 4B depicts an example process to generate updated scene-based trim-pass correction parameters according to an embodiment;

FIG. 4C depicts an example process to generate trim-corrected forward reshaping functions according to an embodiment;

FIG. 5 depicts an example encoder for scene-based encoding using reshaping according to an embodiment of this invention; and

FIG. 6A and 6B depict examples of trim-pass correction of forward reshaping functions according to embodiments of this invention.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Methods for trim-pass correction in cloud-based video coding of HDR video are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating the present invention.

SUMMARY

Example embodiments described herein relate to cloud-based reshaping and coding for HDR images. In an embodiment, in a cloud-based system for encoding HDR video, a current node receives a first video sequence comprising video frames in a high dynamic range. Then, one or more processors in the node:

• • generate (455) for each video frame in the first video sequence a first forward reshaping function and frame-based trim-pass correction parameters, wherein a forward reshaping function maps frame pixels from the high dynamic range to a second dynamic range lower than the high dynamic range;
  • for a subscene in the first video sequence that is part of a parent scene to be processed by the current computing node and one neighbor computing node:
    • determine (457) a trim-pass parameter window comprising frames in the first video sequence and bumper frames, wherein the bumper frames comprise video frames to be processed by the neighbor computing node;
    • compute (459) scene-based trim-pass correction parameters based on the frame-based trim-pass correction parameters of the frames in the trim-pass parameter window;
    • generate a scene-based forward reshaping function based on the first forward reshaping functions of the frames in the subscene; and
    • generate for the subscene an output forward reshaping function based on the scene-based trim-pass correction parameters and the scene-based forward reshaping function.

Example HDR Coding System

FIG. 1A and FIG. 1B illustrate according to prior art an example single-layer backward-compatible codec framework using image reshaping. More specifically, FIG. 1A illustrates an example encoder architecture, which may be implemented with one or more computing processors in an upstream video encoder. FIG. 1B illustrates an example decoder architecture, which may also be implemented with one or more computing processors in one or more downstream video decoders.

Under this framework, given reference HDR content (120) and corresponding reference SDR content (125) (that is, content that represents the same images as the HDR content, but color-graded and represented in standard dynamic range), reshaped SDR content (134) is encoded and transmitted as SDR content in a single layer of a coded video signal (144) by an upstream encoding device that implements the encoder architecture. The reshaped and subsequently compressed SDR content is received and decoded, in the single layer of the video signal, by a downstream decoding device that implements the decoder architecture. Backward-reshaping metadata (152) is also encoded and transmitted in the video signal with the reshaped content so that HDR display devices can reconstruct HDR content based on the (reshaped) SDR content and the backward reshaping metadata. Without loss of generality, in some embodiments, as in non-backward-compatible systems, reshaped SDR content may not be watchable on its own but must be watched in combination with the backward reshaping function, which will generate watchable SDR or HDR content. In other embodiments which support backward compatibility, legacy SDR decoders can still playback the received SDR content without employing the backward reshaping function.

As illustrated in FIG. 1A, given HDR image (120) and SDR image (125), and a target dynamic range, step 130 generates a forward reshaping function. Given the generated forward reshaping function, a forward reshaping mapping step (132) is applied to the HDR images (120) to generate reshaped SDR base layer (134). A compression block (142) (e.g., an encoder implemented according to any known video coding algorithms, like AVC, HEVC, AV1, and the like) compresses/encodes the SDR images (134) in a single layer (144) of a video signal. In addition, a backward reshaping function generator (150) may generate a backward reshaping function which may be transmitted to a decoder as metadata (152). In some embodiments, metadata (152) may represent the forward reshaping function (130), thus, it would be up to the decoder to generate the backward reshaping function (not shown).

Examples of backward reshaping metadata representing/specifying the optimal backward reshaping functions may include, but are not necessarily limited to only, any of: an inverse tone mapping function, inverse luma mapping functions, inverse chroma mapping functions, lookup tables (LUTs), polynomials, inverse display management coefficients/parameters, etc. In various embodiments, luma backward reshaping functions and chroma backward reshaping functions may be derived/optimized jointly or separately, may be derived using a variety of techniques, for example, and without limitation, as described later in this disclosure.

The backward reshaping metadata (152), as generated by the backward reshaping function generator (150) based on the reshaped SDR images (134) and the target HDR images (120), may be multiplexed as part of the video signal 144, for example, as supplemental enhancement information (SEI) messaging.

In some embodiments, backward reshaping metadata (152) is carried in the video signal as a part of overall image metadata, which is separately carried in the video signal from the single layer in which the SDR images are encoded in the video signal. For example, the backward reshaping metadata (152) may be encoded in a component stream in the coded bitstream, which component stream may or may not be separate from the single layer (of the coded bitstream) in which the SDR images (134) are encoded.

Thus, the backward reshaping metadata (152) can be generated or pre-generated on the encoder side to take advantage of powerful computing resources and offline encoding flows (including but not limited to content adaptive multiple passes, look ahead operations, inverse luma mapping, inverse chroma mapping, CDF-based histogram approximation and/or transfer, etc.) available on the encoder side.

The encoder architecture of FIG. 1A can be used to avoid directly encoding the target HDR images (120) into coded/compressed HDR images in the video signal; instead, the backward reshaping metadata (152) in the video signal can be used to enable downstream decoding devices to backward reshape the SDR images (134) (which are encoded in the video signal) into reconstructed images that are identical to or closely/optimally approximate the reference HDR images (120).

In some embodiments, as illustrated in FIG. 1B, the video signal encoded with the reshaped SDR images in the single layer (144) and the backward reshaping metadata (152) as a part of the overall image metadata are received as input on the decoder side of the codec framework. A decompression block (154) decompresses/decodes compressed video data in the single layer (144) of the video signal into the decoded SDR images (156). Decompression 154 typically corresponds to the inverse of compression 142. The decoded SDR images (156) may be the same as the SDR images (134), subject to quantization errors in the compression block (142) and in the decompression block (154), which may have been optimized for SDR display devices. In a backward-compatible system, the decoded SDR images (156) may be outputted in an output SDR video signal (e.g., over an HDMI interface, over a video link, etc.) to be rendered on an SDR display device.

Optionally, alternatively, or in addition, in the same or another embodiment, a backward reshaping block 158 extrats the backward (or forward) reshaping metadata (152) from the input video signal, constructs the backward reshaping functions based on the reshaping metadata (152), and performs backward reshaping operations on the decoded SDR images (156) based on the optimal backward reshaping functions to generate the backward reshaped images (160) (or reconstructed HDR images). In some embodiments, the backward reshaped images represent production-quality or near-production-quality HDR images that are identical to or closely/optimally approximating the reference HDR images (120). The backward reshaped images (160) may be outputted in an output HDR video signal (e.g., over an HDMI interface, over a video link, etc.) to be rendered on an HDR display device.

In some embodiments, display management operations specific to the HDR display device may be performed on the backward reshaped images (160) as a part of HDR image rendering operations that render the backward reshaped images (160) on the HDR display device.

Cloud-Based Coding

Existing reshaping techniques may be frame-based, that is, new reshaping metadata is transmitted with each new frame, or scene-based, that is, new reshaping metadata is transmitted with each new scene. As used herein, the term “scene” for a video sequence (a sequence of frames/images) may relate to a series of consecutive frames in the video sequence sharing similar luminance, color and dynamic range characteristics. Scene-based methods work well in video-workflow pipelines which have access to the full scene; however, it is not unusual for content providers to use cloud-based multiprocessing, where, after dividing a video stream into segments, each segment is processed independently by a single computing node in the cloud. As used herein, the term “segment” denotes a series of consecutive frames in a video sequence. A segment may be part of a scene or it may include one or more scenes. Thus, processing of a scene may be split across multiple processors.

As discussed in Ref. [1], in certain cloud-based applications, under certain quality constraints, segment-based processing may necessitate generating reshaping metadata on a frame-by-frame basis, resulting in undesirable overhead. This may be an issue in very low bit-rate applications (e.g., lower than 1 Mbit/s). Ref. [6] proposed a solution to this problem using a two stage architecture which includes: a) a dispatcher stage implemented on a single computing node, which allocates scenes into segments and b) an encoding stage, where each node in the cloud encodes a sequence of segments. After a scene is segmented, the proposed scene to segment allocation process includes one or more iterations with an initial random assignment of scenes to nodes, followed by a refined assignment based on optimizing the allocation cost across all the nodes. In such an implementation, the total length of video to be processed in each node may vary across all the nodes.

Embodiments discussed in Ref. [7] provided an alternative solution. After a sequence is divided into segments, each segment to be processed by a separate node, in each node, each segment is sub-divided into sub-segments (or scenes) in such a way to minimize the need to update the corresponding reshaping function of each sub-segment, thus minimizing the overhead required to transmit reshaping-related metadata.

In some embodiments, a reference SDR signal (e.g. 125) based on HDR signal (120) may not have the desired “look.” Then, a colorist may adjust “trim parameters” (commonly referred to as lift, gain, and gamma (LGG)) to achieve the desired effect. This process may be referred to as a “trim pass,” and its main goal is to maintain the director's intent or look. As used herein, the term “trim pass” denotes a post-production process which may include trimming, rotating, cropping, flipping and adjusting the brightness, color and saturation of a video sequence to generate a sequence to be displayed on a display with a target dynamic range, typically lower than the dynamic range of the master. For example, given a movie mastered at 4,000 nits, a colorist may generate “trims” at 400, 500, and 1,000 nits. Such a trim pass may preserve artistic intent in the SDR signal, but after reshaping, it may also introduce unpleasant clipping artifacts in the darks (the low intensity regions) or in the highlights (the high intensity regions). Such artifacts may be further exaggerated by the video-to-nodes segmentation process and/or the video coding process (142). To reduce such artifacts, one could try to change the LGG “trim” data itself; however, studios do not allow any changes of their data after a director's approval, and any such change would require an additional review process. Hence, as appreciated by the inventors, it would be beneficial to be able to reduce artifacts introduced by the HDR to SDR mapping process. Such a methodology is described next.

As it was disclosed first in Ref. [7], FIG. 2 depicts an example architecture and processing pipeline for cloud-based encoding of HDR video according to an embodiment. Given a video source (202) for content distribution, typically referred to as a mezzanine file, and a set of working nodes, each node (e.g., node 205-N) fetches video pictures (or frames) and corresponding video metadata (207) (e.g., from an XML file) to be processed as follows.

In preprocessing step 210, the mezzanine input is split into segments and each segment is assigned to a different computing node (e.g., node 205-N). These segments are mutually exclusive, i.e. they have no frames in common. Each node will also get a certain number of frames that are before the first frame in the segment and some frames after the last frame in the segment. These prior and post overlapped frames, called bumper frames, are only used for maintaining temporal consistency with the previous and the next node respectively. Bumper frames are not encoded by the node. Without loss of generality, in an embodiment, these video segments may be all be of equal, fixed-length, except perhaps for the segment assigned to the last node. As an example, a sample distribution of a mezzanine (305) into three segments (307-1, 307-2, 307-3) along with their bumper frames (e.g., 309), and their assignment of these frames to different nodes is illustrated in FIG. 3A. Without limitation, for segments 30 sec long and bumper sections 2 sec long on each side, example embodiments may include any of the following arrangements:

• • Segment sections with 1,800 frames and bumper sections with 120 frames, at 60 fps
  • Segment sections with 1,500 frames and bumper sections with 100 frames, at 50 fps
  • Segment sections with 720 frames and bumper sections with 48 frames, at 24 fps

After the preprocessing step 210 is over, each node gets access to its frames and a two-pass approach follows.

• • In Pass-1 (steps 215, 220), a list of scenes (222) in the segment is generated. Scene cuts (209) extracted from the XML file and scene cuts generated using an automatic scene cut detector (215) are combined in step 220 to get a first list of primary scenes. Primary scenes that belong to a parent scene to be coded across multiple nodes may be subdivided to secondary scenes. To maintain temporal consistency in scenes distributed across multiple nodes, bumper frames and a novel recursive scene splitting algorithm is also provided. Splitting produces additional scenes that are added to the first list of scenes to get a second list. This second list of scenes (222) is handed over to Pass-2. Auxiliary data (212) required to correct trim-pass data is also pass to Pass-2 to reduce computation overhead in Pass-2.
  • Pass-2 (steps 225, 230, and 235) uses the list of scenes received from Pass-1 to perform forward and backward reshaping for each scene in the segment. Trim-pass data correction is performed in the forward reshaping step (225). Forward reshaping using scene-based forward reshaping functions (227) produces a reshaped SDR segment (229) while the backward reshaping unit (235) generates the metadata parameters to be used by a decoder to reconstruct the HDR input. The reshaped SDR input (229) is compressed (230), and compressed video data and reshaping metadata are combined together to generate a compressed bitstream (240).

For ease of discussion, let L denote the number of frames in a segment, and let B denote number of frames in each bumper section. Let the i-th frame in the mezzanine be denoted as f_i. In an embodiment, the first node encodes the frames f₀˜f_L−1 that are in the segment portion. This node has no left bumper, and its right bumper spans the frame range f_L˜f_L+B−1. The segment portion of node-N will process frames f_(N−1)L˜f_NL−1, with f_(N−1)L−B˜f_(N−1)L−being the left bumper and f_NL˜f_NL+B−1 being the right bumper. The last node will have no right bumper section and it may have less then L frames in the segment portion.

Given a node N, node N−1 is the left/previous neighbor-node and node N+1 is the right/next, or subsequent, neighbor. Referring to nodes that are left/previous nodes to N includes all the nodes from 0 to N−1. Similarly, referring to nodes that are right/next to N denotes all the nodes from N+1 to the last node. The two passes described earlier will now be discussed in further detail.

Pass-1: Segment to Scene Generation

The key objective of this pass is to generate a list of scenes in the segment allocated to a node. The process starts by detecting scene cuts in all the frames allocated to the node, which includes the node segment and both the bumper sections. Only those scene cuts inside the segment will eventually be used by the Pass-2 for scene-based encoding. But the scenes in the bumper section are still useful for maintaining temporal consistency with the neighboring nodes.

Colorist-specified scene cuts (209) are read in from the XML file (207). An automatic scene cut detector (215) may also identify possible scene cut locations. These scene cuts from the colorists and the automatic detector are merged to get a first list of scenes, known as primary scenes. Primary scenes on the segment boundaries are split using bumper frames and a novel scene splitting technique. Splitting of a primary scene on a segment boundary creates additional scenes, known as secondary scenes or subscenes. Secondary scenes are added to the first list of scenes to get a second list. This list is then used by the Pass-2 for scene-based encoding. Apart from the list of scenes, Pass-2 may also need auxiliary data (212) for forward reshaping of the secondary scenes. Details for each step are provided next.

Colorists and professional color-graders usually process each scene as a single unit. To meet their goals (e.g., proper color grading, inserting fade-ins and fade-outs, etc.), they need to manually detect scene cuts in the sequence. This information is stored in the XML file and can be used for other purposes as well. Every node will read only the relevant scene cuts for its segment from the XML file. These scene cuts may be in the segment section or in the bumper sections.

Even though XML scene cuts are defined by the colorists, they are not completely accurate. For grading purposes, sometimes colorists introduce scene cuts in the middle of a dissolving scene or at the start of fade in or fade out portion of a scene. These scene cuts, if taken into consideration during the reshaping phase, may cause flashing in the reconstructed HDR video and normally should be avoided. For this reason, in an embodiment, an automatic scene-cut detector (Auto-SCD) 215 is also employed.

An automatic scene-cut detector or Auto-SCD uses the change in luminance levels in different sections of consecutive video pictures to detect a scene change. Any scene cut detector known in the art can be used as the automatic detector. In an embodiment, such an automatic detector is oblivious to dissolving, fade in or fade out parts of the video and it can still detect all the true scene cuts correctly.

A potential problem with an automatic detector is false positives. Sometimes there are brightness changes within a scene due to camera panning, movements, occlusions etc. These brightness changes may also be detected as scene cuts by the Auto-SCD. To discard these false positives, in an embodiment, the scene cuts from the XML file and those from the and Auto-SCD are merged together in step 220. A person skilled in the art will appreciate that if there are no scene cuts defined in the XML file one may simply use the output of the automatic scene detector. Similarly, in other embodiments, one may rely strictly on scene cuts defined in the XML file. Alternatively, one may also use more than two scene-cut detectors, where each one detects different attributes of interest, and then define the primary scenes based on a combination of all of their results (e.g., their intersection or a combination of other set operations, e.g., their union, intersection, and the like).

Let Ψ_XML^N be the set of frame indices representing scene start frames in node N as reported in the XML file. Similarly, let ψ_Auto-SCD^N denote the set of frame indices representing scene start frames in node N as reported by Auto-SCD. In an embodiment, merging the scene cuts from these two sets is equivalent to taking the intersection of these two sets.

Ψ₁^N=Ψ_XML^N∩Ψ_Auto-SCD^N,   (1)

where, Ψ₁^N indicates the first list of scene cuts (or scenes) in the node N. These scenes are also known as primary scenes. FIG. 3B demonstrates an example scenario. In this example, the XML file reports three scene cuts. The Auto-SCD reports also three scene cuts; two are the same as the XML scene cuts, but the third one is in a different location. Since between the six reported scene cuts only two are common, the node segment is divided into only three primary scenes (310-1, 310-2, and 310-3) according to the two common scene cuts. In some embodiments, XML and Auto-SCD scene cuts may be recognized as the same even if they are reported on different frames, provided the scene-cut indices between the two lists differ within a given small tolerance (±n frames, e.g., n in [0 ,6]).

As depicted in FIG. 3B, primary scene 2 (310-2) is entirely in node N. Therefore, it can be processed entirely by node N in Pass-2. On the contrary, primary scenes 1 (310-1) and 3 (310-3) are on the segment boundary. Their parent scenes are distributed across multiple nodes and processed by those nodes independently. There has to be some special handling for primary scenes 1 and 3 to ensure consistent look in boundary frames that are encoded by different nodes. A number of alternative scenarios are examined next.

As depicted in FIG. 3C, consider a simple scenario where P is a parent scene distributed across two nodes N and N+1. Node N and N+1 have access to only a part of the parent scene. Let us assume that these nodes process and encode their respective portions of the parent scene (without bumpers). Reshaping parameters are computed on different sets of frames, so the reshaped SDR and reconstructed HDR for last frame in segment of node N, i.e. f_(N+1)L−1 and first frame in segment of node N+1 i.e. f_(N+1)L may look visually different. Such a visual difference is usually manifested in the form of flickering, flashing and sudden brightness change. This problem is referred to as temporal inconsistency across nodes. Part of the reason for inconsistencies in the above scenario is the lack of common frames while computing reshaping parameters. As shown in FIG. 3C, if the bumper frames are included in generating these statistics, it will provide smoother transition across nodes. But the bumper sections may be relatively short compared to the parent scene, so they may not be long-enough to ensure temporal consistency. In an embodiment, to solve these problems, portions of the parent scene in nodes N and N+1 are split into secondary scenes or subscenes. Even if reshaping statistics in a scene change significantly from one node to the next, those statistics do not change much from one frame to the next. Secondary scenes will only use statistics in a small neighborhood to evaluate reshaping parameters. Therefore, these reshaping parameters do not change much from one subscene to the next. In this way, splitting achieves temporal consistency. Note that the neighboring subscenes can be on the previous/next node as well.

It should be noted that splitting creates additional scenes and thus increases the metadata bitrate. The challenge is to achieve temporal consistency using a minimum number of splits to keep the metadata bitrate low. Bumper frames play a significant role in achieving a good visual quality while reducing the number of splits.

• • 1. Bumper frames help the splitting algorithm to mimic the splits that the previous/next node is going through. Valuable insights gained by mimicking splits on other nodes help minimize the number of splits.
  • 2. A smoother transition can be achieved at segment boundaries by using bumper frames for computing reshaping parameters.A scene splitting algorithm, described first in Ref. [7], is provided in the following subsections; however, a person skilled in the art would appreciate that the trim-pass corrections proposed herein are independent of the scene-splitting algorithm being used and are applicable regardless of the methodology to derive primary and secondary scenes. Discussion starts with splitting a parent scene without considering a multi-node allocation, then the method is extended to scene splitting for a parent scene distributed across two or more neighboring nodes.

Consider a case with a parent scene P with M frames (M>1) ranging from the Q -th index frame to the Q+M−1 frame in the mezzanine. FIG. 4A depicts an example process (400) for splitting a primary scene to subscenes according to an embodiment. The goal of the process is to divide a primary scene into subscenes with “temporal stability” or “temporal consistency.” All frames in each subscene will be reshaped with the same scene-based reshaping function. Thus, temporal stability, allows for reduced number of reshaping metadata while maintaining the video quality at a given bitrate.

Process 400 starts with an initialization step 410, where, given input HDR and SDR frames (405) for primary scene P, HDR and SDR histograms h^v and h^s, and individual forward reshaping functions {tilde over (T)}_j^F (FLUTs) are computed for each frame in scene P. As an example, and without limitation, given frame histograms, one can apply cumulative density function (CDF) matching (Ref. [4-5]) to generate the forward mapping function (FLUT) from HDR to SDR, e.g.,

{tilde over (T)}^F=CDF_MATCHING(h^v(b),h^s(b)).  (2)

thus, for the j-frame, this step generates:

{tilde over (T)}_j^F, h_j^v∀j∈[Q, Q+M−1],  (3)

where h_j^v denotes a histogram of HDR frame j.

The segmentation methods described herein are agnostic on how frame-based reshaping functions are generated. Thus, in an embodiment, such reshaping functions may be generated directly from the available HDR video using any of the known reshaping techniques and without any dependency on the availability of a corresponding SDR video.

A scene FLUT {tilde over (T)}_P^F is constructed for P by averaging all the frame FLUTs in the scene. In the following equation, b indicates the index in the FLUT. In an embodiment, FLUT values may be normalized, i.e. {tilde over (T)}_j^F (b)∈[0.0, 1.0].

$\begin{array}{cc}{\tilde{T}}_P^F\left(b\right)=\frac{1}{M}{\sum }_{j=Q}^{Q+M-1}{\tilde{T}}_j^F\left(b\right).& \left(4\right)\end{array}$

The scene FLUT and the generated histograms are used to predict a “DC” value χ_j for every frame in scene P. If the height and width of a frame are H and W respectively, then its DC value is computed as

$\begin{array}{cc}{\chi }_j=\frac{1}{H\times W}{\sum }_b{\tilde{T}}_P^F\left(b\right)\times h_j^v\left(b\right)\forall j\in \left[Q,Q+M-1\right].& \left(5\right)\end{array}$

In an embodiment, the DC difference of every frame with its previous frame, denoted as ℑ_j^DC, is used as one set of thresholds to make the splitting decisions. These ℑ_j^DC values are calculated once during the initialization phase and are used several times during the splitting process:

ℑ_j^DC=χ_j−χ_j−1∀j∈[Q+1, Q+M−1].  (6)

The maximum absolute element-wise difference between the FLUT of every frame with its previous frame's FLUT is stored also at the initialization stage to be used as an additional set of thresholds for detecting smoothness violations,

ℑ_j^FLUT=max(α×max(({|{tilde over (T)}_j^F(b)−{tilde over (T)}_j−1^F(b)||∀b}), β) ∀j∈[Q+1, Q+M−1],   (7)

where α and β are configurable parameters, with typical values 2.0 and 0.60, respectively

Secondary scene cuts C_g are collected in a sorted subscene set Ω_p, where g is an index in the set. The frame index Q+M acts as the end of the list marker and is not used as a scene cut. In an embodiment, secondary scene cuts at initialization are as follows:

Ω_p={Q, Q+M}={C₀, C₁},   (8)

In an embodiment, a violation subscene set Y is used to store the subscenes that violate the smoothness criteria. To start splitting parent scene P, at initialization, ={P}. Only the scenes or subscenes in the violation set will be split later on. In summary, in step 410, the initialization step generates: ℑ_j^DC and ℑ_j^FLUT values, a violation set , and a sorted set of scene cuts Ω_p.

In step 415, given a violation set and a sorted set of secondary scene cuts Ω_p as the input, a new round of subscene splitting begins. One iterates through all the subscenes in violation set and decides on how to split them.

Let P_g be a subscene in the violation set that spans the frame range [C_g, C_g+1−1]. For splitting, one compares subscene FLUT {tilde over (T)}_Pg^F with every frame FLUT {tilde over (T)}_j^F in the subscene and selects as the split frame the one at index location C_s with the most similar FLUT,

$\begin{array}{cc}{C}_s=\underset{j}{\min }{\sum }_b❘{\tilde{T}}_{P_g}^F\left(b\right)-{\tilde{T}}_j^F\left(b\right)❘,& \left(9\right)\end{array}$

where the mathematical operator |⋅| denotes the absolute value.

After the split, the subscene P_g is divided into two subscenes or secondary scenes and the new splitting index is inserted into the secondary set at the correct location.

Ω_p=Ω_p∪{C_s}.  (10)

All the new splits from all the subscenes in the violation set are inserted into the set Ω_p in a sorted manner The violation set is set to an empty set after iterating through every subscene in it. The updated set Ω_p is passed on to the next step in the splitting process.

In step 420, new subscene FLUTs are computed for every secondary scene in the updated set Ω_p. Suppose at this time, the set Ω_p contains G+1 secondary scene cuts from C₀ to C_G:

Ω_p={C₀, C₁ . . . C_g . . . C_G−1, C_G}.  (11)

There are G subscenes in this iteration round, and the frame indices in the set Ω_p are in ascending order, i.e.

Q=C ₀ <C ₁ . . . <C _g < . . . <C _G−1 <C _G =Q+M.  (12)

Consider subscene P_g that spans the frame range [C_g, C_g+1−1]. To build a subscene FLUT, i.e. {tilde over (T)}_Pg^F, for P_g for g∈[0, G−1], a subscene overlap parameter θ is introduced to allow a small overlap between neighboring subscenes.

Θ′=max (Q, C_g−θ),

Θ″=min (Q+M−1, C_g+1−1+θ).  (13)

The overlap frames are used to estimate the forward LUT for the subscene P_g by averaging the FLUTs in the subscene and the overlap portion.

$\begin{array}{cc}{\tilde{T}}_{P_g}^F\left(b\right)=\frac{1}{{\Theta }^{″}-{\Theta }^{\prime }+1}{\sum }_{j={\Theta }^{\prime }}^{{\Theta }^{″}}{\tilde{T}}_j^F\left(b\right),\forall g\in \left[0,G-1\right].& \left(14\right)\end{array}$

In the current round of the splitting process, let the DC value be defined by λ. These DC values will be used later on in step 425 to find threshold violations at the subscene boundaries. Let

$\begin{array}{cc}{\lambda }_j=\frac{1}{H\times W}{\sum }_b{\tilde{T}}_{P_g}^F\left(b\right)\times h_j^v\left(b\right)\forall j\in \left[C_g,C_{g+1}-1\right],\forall g\in \left[0,G-1\right].& \left(15\right)\end{array}$

These new DC values for all the frames in the primary scene P are collected after iterating through all the subscenes in Ω_p and computing statistics in them.

In step 425, temporal stability violations at boundaries between subscenes are detected. For example, for secondary scenes P_g−1 in {C_g−1, C_g−1} and P_g in {C_g, C_g+1−1}, a boundary check needs to be computed at C_g. If any one of the checks fail, then both subscenes P_g−1 and P_g are moved to the violation set . For subscenes P_g and P_g+1, a boundary check needs to be computed at C_g+1. The same checks are applied at each subscene boundary C_g, except at the first frame of the segment, C₀ (Q), and the last frame of the segment, Q+M−1=C_G−1.

Using equation (15), updated DC values (λ_j) are available for all the frames in the primary scene P after iterating through all the subscenes in Ω_p. These values will be used in steps 425 and 430 to perform boundary-violation checks. The DC difference Δ_cg, is the difference between DC values of frame with index C_g and its previous frame with index C_g−1.

Δ_cj=λ_cg−λ_cg−1.  (16)

Violation Check #1:

Is it true that |Δ_cg|>|ℑ_cg^DC|+ω?  (17)

If absolute DC difference |Δ_cg| is greater than DC threshold |ℑ_cg^DC| plus a non-negative bias ω, then there is a brightness change at the transition point C_g between subscenes P_g−1 and P_g. This is a threshold violation and these subscenes are appended to the violation set . Here, bias ω denotes a configurable parameter with the default value being 0.001 (or ¼ codeword in 8-bit). This is the most important violation check in the splitting process as it has more impact on the splitting results than the next two checks.

Violation Check #2:

Is it true that sign (Δ_cg)≠sign(ℑ_cg^DC)?  (18)

The sign(x) (or signum) operator for a real number x is defined as follows:

$\mathrm{sign}\left(x\right)=\{\begin{array}{c}-1\mathrm{if}x<0\\ 0\mathrm{if}x=0\\ 1\mathrm{if}x>0\end{array}.$

A positive DC difference Δ_cg indicates that the DC value has increased from the previous frame to the current frame. Similarly, a negative DC difference indicates that the DC value has decreased. The general trend of increasing/decreasing DC value from previous frame to the next frame should be maintained after splitting in the subscenes. So, if threshold ℑ_cg^DC and DC difference Δ_cg have opposite signs at point C_g, then both the subscenes P_g−1 and P_g should be split further by adding them to the violation set . This criterion is helpful in maintaining temporal stability in fade in/out, dissolving scenes or any scene with gradual brightness changes.

Violation Check #3:

Is it true that max ({|{tilde over (T)}_Pg^F(b)||∀b})ℑ_Cg^FLUT?  (19)

If maximum of absolute element-wise difference between FLUTs {tilde over (T)}_Pg^F and {tilde over (T)}_Pg−1^F is greater than threshold ℑ_Cg^FLUT at C_g, then these subscenes P_g−1 and P_g violate smoothness constraint and should be included in .

All the violation checks are at subscene boundaries. If there is a violation, then both subscenes are entered into the violation set. This ends the current round of splitting. At step 430, if the updated violation set is not empty, control goes back to step 415 with the updated Ω_p and sets for the next round of splitting. Otherwise, if there are no boundary violations and the violation set is empty, the process terminates and step 440 outputs the final secondary set of subscenes. In an embodiment, in step 425, if a secondary scene in is only one frame long, it can be removed from the set, since it is impossible to be further split. Alternatively, such single-frame scenes can be ignored in step 415.

In practice, a parent scene is sub-divided only if it is processed across two or more nodes. For example, a node may look for scene cuts at the left and right bumper sections. If no such scene cuts are detected, then it can deduce that the beginning or end of its segment is processed by neighboring nodes as well, thus, one or more primary scenes need to be subdivided.

Consider a scenario shown in FIG. 3C, where a parent scene Pis processed by two nodes. Each portion of the parent scene in a node is the primary scene for that node. One approach is to split them independently using the splitting approach described earlier. Then, because of missing statistics, scene cuts in the overlapping regions of the nodes N and N+1 may not match each other. The proposed splitting algorithm works much better in resolving temporal inconsistencies if it has a good estimation of secondary scene boundaries in the neighboring subscenes on the neighbor node.

In an embodiment, for the example in FIG. 3C, two new synchronization subscene cuts (320) in those two primary scenes are introduced, one in each node. These synchronization cuts split the primary scene into two portions:

• • 1. A first portion (e.g., between the first cut and the first sync cut in node N) is visible to the current node but not visible to the other node. As shown in FIG. 3C, in an embodiment, the first sync cut, on node N, may be at position C_L−1−B, where B denotes the number of bump frames and C_L−1 denotes the last frame of the segment.
  • 2. A second portion (e.g., the end bumper frames for node N and the initial bumper frames for node N+1) is “visible” to both nodes. As shown in FIG. 3C, in an embodiment, the second sync cut, on node N+1, may be at position C₀+B, where C₀ denotes the first frame of the segment.

In an embodiment, these initial sync splits may be performed as part of step 410, and one can apply the splitting algorithm 400 on these primary scenes. The only minor change will be in the initialization step 410, where the set Ω_p for each node will include one additional synchronization scene cut (320). Then, since there is no need to do further splits one can directly jump to step 420 after initialization. Next, the algorithm proceeds as usual.

Alternatively, given the original Ω_p set, upon detecting that a primary scene is not fully in the current node, this sync subdivision may be performed in step 415 using the rules described earlier (e.g., for node N, if the primary scene does not terminate at node N, adding a scene cut at position C_L−1−B) in lieu of using equation (9).

With these initial synchronization splits, the subscene cuts computed by nodes N and N+1 in isolation are expected to be reasonably aligned with each other. For node N, let Ψ₁^N denote a first list of scenes obtained after merging the XML scene cuts with Auto-SCD scene cuts as seen in FIG. 3B. These scenes are termed as primary scenes. The scenes that lay on the segment boundaries are split into secondary scenes or subscenes. Secondary scenes or subscenes generate additional scenes cuts that are appended to the first list. Apart from these secondary scene cuts, the first frame of the segment f_NL is also a scene cut. The node can only start encoding from start of the segment, so that frame is treated as a scene cut. Similarly, the last frame of the segment f_(N+1)L−1 is the end of the last scene in the segment. For example, consider node N with the following initial allocation: Primary scenes 1, 2, and 3. Primary scene 1 is subdivided into secondary scenes A, B, and C, primary scene 2 remains unchanged, and primary scene 3 may be divided into secondary scenes D, E, and F. Suppose Ψ_l^N and Ψ_r^N indicate the set of scene cuts near the left and right segment boundary respectively for node N. Then a second list of scenes Ψ₂^N can be mathematically represented as:

Ψ₂^N=Ψ₁^N∪Ψ_l^N∪Ψ_r^N∪f_NL  (20)

There is a possibility that for a scene longer than a segment length, there may not be separate left or right sets, but a single set of secondary scene cuts. Let S_k denote the starting frame index for the k^th scene in the list Ψ₂^N. Suppose there are K scenes in the list, then the elements in the list can be expressed as:

Ψ₂^N={S₀, S₁, S₂ . . . S_k−1, S_k,S_k+1 . . . S_k}  (21)

where S_K denotes a dummy scene cut that is just after the last frame of the segment. It is only used as an end-of-list marker. By default, S₀=f_NL as the first frame of the segment is also a beginning of a new scene for node N. The second list of scenes Ψ₂^N (222) is handed over to Pass-2 along with subscene-related auxiliary data.

The second list of scenes Ψ₂^N has details about the primary and secondary scenes in the segment. Primary scenes do not need any additional data from Pass-1, but secondary scenes require the following auxiliary data from Pass-1.

• • 1. The number of overlapping frames on the left and right side for every secondary scene.
  • 2. Trim pass correction data, which may include one or more of:
    • a. the mean HDR luma value for every subscene
    • b. the mean SDR luma value for every subscene
    • c. a merging point in the low-intensity region for every subscene (the darks)
    • d. a merging point in the high-intensity region for every subscene (the highlights)Collectively, the trim-pass correction data may also be referred to as trim-pass parameters, trim parameters, trim-pass correction parameters, or trim correction parameters.

The term “merging point” refers to a luminance value in the SDR or HDR domain which acts as a boundary point to modify an original forward reshaping function (FLUT) to correct for potential effects of trim-pass operations. For example, given the low and high merging points and in a set (or window) of consecutive frames denoted as , and normalized SDR values in [0, 1), the FLUT may be adjusted in the SDR luminance ranges [0, ]and [, 1). Without limitation, examples of low-intensity and high-intensity luminance regions may be normalized luminance values in [0, 0.5) and [0.5, 1) respectively. Example methods to generate merging points are provided in Ref. [5] and in later sections of this specification.

There are two major types of scenes in the proposed architecture, namely, primary scenes and secondary scenes. Pass-2 processes all scenes to produce the same set of metadata parameters for every frame in that scene. In the forward phase of Pass-2, reshaping parameters are computed from all the frames in the statistics collection window of that scene.

• • For a primary scene, the statistics collection window includes all the frames in the primary scene. Frames outside the primary scenes are not referenced.
  • On the contrary, for a secondary scene, the statistics collection window includes all the frames in that secondary scene plus some overlapping frames from the previous or next subscene. These extra frames are termed as overlapping frames.

As a rule, a primary scene will have no overlap with any neighboring scene, and secondary scenes are only allowed to have overlap with neighboring secondary scenes. In other words, the overlapping frames for a subscene can never come from a neighboring primary scene. The overlap parameter θ (see equation (13)) is set by the user and the default value is 1. Backward phase in Pass-2 uses no such overlap for primary or secondary scenes.

As mentioned earlier, a colorist may introduce trims in the reference SDR (125) to meet the director's intent. Sometimes the trim results in clipping of the highlights and/or crushing of the low intensity values. Reconstructing HDR from a trim-affected SDR introduces undesirable artifacts in the reconstructed HDR domain. To reduce these artifacts, as discussed in Ref. [5] and later in this specification, a trim correction algorithm needs (i) a reference display mapping (DM) curve computed using minimum, maximum and mean (or average) HDR and SDR luma values in the content (ii) merging points in the low and/or high intensity regions

As used herein, the term “display mapping” (DM) curve denotes a function which maps pixel values in an image frame (in a first dynamic range) to pixel values on a target display (in a second, different, dynamic range). For example, and without limitation, as discussed in Ref. [8], given min, max, and average luminance values in a video frame and corresponding min, average, and maximum luminance values in a target display, one may determine a tone-mapping curve mapping the input HDR values to corresponding HDR or SDR values. In an embodiment, such a tone-mapping or display-mapping (DM) curve may be used to better determine the range of HDR values where the original forward reshaping function needs to be adjusted so that artifacts due to the trim pass are reduced.

For trim handling in secondary scenes, the minimum and maximum HDR and SDR luma values used in the reference DM curve are fixed globally for the entire video sequence. In other words, there is no need to compute the minimum and maximum HDR or SDR luma values in each secondary scene. Let us denote the luma mean HDR and SDR values by v _avg and s_avg respectively. Let the merging points in the low and high intensity SDR regions to be represented by s_l^m and s_h^m respectively. As an example, consider the scenario in FIG. 3D, where segment P is to be processed by two nodes. After the insertion of the two synchronization scene cuts, node N processes subscenes A, B, and C, while node N+1 processes subscenes D, E, and F. Nodes N and N+1 have access to the same set of frames in trim-parameter window (345) to be used for collecting trim parameters. For every frame index j in , a luma mean value v_avg^Y,j and two SDR merging points s_l^m,j, s_h^m,j (later, to be converted to scene-based HDR merging points v_l^m, v_h^m) are computed. As shown below, the mean values are averaged to get the output trim parameter set for .

$\begin{array}{cc}{v}_{\mathrm{avg}}^{ℒ}=\frac{1}{2B}{\sum }_{j\in ℒ}{v}_{\mathrm{avg}}^{Y,j},& \left(22\right)\end{array}$

where B denotes the number of bumper frames. For the trim-parameter window , the low-intensity and high-intensity merging points are derived as:

=max({s_l^m,j|j∈}).

=min({s_h^m,j|j∈})  (23)

The trim parameter set, and is computed on all frames in , so they are the same for both the nodes. These values are copied to every subscene of the parent scene on both the nodes (e.g., scenes A, B, E, and F). With the same set of values used by all the subscenes, temporal stability is maintained.

In an embodiment, the trim-parameter window may be defined as follows:

• • when sharing a parent scene with a subsequent neighbor node: the frames from frame C_sr=C_L−1−B, including frame C_sr, until the end of the bumper frames at the end of the node segment, where B denotes the number of bumper frames and C_L−1 denotes the last frame of the segment;
  • when sharing a parent scene with a prior neighbor node: the frames from the beginning of the bumper frames before the start of the node segment, until frame C_sl=C₀+B, but without including C_sl, where C₀ denotes the first frame of the segment.

For longer parent scenes that span multiple nodes, the trim parameters are either copied or interpolated. FIG. 3E shows an example of a long parent scene processed by three computing nodes. Let and be the set of trim parameters in 1 (350). Every subscene in node N−1 will get a copy of these parameters. Let trim parameters in 2 (355) be and , then every subscene in node N+1 will use the 2 set of values for trim correction.

For node N, the trim value for any subscene is computed as follows. Suppose ρ represent any of the three parameters in the trim set. Let a₀ denote the first frame (351) after the end of the start trim-parameter window 350 in node N. Frame a₀ and all frames before a₀ will use trim-set 1. Let a_n (353) denote the first frame of the end trim-window 355 in node N. Frame a_n and all frames starting at a_n will use trim-set 2. For all other frames between these two trim-parameter windows (i.e., frames with frame indices j∈[a_o, a_n−1]), for example, frames in the subscene A, the trim parameters will be computed by linear interpolation from the trim set 1 and 2 depending on their distance from the anchor frames a₀ and a_n. For example, in an embodiment, for the j-th frame, an ρ^j trim parameter may be computed as:

$\begin{array}{cc}{\rho }^j=\frac{{\rho }^{ℒ2}-{\rho }^{ℒ1}}{a_n-a_0}\left(j-a_0\right)+{\rho }^{ℒ1},& \left(24\right)\end{array}$

where and denote the corresponding 1 and 2 trim parameters (see equations (22) and (23)). Then, in node N, for any subscene A within the two trim parameter windows, that is within frames a_s>a₀ and a_e−1<a_n, the scene-related trim parameters in subscene A are computed using an average of the frame trim parameters within that subscene, that is:

$\begin{array}{cc}{\rho }^A=\frac{1}{a_e-a_s}{\sum }_{j=a_s}^{a_e-1}{\rho }^j.& \left(25\right)\end{array}$

The same formula applies for any parameter in the trim set. Interpolation of trim parameters across subscenes ensures a gradual transition from trim parameters in 1 (i.e. ) and 2 (i.e. ) computed at either ends of node N. Without the interpolation, different trim parameters at the edges of the segment will cause drastic changes in the forward reshaping parameters from one subscene to the next, that may cause flashing or sudden brightness change.

While equation (24) applies simple linear interpolation, alternative interpolation methods known in the art, such as second order, bilinear, polynomial, spline, and the like, may also be used.

FIG. 4B depicts a summary of an example process (450) to generate updated scene-based correction trim-pass parameters according to an embodiment. As depicted in FIG. 4B, the process starts in step 455 by accessing the original trim-pass correction parameters (e.g., mean SDR and HDR values for each frame and merging points). If a primary scene is executed in-full in the current node, no action is needed, and the process proceeds to forward reshaping (225) where a forward reshaping function may be trim-pass-corrected as will be explained later (see process 470). If a node detects that a scene is shared with a single neighbor node (e.g., a prior or a subsequent node), then, in step 457, it uses its synchronization scene cut to define a trim-pass parameter window (e.g. 345) and generate a new set, , of trim-pass correction parameters (e.g., see equations (22) and (23)). The new set is to be used for all secondary scenes in the current node (step 459) to generate scene-based forward reshaping functions (225).

If a node detects that one or more scenes are shared with both of is neighbor nodes (e.g., a prior and a subsequent node), then, in step 462, it uses the two synchronization scene cuts to define two trim-pass parameter windows (e.g., 350 and 355), and determine two new sets (1 and 2) of updated trim-pass correction parameters (e.g., see equations (22) and (23)). Using 1 and 2, in step 464, the node will generate updated trim-pass parameters for all frames between the two trim-pass parameter windows by interpolating values between 1 and 2 according to their distance. Next (step 466), the following assignment is made:

• • For all secondary scenes before the first sync frame use the 1 set of trim correction parameters
  • For all secondary scenes after the second sync frame use the 2 set of trim correction parameters
  • For each secondary scene between the first and the second sync frames generate a new scene-based trim correction set (e.g., ρ^A) by averaging the corresponding interpolated correction trim parameters for all frames within that secondary scene (e.g., see equation (25))

Pass-2: Scene-Based Encoding

As depicted in FIG. 2, every node is allocated a segment which is encoded into the bitstream. Pass-1, described earlier, generates a list of primary and secondary scenes that minimize the bitrate of reshaping-related metadata while maintaining temporal consistency across neighboring nodes. Pass-2 uses this list to encode the scenes in the segment and generate a reshaped SDR coded bitstream. As depicted in FIG. 3A, bumper frames (309) are not encoded, but they are used to collect statistics in a forward reshaping pass to maintain the temporal stability of the secondary scenes.

As depicted in FIG. 2, Pass-2 includes steps 225, 230, and 235. An alternative view of the same processing pipeline, at the scene or subscene level, is also depicted in FIG. 5. For forward reshaping, primary and secondary scenes are handled in a similar manner except for one main difference: primary scenes have no overlap in forward reshaping, but secondary scenes have some overlap with neighboring subscenes. For backward reshaping, the process is exactly the same for both primary and secondary scenes. There is no overlap in the backward reshaping phase. A composite bitstream (240) consisting of reshaping metadata and a compressed base layer is generated as the output. Details of each block are described next.

Given a segment to scenes list (222), FIG. 5 depicts an example architecture for scene-based encoding on each node in the cloud. Without limitation, block 225 in FIG. 2 may be split as shown, using blocks 505 and block 132. Let the starting frame index for the k-th scene be denoted a S_k. Thus, given a scene k, a node needs to process frames S_k, S_k+1, S_k+2, . . . , and S_k+1−1. Reference HDR frames (504) and corresponding SDR frames (502) for a scene may be stored in corresponding SDR and HDR scene buffers (not shown). As discussed, bumper frames are used only for generating statistical data for the secondary scenes and are ignored when processing primary scenes.

From FIG. 5, in step 505, the input SDR and HDR frames are used to generate a scene-based forward reshaping function. The parameters of such a function will be used for the whole scene (vs. being updated on a frame-by-frame basis), thus reducing the overhead for metadata 152. Next, in step 132, forward reshaping is applied to the HDR scene (504) to generate a reshaped base layer 229, to be coded by compression unit (230) to generate the coded bitstream 144. Finally, in step 235, the reshaped SDR data 229 and the original HDR data (504) are used to generate parameters 152 for a backward reshaping function to be transmitted to a downstream decoder together. These steps will be described in more detail next. Without limitation, steps are described in the context of what is referred to as a three-dimensional mapping table (3DMT) representation where, to simplify operations, each frame is represented as a three dimensional mapping table, where each color component (e.g., Y, Cb, or Cr) is subdivided into “bins,” and instead of using explicit pixel values to represent an image one uses the pixel averages within each bin. Details of the 3DMT formulation can be found in Ref. [3].

Scene-based generation of a forward reshaping function (505) consists of two levels of operation. First, statistics are collected for each frame. For example, for luma, one computes the histograms for both SDR (h_j^s(b)) and HDR (h_j^v(b)) frames and stores them in the frame buffer for the j-th frame, where b is the bin index. After generating the 3DMT representation for each frame, one generates an “a/B” matrix representation denoted as:

B _j ^F=(S_j^F)^TS_j^F,

a _j ^F,ch=(S_j^F)^Tv_j^F,ch,  (26)

where ch refers to a chroma channel (e.g., Cb or Cr), (S_j^F)^T denotes a transpose matrix based on the reference HDR scene data and a parametric model of the forward reshaping function, and v_j^F,ch denotes a vector based on the SDR scene data and the parametric model of the forward reshaping function.

Given the statistics of each frame within the current scene, one can apply a scene-level algorithm to compute the optimal forward reshaping coefficients. For example, for luma, one can generate scene-based histograms for SDR (h^s(b)) and HDR data (h^v(b)) by summing or averaging the frame-based histograms. For example, in an embodiment,

h^s(b)=Σ_j=Sk^Sk+1−1h_j^s(b),

h^v(b)=Σ_j=Sk^Sk+1−1h_j^v(b).  (27)

Having both scene-level histograms, in an embodiment, as an example and without limitation, one can apply cumulative density function (CDF) matching (Ref. [4-5]) to generate the forward mapping function (FLUT) from HDR to SDR, e.g.,

{tilde over (T)}^F=CDF_MATCHING(h^v(b),h^s(b)).  (28)

However, one can apply any known reshaping method to generate the original forward mapping function. If a node detects the need to apply trim-pass correction, then this forward reshaping function will need to be corrected using a reference DM curve and the trim-pass correction methods discussed next.

While the frame-based FLUTs are typically designed based on a frame's minimum, average, and maximum HDR and SDR luminance values, a reference DM curve is constructed a bit differently. For primary scenes, a reference DM curve uses the minimum and maximum HDR and SDR luminance values for all the frames in the scene. For secondary scenes, a reference DM curve uses the global minimum and maximum values. These global-minimum and global-maximum HDR and SDR values may define a broader dynamic range than the corresponding frame-based values. In an embodiment, these values may be preselected to represent the allowable legal range of values during data transmission. For example, in an embodiment, they may be defined based on the SMPTE range, thus, for 16-bit HDR data and 8-bit SDR data, a DM curve will map HDR codewords in [4096, 60160] to SDR codewords in [16, 235]. Alternatively, one may apply the full range of possible values, for example, by mapping the full 16-bit HDR range [0, 65535] to the full 10-bit SDR range [0, 1023].

For primary scenes, clipping-related distortion in the scene is measured using the SDR and HDR scene histograms. For example, if values of the normalized SDR histograms (e.g., see equation (55)) are lower than a certain threshold (e.g., 0.20 for 8-bit SDR data and 0.05 for 10-bit SDR data) and/or if there are no “variance peaks” (see equations (56)-(58)), then no trim correction is required, i.e. T^F={tilde over (T)}^F. Otherwise, if trim-related clipping is detected, then the trim-correction process is enabled. A range-restricted DM curve {tilde over (T)}_rr^DM is used frequently in trim correction and it is built as follows. Initially, a reference DM curve {tilde over (T)}^DM is constructed using, for example, the minimum, average and maximum HDR luma values in the scene, i.e., v_min^Y, v_avg^Y, and v_max^Y:

{tilde over (T)}^DM=DM_TONE_MAPPING(v_min^Y, v_avg^Y, v_max^Y)  (29)

where DM_TONE_MAPPING() refers to a tone-mapping representation mapping the HDR luminance range [v_min^Y, v_max^Y] to the corresponding SDR range [s_min^Y, s_max^Y] (e.g., see Ref. [8])). Differential FLUT {tilde over (t)}^F and DM curve {tilde over (t)}^DM values are derived from their original counterparts as:

{tilde over (t)}^F(i)={tilde over (T)}^F(i)−{tilde over (T)}^F(i−1) for i∈[v_min^Y+1, v_max^Y],

{tilde over (t)}^DM(i)={tilde over (T)}^DM(i)−{tilde over (T)}^DM(i−1) for i∈[v_min^Y+1, v_max^Y],  (30)

where i denotes an index in the FLUT array. Values outside the index range [v_min^Y+1, v_max^Y] are set to zero. The differential DM curve is then bounded (or scaled) to the SDR range of the FLUT curve:

$\begin{array}{cc}{\tilde{t}}_{\mathrm{rr}}^{\mathrm{DM}}\left(i\right)={\tilde{t}}^{\mathrm{DM}}\left(i\right)\times \frac{{\sum }_i{\tilde{t}}^F\left(i\right)}{{\sum }_i{\tilde{t}}^{\mathrm{DM}}\left(i\right)}.& \left(31\right)\end{array}$

This bounded differential DM curve is denoted as the range-restricted differential DM curve. A cumulative sum of the elements of this range-restricted differential DM curve (corresponding to a simple integration operation) gives the range restricted DM curve {tilde over (T)}_rr^DM:

{tilde over (T)}_rr^DM(i)={tilde over (t)}_rr^DM(i)+{tilde over (T)}_rr^DM(i−1), for i>0,

and

{tilde over (T)}_rr^DM(i)={tilde over (T)}^F(i) for i=0  (32)

For low-intensity trim correction, the original FLUT and DM curves are merged from v_min^Y in until the merging point. On the other hand, in high intensity trim correction, the curves are combined from the high-merging point forwards. A merging point can be defined as a FLUT index value that marks the beginning/end of the FLUT and DM merging process depending on the area of trim correction. There can be one merging point in a low-intensity region and/or one for the high-intensity region.

As described in more detail in in Ref. [5] and in a later section (see “Computing Merging Points for Trim-pass Correction”), one begins with an initial estimate of the merging points with respect to the SDR codeword range. Let these first estimates be s_l^m,1 and s_h^m,1 for the low- and high-intensity regions respectively. The FLUT curve {tilde over (T)}^F helps procure the equivalent merging points in the HDR range, namely, v_l^m,1 and v_h^m,1, by reverse mapping SDR codewords to HDR codewords:

v _l ^m,1=max{i|{tilde over (T)}^F(i)=s_l^m,1},

v _h ^m,1=min{i|{tilde over (T)}^F(i)=s_h^m,1}  (33)

These first estimates are refined using the range restricted DM curve to get a second, more accurate, estimate of the merging point in the SDR range, s_l^m,2 and s_h^m,2, where

s _l ^m,2={tilde over (T)}_rr^DM(v_l^m,1),

s _h ^m,2={tilde over (T)}_rr^DM(v_h^m,1).tm (34)

These second estimates in the SDR domain are then reversed mapped to equivalent HDR merging point values. These second HDR merging points, v_l^m,2 and v_h^m,2, are used for the final merging.

v _l ^m,2=max{i|{tilde over (T)}^F(i)=s_l^m,2},

v _h ^m,2=min{i|{tilde over (T)}^F(i)=s_h^m,2}  (35)

During merging, the differential FLUT curve is replaced by the range-restricted differential DM curve in the low or high intensity regions to avoid clipping. In order to keep the SDR mapped range of the resulting hybrid curve the same as the range in the original FLUT curve, the range-restricted differential DM curve should be scaled appropriately, as shown.

• • Low intensity trim correction:

$\begin{array}{cc}{\tilde{t}}_{\mathrm{corr}}^F\left(i\right)={\tilde{t}}_{\mathrm{scaled}}^{\mathrm{DM}}\left(i\right)\mathrm{for}i\in \left[v_{\min }^Y+1,v_l^{m,2}\right],\mathrm{where}{\tilde{t}}_{\mathrm{scaled}}^{\mathrm{DM}}\left(i\right)={\tilde{t}}_{\mathrm{rr}}^{\mathrm{DM}}\left(i\right)\times \frac{{\sum }_{i=v_{\min }^Y}^{v_l^{m,2}}+{\tilde{t}}^F\left(i\right)}{{\sum }_{i=v_{\min }^Y}^{v_l^{m,2}}{\tilde{t}}_{\mathrm{rr}}^{\mathrm{DM}}\left(i\right)}.& \left(36\right)\end{array}$

• • High intensity trim correction:

$\begin{array}{cc}{\tilde{t}}_{\mathrm{corr}}^F\left(i\right)={\tilde{t}}_{\mathrm{scaled}}^{\mathrm{DM}}\left(i\right)\mathrm{for}i\in \left[v_h^{m,2}+1,v_{\max }^Y\right],\mathrm{where},{\tilde{t}}_{\mathrm{scaled}}^{\mathrm{DM}}\left(i\right)={\tilde{t}}_{\mathrm{rr}}^{\mathrm{DM}}\left(i\right)\times \frac{{\sum }_{i=v_h^{m,2}+1}^{v_{\max }^Y}{\tilde{t}}^F\left(i\right)}{{\sum }_{i=v_h^{m,2}+1}^{v_{\max }^Y}{\tilde{t}}_{\mathrm{rr}}^{\mathrm{DM}}\left(i\right)}.& \left(37\right)\end{array}$

• • For the rest of the index locations:

{tilde over (t)}_corr^F(i)={tilde over (t)}^F(i).  (38)

Finally, a cumulative sum of the updated or trim-corrected differential FLUT curve, {tilde over (t)}_corr^F, provides the final, trim-corrected, luma FLUT T^F.

T ^F(i)={tilde over (t)}_corr^F(i)+T^F(i−1) for i>0 and

T ^F(i)={tilde over (T)}^F(i) for i=0.   (39)

An example of the process is illustrated in FIG. 6A. Plots 610 and 605 provide respectively examples of an original FLUT and a reference DM curve. Plot 615 provides an example of a scaled and range-restricted DM curve. Let HDR value 620 denote an example highlights-merging point to be used for trim-pass correction. Then to generate a trim-pass-corrected FLUT, after the merging point (620), the mapping function in the original FLUT (610) is replaced based on the corresponding part of the scaled DM curve. A similar approach can also be applied to replace the original FLUT below a merging points for the darks (not shown).

For secondary scenes, the process starts with the updated trim parameters generated earlier (see process 450). As discussed earlier (see process 450 in FIG. 4B), a new trim correction parameter set is determined for every subscene using only the common frames in the trim-parameter windows in the neighboring nodes. This trim correction parameter set is read in from the auxiliary data (212) and is used during trim correction for that subscene.

Suppose the SDR merging points that are read in from the data block for the current subscene are now _l^m and s_h^m for the low- and high-intensity range respectively. The HDR equivalents, v_l^m and v_h^m, of these merging points can be determined using reverse mapping. Then, these merging points may be used to construct the hybrid FLUT and DM curve. Let

v _l ^m=max{i|{tilde over (T)}^F(i)=s_l^m},

v _h ^m=min{i|{tilde over (T)}^F(i)=s_h^m}  (33)

In an embodiment, a reference DM curve is generated based on global minimum and maximum possible HDR luma codeword values, denoted as HDR_min and HDR_max, (e.g., as those defined by the restricted SMPTE range) within the maximum possible range [0, 2^BH−1], where B_H is the bit-depth of the HDR codewords. However, the mean value v_arg^Y taken from the trim correction parameter set for that subscene. Thus,

T ^DM=DM_TONE_MAPPING(HDR_min,v_avg^Y, HDR_max).  (41)

where the HDR range [HDR_min, HDR_max] is mapped to [SDR_min,SDR_max] (e.g., the restricted SMPTE range, the full range [0, 2^Bs−1], or any other range within [0, 2^Bs−1], where B_S denotes the bitdepth of the SDR sequence).

For temporal stability, it is advisable to use a similar DM curve for every subscene. Choosing fixed min/max values allow the DM curve to be consistent throughout the sequence and it also covers the entire valid codeword range. These min/max values are configurable and should be set to lowest/highest possible codeword values in the sequence.

Next, the differential DM curve {tilde over (t)}^DM and FLUT curve {tilde over (t)}^F are derived as:

{tilde over (t)}^DM(i)={tilde over (T)}^DM(i)−{tilde over (T)}^DM(i−1) for i∈[HDR_min+1, HDR_max],

{tilde over (t)}^F(i)={tilde over (T)}^F(i)−{tilde over (T)}^F(i−1) for i∈[HDR_min+1, HDR_max]  (42)

For trim correction in the low intensity region, the DM curve is scaled similarly to the legacy method. In an embodiment, scaling is preferred in the low intensity region to avoid introducing any constant offset in the FLUT. In the high intensity region, the FLUT curve is replaced by the original DM curve, without any scaling.

• • Low intensity trim correction:

$\begin{array}{cc}{\tilde{t}}_{\mathrm{corr}}^F\left(i\right)={\tilde{t}}_{\mathrm{scaled}}^{\mathrm{DM}}\left(i\right)\mathrm{for}i\in \left[v_{\min }^Y+1,v_l^m\right],\mathrm{where},{\tilde{t}}_{\mathrm{scaled}}^{\mathrm{DM}}\left(i\right)={\tilde{t}}^{\mathrm{DM}}\left(i\right)\times \frac{{\sum }_{i=v_{\min }^Y+1}^{v_l^m}{\tilde{t}}^F\left(i\right)}{{\sum }_{i=v_{\min }^Y}^{v_l^m}{\tilde{t}}^{\mathrm{DM}}\left(i\right)}.& \left(43\right)\end{array}$

• • High intensity trim correction:

{tilde over (t)}_corr^F(i)={tilde over (t)}^DM(i) for i∈[v_h^m+1, v_max^Y].  (44)

• • For rest of the index locations:

{tilde over (t)}_corr^F(i)={tilde over (t)}^F(i).  (45)

Finally, a cumulative sum of the updated or trim-corrected differential FLUT curve, {tilde over (t)}_corr^F, yields the final luma FLUT T^F.

T ^F(i)={tilde over (t)}_corr^F(i)+T^F(i−1) for i>0 and

T ^F(i)={tilde over (T)}^F(i) for i=0.   (46)

An example, based on the curves in FIG. 6A and a high-intensity only merging point, is depicted in FIG. 6B. The reason for using the original DM curve rather than the scaled version of the DM curve is to avoid modifying the FLUT when there are minor changes in the SDR range and to maintain better temporal consistency between consecutive subscenes that may be processed by different nodes.

FIG. 4C provides an example process (470) of these steps according to an embodiment. As depicted in FIG. 4C, in step 472, the process starts by accessing an original FLUT to be trim-corrected, a set of original merging points to be used for trim-correction (e.g., see equation (23) or equation (25)), and scene-based minimum, average, and maximum HDR luminance values (e.g., v_min^Y, v_arg^Y, v_max^Y). Next, the node identifies whether a scene to be processed is primary or secondary. For a primary scene, if trim-pass correction is needed, in step 474, a reference DM curve is generated (see equation (29)). Using a set of differential FLUT and DM curves (step 476), in step 478, it generates a range-restricted differential DM curve and the corresponding range-restricted DM curve (see equations (30)-(32)), so that the mapped SDR range of the range-restricted DM curve is the same as that of the original FLUT. Given the range-restricted DM curve, in step 480, it generates a second, more accurate set of merging points (see equation (35)). Then, in step 482, using the new merging points, the original differential FLUT curve is edited with segments from the range-restricted differential DM curve that was generated in step 478 and scaled as needed to match the original SDR range of the FLUT curve (see equations (36)-(38)). Finally, in step 485, the trim-corrected FLUT is generated from the edited differential FLUT.

For a secondary scene, in step 490, a reference DM is generated using scene-based average luminance values, but global minimum and maximum HDR luminance values (e.g., HDR_min, v_avg^Y, HDr_max). As done for primary scenes, in step 492, differential FLUT and reference DM curves are constructed. Unlike what is done in primary scenes, the original merging points are not refined. Step 494 is similar to step 482; using the original merging points, the original differential FLUT curve is edited with segments from the differential DM curve that was generated in step 492, except that the low and high merging points are treated a bit differently. For a low-intensity merging point, the differential reference DM curve is scaled to match the SDR dynamic range (equation (43)). For a high-intensity merging point, the differential reference DM curve is used with no additional scaling (see equation (44)). Finally, in step 485, the trim-corrected FLUT is generated from the edited differential FLUT.

For chroma (e.g., ch=Cb or ch=Cr), one may again average over the a/B frame-based representations in equations (26) to generate a scene-based a/B matrix representation given by

$\begin{array}{cc}{B}^F=\frac{1}{S_{k+1}-S_k}{\sum }_{j=S_k}^{S_{k+1}-1}{B}_j^F,a^{F,\mathrm{ch}}=\frac{1}{S_{k+1}-S_k}{\sum }_{j=S_k}^{S_{k+1}-1}{a}_j^{F,\mathrm{ch}},& \left(47\right)\end{array}$

and generate parameters for a multiple-color, multiple- regression (MMR) model of a reshaping function as (Ref. [2-3])

m ^F,ch=(B^F)⁻¹a^F,ch.  (48)

Then, the reshaped SDR signal (229) can be generated as:

{circumflex over (v)}_j^F,ch=B^Fm^F,ch.  (49)

Generating the scene-based backward reshaping function (152) includes also both frame-level and scene-level operations. Since the luma mapping function is a single-channel predictor, one can simply revert the forward reshaping function to obtain the backward reshaping function. For chroma, one forms a 3DMT representation using the reshaped SDR data (229) and the original HDR data (504) and computes a new frame-based a/B representation as:

B _j ^B=(S_j^B)^TS_j^B,

a _j ^B,ch=(S_j^B)^Tv_j^B,ch.  (50)

At the scene-level, for luma, one may apply the histogram-weighted BLUT construction in Ref. [3] to generate the backward luma reshaping function. For chroma, one can again average the frame-based a/B representation to compute a scene-based a/B representation

$\begin{array}{cc}{B}^B=\frac{1}{S_{k+1}-S_k}{\sum }_{j=S_k}^{S_{k+1}-1}{B}_j^B,a^{B,\mathrm{ch}}=\frac{1}{S_{k+1}-S_k}{\sum }_{j=S_k}^{S_{k+1}-1}{a}_j^{B,\mathrm{ch}},& \left(51\right)\end{array}$

with an MMR model solution for the backward reshaping mapping function given by

m ^B,ch=(B^B)⁻¹a^B,ch.  (52)

Then, in a decoder, the reconstructed HDR signal (160) can be generated as:

{circumflex over (v)}_j^B,ch=B^Bm^B,ch.  (53)

Computing Merging Points for Trim-Pass Correction

Trim-related clipping may be present in low- or high-intensity regions. To fix these clipping artifacts, as discussed earlier, the trim-affected forward reshaping curve (FLUT) is merged with a trim-free display-mapping (DM) curve to build a hybrid curve that avoids clipping. Two important parameters for generating this hybrid curve are the merging points for the FLUT and DM curves. Initially, these merging points (e.g., s_l^m and s_h^m), one for the low-intensity region and one for the high-intensity region, are calculated in the SDR codeword range. Then, equivalent merging points (e.g., v_l^m and v_h^m) are derived in the HDR range using one or more transformations, as discussed earlier and in Ref. [5]. These HDR merging points are eventually the ones used for constructing the hybrid (trim-corrected) FLUT. This section presents an example method for computing the merging points s_l^m and s_h^m in the SDR codeword range.

The SDR and HDR histograms and the original FLUT needed for trim pass correction may be frame-based, scene-based or window-based (e.g., computed within a narrow window of frames), but the merging points are evaluated in the same way. Let {tilde over (T)}^F denote the FLUT curve affected by trim-related clipping, and let h^s, h^v be the SDR and HDR codeword histograms respectively.

The first step is to detect peaks in the SDR luma histogram h^s using a moving average filtered version of the SDR histogram, denoted as h_sm^s. In an embodiment, element-wise differences between the original and the smoothened SDR histogram may be used to estimate the locations of these peaks.

h _pk ^s(b)=max(0, h^s(b)−h_sm^s(b))∀b∈[0,2^BS−1],  (54)

where B_s denotes the bit-depth of the SDR codewords. Histogram h_pk^s is normalized by dividing each element by the sum of all the entries in the histogram to generate a normalized histogram

$\begin{array}{cc}{\stackrel{\_}{h}}_{\mathrm{pk}}^s\left(b\right)=\frac{h_{\mathrm{pk}}^s\left(b\right)}{{\sum }_{\forall i}{h}_{\mathrm{pk}}^s\left(i\right)}\forall b\in \left[0,2^{B_s}-1\right].& \left(55\right)\end{array}$

Each SDR bin b that has a normalized value h_pk^s(b) higher than a threshold (e.g., 0.20 for 8-bit SDR or 0.05 for 10-bit SDR) is a valid peak. Location of that peak τ_loc is b, i.e. τ_loc=b and its severity τ_sev is h_pk^s(b), i.e. τ_sev=h_pk^s(b). These peaks signify that multiple pixels are mapped to a single SDR codeword, which is an indication of clipping. The location of these peaks helps identify the regions of potential clipping, say, in the low/high intensity regions. For example, in an embodiment, the SDR range is divided into two, and any peak below the half (2^BS−1) is considered a low-intensity peak, otherwise it is considered a high-intensity peak.

Due to HDR-to-SDR mapping, as the HDR codeword bit-depth is higher than the SDR bit-depth, multiple HDR codewords are mapped to a single SDR codeword. Clipping exacerbates this many-to-one mapping, as many more codewords get mapped to the same SDR codeword in the clipped regions. For any SDR codeword c, one can find the HDR codewords that are assigned to it and compute the variance of these codewords. Let

v _min ^c=min{i|{tilde over (T)}^F(i)=c},

v _max ^c=max{i|{tilde over (T)}^F(i)=c}.  (56)

where c denotes any SDR codeword value from the entire SDR codeword range. Then, the cross-domain variance a for c can be calculated as

$\begin{array}{cc}{\sigma }_c^2={\sum }_{b=v_{\min }^c}^{v_{\max }^c}{h}^v\left(b\right)\times {\left(b-{\mu }_c\right)}^2,\mathrm{where}{\mu }_c=\frac{{\sum }_{b=v_{\min }^c}^{v_{\max }^c}b\times h^v\left(b\right)}{{\sum }_{b=v_{\min }^c}^{v_{\max }^c}{h}^v\left(b\right)}.& \left(57\right)\end{array}$

The cross domain variance σ_c² is evaluated at every SDR codeword value and normalized by dividing with the sum of all the SDR codeword variances to get σ_c²

$\begin{array}{cc}{\stackrel{\_}{\sigma }}_c^2=\frac{{\sigma }_c^2}{{\sum }_{\forall c}{\sigma }_c^2},\forall c\in \left[0,2^{B_s}-1\right].& \left(58\right)\end{array}$

Normalized variance values σ_c² that are higher than a threshold are designated as variance peaks with τ_loc=c and τ_sev=σ_c².

The peaks in the SDR histogram and the cross domain variance array are collected in a set and separated out into low intensity and high intensity groups. Suppose j represents any peak in the set and τ_loc^j<2^BS−1, then it is a low intensity peak or else the peak belongs to high intensity region. For low intensity region, the merging point s_l^m,j is estimated for jth peak as

s _l ^m,j=τ_loc^j+η×τ_sev^j,  (59)

where η denotes a constant that is determined heuristically (e.g., η=25 for 8-bit SDR and η=100 for 10-bit SDR).

The low-merging point in the SDR range, s_l^m, is then the maximum of all these merging points bounded by 2^BS−1.

s _l ^m=min (max{s_l^m,j}, 2^BS−1).  (60)

For the high intensity region, the high-merging point, s_l^m, in SDR range is derived as follows:

s _h ^m,j=τ_loc^j−η×τ_sev^j,  (61)

s _h ^m=max (min[s_h^m,j],2^BS−1).  (62)

These SDR merging points are later used to find the final HDR merging points using one or more transformations.

REFERENCES

Each of these references is incorporated by reference in its entirety.

1. H. Kadu et al., “Coding of high-dynamic range video using segment-based reshaping,” U.S. Pat. No. 10,575,028.

2. G-M. Su et al., “Multiple color channel multiple regression predictor,” U.S. Pat. No.8,811,490.

3. Q. Song et al., PCT Patent Application Ser. No. PCT/US2019/031620, “High-fidelity full reference and high-efficiency reduced reference encoding in end-to-end single-layer backward compatible encoding pipeline,” filed on May 9, 2019, published as WO 2019/217751.

4. B. Wen et al., “ Inverse luma/chroma mappings with histogram transfer and approximation,” U.S. Pat. No. 10,264,287.

5. H. Kadu and G-M. Su, “Reshaping curve optimization in HDR coding,” U.S. Pat. No. 10,397,576.

6. G-M. Su et al., “Workload allocation and processing in cloud-based coding of HDR video,” U.S. Provisional Patent Application, Ser. No. 63/049,673, filed on Jul. 9, 2020, also filed on Jul. 9, 2021, as PCT/US2021/040967.

7. H. Kadu et al., “Recursive segment to scene segmentation for cloud-based coding of HDR video,” U.S. Provisional Patent Application, Ser. No. 63/080255, filed on Sep. 18, 2020.

8. A. Ballestad and A. Kostin, “Method and apparatus for image data transformation,” U.S. Pat. No. 8,593,480,

EXAMPLE COMPUTER SYSTEM IMPLEMENTATION

Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and/or apparatus that includes one or more of such systems, devices or components. The computer and/or IC may perform, control or execute instructions relating to trim-pass correction in cloud-based video coding of HDR video, such as those described herein. The computer and/or IC may compute, any of a variety of parameters or values that relate to trim-pass correction and node-based processing in cloud- based video coding of HDR video as described herein. The image and video dynamic range extension embodiments may be implemented in hardware, software, firmware and various combinations thereof.

Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display, an encoder, a set top box, a transcoder or the like may implement methods for trim-pass correction and node-based processing in cloud-based video coding of HDR video as described above by executing software instructions in a program memory accessible to the processors. The invention may also be provided in the form of a program product. The program product may comprise any non-transitory and tangible medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of non-transitory and tangible forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. he computer-readable signals on the program product may optionally be compressed or encrypted.

Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a “means”) should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.

EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

Example embodiments that relate to trim-pass correction and node-based processing in cloud-based video coding of HDR video are thus described. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and what is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Various aspects of the present invention may be appreciated from the following enumerated example embodiments (EEEs):

EEE1. A method for trim-pass correction for a video segment to be encoded in a computing node, the method comprising:

receiving in a current computing node a first video sequence comprising video frames in a high dynamic range;

generating (455) for each video frame in the first video sequence a first forward reshaping function and frame-based trim-pass correction parameters, wherein a forward reshaping function maps frame-pixels from the high dynamic range to a second dynamic range lower than the high dynamic range;

for a subscene in the first video sequence that is part of a parent scene to be processed by the current computing node and one neighbor computing node:

• • determining (457) a trim-pass parameter window comprising frames in the first video sequence and bumper frames, wherein the bumper frames comprise video frames to be processed by the neighbor computing node;
  • computing (459) scene-based trim-pass correction parameters based on the frame-based trim-pass correction parameters of the frames in the trim-pass parameter window;
  • generating a scene-based forward reshaping function based on the first forward reshaping functions of the frames in the subscene; and
  • generating for the subscene an output forward reshaping function based on the scene-based trim-pass correction parameters and the scene-based forward reshaping function.EEE2. The method of EEE 1, further comprising, for the subscene in the first video sequence:

applying the output forward reshaping function to the frames in the subscene to generate an output video subscene in the second dynamic range; and

compressing the output video subscene to generate a coded subscene in the second dynamic range.

EEE3. The method of EEE 1 or EEE 2, wherein for a high-dynamic range (HDR) frame in the first video sequence, frame-based trim-pass correction parameters comprise one or more of:

an average luminance value of the HDR frame,

a first merging point in a low-luminance region of a standard-dynamic range (SDR) frame representing the same scene as the HDR frame but at the second dynamic range, and

a second merging point in a high-luminance region of the SDR frame.

EEE4. The method of EEE 3, wherein computing the scene-based trim-pass correction parameters comprises computing:

$v_{\mathrm{avg}}^{ℒ}=\frac{1}{2B}{\sum }_{j\in ℒ}{v}_{\mathrm{avg}}^{Y,j},s_l^{m,ℒ}=\max \left(\left\{s_l^{m,j}❘j\in ℒ\right\}\right),s_h^{m,ℒ}=\min \left(\left\{s_h^{m,j}❘j\in ℒ\right\}\right),$

where B denotes a number of bumper frames, denotes a set of frames in the trim-pass parameter window, v_avg^j denotes an average luminance value of the j-th HDR frame, s_l^m,j denotes a first merging point of the j-th HDR frame, s_h^m,j denotes a second merging point of the j-th HDR frame, and and denote the scene-based trim-pass correction parameters.EEE5. The method of EEE 4, further comprising assigning the scene-based trim-pass correction parameters to every subscene to be processed in the current computing node.EEE6. The method of any of EEEs 3-5, wherein generating for an HDR frame in the subscene an output forward reshaping function comprises:

generating a reference tone-mapping function mapping pixel values from the high dynamic range to pixel values in the second dynamic range based at least on an average luminance value in the subscene and global minimum and maximum HDR and SDR luminance values; and

generating the output forward reshaping function by:

• • computing a high-intensity merging point based on the second merging point; and

starting from the high-intensity merging point, replacing the scene-based forward reshaping function based on a corresponding section in the reference tone-mapping function.

EEE7. The method of EEE 6, wherein generating the output forward reshaping function comprises:

generating a differential tone-mapping function based on the reference tone-mapping function;

generating a differential scene-based forward reshaping function based on the scene-based forward reshaping function;

generating an updated differential scene-based forward reshaping function by replacing, starting from the high-intensity merging point, the differential scene-based forward reshaping function with a corresponding part of the differential tone-mapping function; and

generating the output forward reshaping function by integrating the updated differential scene-based forward reshaping function.

EEE8. The method of EEE 6 or EEE 7, wherein given an average HDR luminance value and an average SDR luminance value in the subscene, the reference tone-mapping function maps:

the average HDR luminance value to the average SDR luminance value;

a maximum global luminance in the high dynamic range to a maximum global luminance value in the second dynamic range; and

a minimum global luminance value in the high dynamic range to a minimum global luminance value in the second dynamic range.

EEE9. The method of any of EEEs 1-8, wherein for a subscene in the first video sequence that is part of a parent scene to be processed by the current computing node and one neighbor computing node:

if the neighbor computing node is prior to the current computing node, then the trim-pass parameter window comprises all (B) bumper frames before the first frame of the first video sequence (C₀), up to, but not including, frame C_s=C₀+B; and

if the neighbor computing node is subsequent to the current computing node, then

• • the trim-pass parameter window comprises frame C_s=C_L−1−B and all subsequent frames, including all bumper frames after the last frame of the first video sequence.EEE10. The method of any of EEEs 1-9, wherein for a subscene in the first video sequence that is part of a parent scene to be processed by the current computing node, a neighbor prior node, and a neighbor subsequent node, further comprising:

determining (462) a first trim-pass parameter window comprising frames in the first video sequence and bumper frames shared with the neighbor prior node;

determining a second trim-pass parameter window comprising frames in the first video sequence and bumper frames shared with the neighbor subsequent node;

computing a first set of scene-based trim-pass correction parameters based on the frame-based trim-pass correction parameters of the frames in the first trim-pass parameter window;

computing a second set of scene-based trim-pass correction parameters based on the frame-based trim-pass correction parameters of the frames in the second trim-pass parameter window;

computing interpolated frame-based trim-pass correction parameters based on the first set and second set of the scene-based trim-pass correction parameters; and

• • if the subscene is within the first trim-pass parameter window, then assigning to the subscene the first set of scene-based trim-pass correction parameters;
  • if the subscene is within the second trim-pass parameter window, then assigning to the subscene the second set of scene-based trim-pass correction parameters; and
  • if the subscene is between the first and second trim-pass parameter windows, then generating scene-based trim-pass correction parameters based on the interpolated frame-based trim-pass correction parameters for frames within the subscene.EEE11. The method of EEE 10, wherein computing the interpolated frame-based trim-pass correction parameters comprises computing:

${\rho }^x=\frac{{\rho }^{ℒ2}-{\rho }^{ℒ1}}{a_n-a_0}\left(x-a_0\right)+{\rho }^{ℒ1}$

where denotes a trim-pass correction parameter in the first set of scene-based trim-pass correction parameters, denotes a trim-pass correction parameter in the second set of scene-based trim-pass correction parameters, ρ^x denotes an interpolated frame-based trim-pass correction parameter for the x-th frame between a₀, the last frame of the first trim-pass parameter window, and a_n, the first frame of the second trim-pass parameter window.EEE12. The method of EEE 11, wherein generating a scene-based trim-pass correction parameter (ρ^A) based on the interpolated scene-based trim-pass correction parameters comprises computing

${\rho }^A=\frac{1}{a_e-a_s}{\sum }_{x=a_s}^{a_e-1}{\rho }^x$

wherein a_s denotes the starting frame of the subscene and a_e denotes the starting frame of the next subscene.EEE13. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for executing with one or more processors a method in accordance with any one of the EEEs 1-12.EEE14. An apparatus comprising a processor and configured to perform any one of the methods recited in EEEs 1-12.

Claims

1. A method for trim-pass correction for a video segment to be encoded in a computing node, the method comprising: receiving in a current computing node a first video sequence comprising video frames in a high dynamic range;generating (455) for each video frame in the first video sequence a first forward reshaping function and frame-based trim-pass correction parameters, wherein a forward reshaping function maps frame-pixels from the high dynamic range to a second dynamic range lower than the high dynamic range;for a subscene in the first video sequence that is part of a parent scene to be processed by the current computing node and one neighbor computing node: determining (457) a trim-pass parameter window comprising frames in the first video sequence and bumper frames, wherein the bumper frames comprise video frames to be processed by the neighbor computing node;computing (459) scene-based trim-pass correction parameters based on the frame-based trim-pass correction parameters of the frames in the trim-pass parameter window;generating a scene-based forward reshaping function based on the first forward reshaping functions of the frames in the subscene; andgenerating for the subscene an output forward reshaping function based on the scene-based trim-pass correction parameters and the scene-based forward reshaping function.

2. A method for trim-pass correction for a video segment to be encoded in a computing node, the method comprising: receiving in a current computing node a first video sequence being part of the video segment, the first video sequence comprising video frames in a high dynamic range;generating (455) for each video frame in the first video sequence a first forward reshaping function and frame-based trim-pass correction parameters, wherein a forward reshaping function maps frame-pixels from the high dynamic range to a second dynamic range lower than the high dynamic range, wherein trim-pass correction parameters denote parameters to modify given trim-pass parameters relating to one or more of trimming, rotating, cropping, flipping, adjusting the brightness, color or saturation of the first video sequence, wherein frame-based trim-pass correction parameters are generated on a per frame basis based on the respective frame for which they are generated;for a subscene in the first video sequence that is part of a parent scene, wherein the parent scene is to be processed by the current computing node and one neighbor computing node, wherein the subscene and the first video sequence are to be processed by the current computing node, performing the steps of: determining (457) a trim-pass parameter window comprising frames in the first video sequence and bumper frames, wherein the bumper frames comprise video frames to be processed by the neighbor computing node;computing (459) scene-based trim-pass correction parameters based on the frame-based trim-pass correction parameters of the frames in the trim-pass parameter window;generating a scene-based forward reshaping function based on the first forward reshaping functions of the frames in the subscene; andgenerating for the subscene an output forward reshaping function based on the scene-based trim-pass correction parameters and the scene-based forward reshaping function.

3. The method of claim 1, further comprising, for the subscene in the first video sequence: applying the output forward reshaping function to the frames in the subscene to generate an output video subscene in the second dynamic range; andcompressing the output video subscene to generate a coded subscene in the second dynamic range.

4. The method of claim 1, wherein for a high-dynamic range (HDR) frame in the first video sequence, frame-based trim-pass correction parameters comprise one or more of: an average luminance value of the HDR frame,a first merging point in a low-luminance region of a standard-dynamic range (SDR) frame representing the same scene as the HDR frame but at the second dynamic range, anda second merging point in a high-luminance region of the SDR frame.

5. The method of claim 4, wherein computing the scene-based trim-pass correction parameters comprises computing:

6. The method of claim 5, further comprising assigning the scene-based trim-pass correction parameters to every subscene to be processed in the current computing node.

7. The method of claim 4, wherein generating for an HDR frame in the subscene an output forward reshaping function comprises: generating a reference tone-mapping function mapping pixel values from the high dynamic range to pixel values in the second dynamic range based at least on an average luminance value in the subscene and global minimum and maximum HDR and SDR luminance values; andgenerating the output forward reshaping function by: computing a high-intensity merging point based on the second merging point; andstarting from the high-intensity merging point, replacing the scene-based forward reshaping function based on a corresponding section in the reference tone-mapping function.

8. The method of claim 7, wherein generating the output forward reshaping function comprises: generating a differential tone-mapping function based on the reference tone-mapping function;generating a differential scene-based forward reshaping function based on the scene-based forward reshaping function;generating an updated differential scene-based forward reshaping function by replacing, starting from the high-intensity merging point, the differential scene-based forward reshaping function with a corresponding part of the differential tone-mapping function; andgenerating the output forward reshaping function by integrating the updated differential scene-based forward reshaping function.

9. The method of claim 7, wherein given an average HDR luminance value and an average SDR luminance value in the subscene, the reference tone-mapping function maps: the average HDR luminance value to the average SDR luminance value;a maximum global luminance in the high dynamic range to a maximum global luminance value in the second dynamic range; anda minimum global luminance value in the high dynamic range to a minimum global luminance value in the second dynamic range.

10. The method of claim 1, wherein for a subscene in the first video sequence that is part of a parent scene to be processed by the current computing node and one neighbor computing node: if the neighbor computing node is prior to the current computing node, then the trim-pass parameter window comprises all (B) bumper frames before the first frame of the first video sequence (C0), up to, but not including, frame Cs=C0+B; andif the neighbor computing node is subsequent to the current computing node, then the trim-pass parameter window comprises frame Cs=CL−1−B and all subsequent frames, including all bumper frames after the last frame of the first video sequence, wherein L denotes the number of frames in the video segment.

11. The method of claim 1, wherein for a subscene in the first video sequence that is part of a parent scene to be processed by the current computing node, a neighbor prior node, and a neighbor subsequent node, further comprising: determining (462) a first trim-pass parameter window comprising frames in the first video sequence and bumper frames shared with the neighbor prior node;determining a second trim-pass parameter window comprising frames in the first video sequence and bumper frames shared with the neighbor subsequent node;computing a first set of scene-based trim-pass correction parameters based on the frame-based trim-pass correction parameters of the frames in the first trim-pass parameter window;computing a second set of scene-based trim-pass correction parameters based on the frame-based trim-pass correction parameters of the frames in the second trim-pass parameter window;computing interpolated frame-based trim-pass correction parameters based on the first set and second set of the scene-based trim-pass correction parameters; and if the subscene is within the first trim-pass parameter window, then assigning to the subscene the first set of scene-based trim-pass correction parameters;if the subscene is within the second trim-pass parameter window, then assigning to the subscene the second set of scene-based trim-pass correction parameters; andif the subscene is between the first and second trim-pass parameter windows, then generating scene-based trim-pass correction parameters based on the interpolated frame-based trim-pass correction parameters for frames within the subscene.

12. The method of claim 11, wherein computing the interpolated frame-based trim-pass correction parameters comprises computing:

13. The method of claim 12, wherein generating a scene-based trim-pass correction parameter (ρA) based on the interpolated scene-based trim-pass correction parameters comprises computing

14. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for executing with one or more processors a method in accordance with claim 1.

15. An apparatus comprising a processor and configured to perform any one of the methods recited in claim 1.