The present application relates to high-dynamic-range images, more particularly encoding of high-dynamic-range images at a standard dynamic range in a manner that enables subsequent reconstruction of the high-dynamic-range images from the standard-dynamic-range data.
Historically, most digital imagery, such as digital movies, has been created at a bit depth of 8 bits. This bit depth however presents a challenging tradeoff between contrast and resolution, as this bit depth is inferior to the human visual system. If a high dynamic range between brightest and darkest pixels is encoded at a bit depth of 8 bits, the discrete steps between adjacent pixel values on the brightness scale may be perceptible to a viewer. For example, a scene portion that has only a very gradual brightness variation may appear striped. If, on the other hand, the discrete steps between pixel values are set to a value that is imperceptible by a human, the dynamic range will be limited to less than the dynamic range of the human visual system. To overcome these issues and further enhance the viewing experience, digital imagery is now often created with a greater bit depth, such as 10 or 12 bits, to produce high-dynamic-range (HDR) content. To ensure compatibility with a variety of viewing devices, digital imagery is often provided at both the standard bit depth of 8 bits and a higher bit depth, such as 10 or 12 bits.
The relaying of digital imagery to a viewing system typically involves a codec. First, the imagery is encoded as data in a desired data format. The encoded data may include both image data and metadata. After communicating the data via a communication link to the viewing system, the data is decoded by the viewing system to reconstruct and display the imagery. Even when relaying HDR content, the communication link is often limited to the standard bit depth of 8 bits. Thus, some HDR codecs are configured to (a) in the encoding, compress the HDR imagery to a bit depth of 8 bits, and (b) in the decoding, decompress the compressed data to at least approximately reconstruct the HDR imagery. Furthermore, since not all viewing systems are capable of decoding and displaying HDR imagery, a dual-layer codec may be used to provide both standard-dynamic-range (SDR) imagery and HDR imagery. However, in some scenarios, a single-layer codec is preferred, wherein a single image data set is communicated to a decoder of a viewing system whereafter, depending on the properties of the viewing system, the image data set may be decoded to generate SDR imagery or HDR imagery.
In the single-layer codec, the encoding may utilize a forward reshaping function to generate reshaped SDR image data from input HDR image data, and the decoding may utilize a backward reshaping function to at least approximately reconstruct the input HDR image data from the reshaped SDR image data. The backward reshaping function is relayed from the encoder to the decoder as metadata accompanying the reshaped SDR image data. If the viewing system is not configured to process HDR image data, the decoder may forego using the backward reshaping function and instead directly use the reshaped SDR image data to generate SDR imagery. The quality of the reconstructed HDR image data is at least in part determined by the quality of the forward and backward reshaping functions.
Conventionally, for an input SDR and HDR image pair, the forward reshaping function is optimized to minimize the difference between the reshaped SDR image and the input SDR image, and the backward reshaping function is optimized to minimize the difference between the reconstructed HDR image and the input HDR image, with the only adjustable optimization parameters being properties of the forward and backward reshaping functions.
Disclosed herein are methods, software products, and devices that utilize an iterative optimization process to determine forward and backward reshaping functions in a single-layer high-dynamic-range codec. Conventionally, for an input standard-dynamic-range (SDR) and high-dynamic-range (HDR) image pair, the forward reshaping function is optimized to minimize the difference between the reshaped SDR image and the input SDR image, and the backward reshaping function is optimized to minimize the difference between the reconstructed HDR image and the input HDR image, with the only adjustable optimization parameters being properties of the forward and backward reshaping functions. The processing required to optimize the properties of these conventionally determined forward and backward reshaping function to a high quality may require a substantial amount of computation, which is unfeasible in some potential application scenarios. Instead of adjusting only properties of the reshaping functions, the present iterative optimization process iteratively evaluates an optimization error and feeds this error backwards as a correction to the input SDR image. This iterative process with backwards error propagation to the input SDR image simplifies the optimization of the reshaping functions, and may thus improve the performance of the reshaping functions and/or make the associated codec suitable for a wider range of applications.
In an embodiment, a method, for generating (a) a forward reshaping function for compressing an input high-dynamic range (HDR) image into a reshaped standard-dynamic-range (SDR) image and (b) a backward reshaping function for decompressing the reshaped SDR image into a reconstructed HDR image, includes (i) optimizing the forward reshaping function to minimize a deviation between the reshaped SDR image and an input SDR image corresponding to the input HDR image, (ii) optimizing the backward reshaping function to minimize a deviation between the reconstructed HDR image and the input HDR image, and (iii) until a termination condition is met, applying a correction to the input SDR image and reiterating, based on the input SDR image as corrected, the steps of optimizing the forward and backward reshaping functions.
In an embodiment, a software product, for generating (a) a forward reshaping function for compressing an input high-dynamic range (HDR) image into a reshaped standard-dynamic-range (SDR) image and (b) a backward reshaping function for decompressing the reshaped SDR image into a reconstructed HDR image, includes non-transitory, computer-readable media storing machine-readable instructions. The instructions include (i) forward optimization instructions that, when executed by a processor, cause the processor to optimize the forward reshaping function to minimize deviation between the reshaped SDR image and an input SDR image corresponding to the input HDR image, (ii) backward optimization instructions that, when executed by the processor, cause the processor to optimize the backward reshaping function to minimize deviation between the reconstructed HDR image and the input HDR image, (iii) SDR-image correction instructions that, when executed by the processor, cause the processor to apply a correction to the input SDR image, and (iv) iteration control instructions that, when executed by the processor, cause the processor to evaluate if a termination condition has been met and, as long as the termination condition has not been met, execute the SDR image correction instructions and re-execute the forward optimization instructions and the backward optimization instructions based on the input SDR image as corrected.
In operation, production system 110 generates or receives an input SDR image 180 and an input HDR image 182, representing two different versions of the same imagery. In another scenario, input SDR image 180 represents original image content, and input HDR image 182 is produced from input SDR image 180 as an HDR-compatible, and possibly enhanced, version of input SDR image 180. Encoder 112 runs iterative reshaping-function optimizer 114 to process input SDR image 180 and input HDR image 182 to optimize forward and backward reshaping functions. Based on these optimized reshaping functions, encoder 112 generates a reshaped SDR image 184 together with metadata 190. Metadata 190 encodes at least a specification of an optimized backward reshaping function to be used by HDR reconstructor 134 of decoder 132.
Production system 110 relays reshaped SDR image 184 and metadata 190 to viewing system 130 via communication link 120. Communication link 120 may be limited to a bit-depth that does not allow for direction transmission of HDR imagery at its native bit depth. In one example, communication link 120 includes one or more interfaces and/or storage media limited to 8 bits, or 10 bits. Communication link 120 may include one or more wireless or wired transmission paths, for example such as used in movie streaming. Alternatively, communication link 120 may utilize a physical storage medium such as a DVD or blu-ray disc. Viewing system 130 may be a television with decoder 132 built in, a television (or other display) coupled with a DVD/blu-ray player with decoder 132 incorporated in the DVD/blu-ray player, or a computer (e.g., desktop, laptop, smartphone, or tablet) with decoder 132 built in.
After receiving reshaped SDR image 184 and metadata 190, decoder 132 runs HDR reconstructor 134 to generate reconstructed HDR image 186 by applying to reshaped SDR image 184 a backward reshaping function specified by metadata 190. Viewing system 130 may then display reconstructed HDR image 186 on display 136.
If, in an alternative use scenario, a non-HDR-enabled viewing system receives reshaped SDR image 184 and metadata 190, this non-HDR-enabled viewing system may ignore the backward-reshaping-function specification included in metadata 190 and instead either display reshaped SDR image 184 as is, or display an SDR image that is generated from reshaped SDR image 184.
Iterative reshaping-function optimizer 114 is configured to perform an iterative process to determine the forward and backward reshaping functions. In each iteration, iterative reshaping-function optimizer 114 optimizes (a) the forward reshaping function to minimize a deviation between reshaped SDR image 184 and input SDR image 180, and (b) the backward reshaping function to minimize a deviation between reconstructed HDR image 186 and input HDR image 182. Where optimization process of a conventional single-layer HDR codec relies exclusively on adjusting properties of the forward and backward reshaping functions, iterative reshaping-function optimizer 114 utilizes a backwards error propagation to input SDR image 180. More specifically, until a termination condition is met, iterative reshaping-function optimizer 114 keeps correcting SDR input image 180 and reiterating the optimization of the forward and backward reshaping functions based on the corrected version of SDR input image 180. This iterative process with backwards error propagation to input SDR image 180 relieves complexity of the individual iterations of optimizing the reshaping functions, and may improve the performance of the resulting reshaping functions as well as make codec 100 suitable for a wider range of applications than conventional single-layer HDR codecs.
Encoder 112 may include a processor and a non-transitory memory, wherein iterative reshaping-function optimizer 114 is implemented as software stored in the non-transitory memory. This software may be provided as a standalone product to be implemented together with a third-party processor, e.g., on a third-party computer that is part of a production system.
In each iteration, step 210 optimizes the forward reshaping function to minimize deviation between reshaped SDR image 184 and input SDR image 180, and step 220 optimizes the backward reshaping function to minimize deviation between reconstructed HDR image 186 and input HDR image 182. Each of steps 210 and 220 may utilize a variety of optimization methods, for example multivariate multiple regression (MMR), tensor product b-spline (TPBS) approximation, or other optimization methods known in the art. Step 220 may implement a step 222 of applying the backward reshaping function to reshaped SDR image 184 as generated by the forward reshaping function optimized in step 210 of the same iteration. In one example of step 210, iterative reshaping-function optimizer 114 receives input SDR image 180 and input HDR image 182, optimizes the forward reshaping function to minimize deviation between (a) input SDR image 180 and (b) reshaped SDR image 184 as generated by applying the forward reshaping function to input HDR image 182. In a related example of step 220, iterative reshaping-function optimizer 114 processes input HDR image 182 and reshaped SDR image 184 (as generated by the applying the optimized forward reshaping function to input SDR image 180). In this processing, iterative reshaping-function optimizer 114 optimizes the backward reshaping function to minimize deviation between (a) input HDR image 182 and (b) reconstructed HDR image 186 as generated by applying the backward reshaping function to reshaped SDR image 184.
Step 230 is a decision step, performed after steps 210 and 220, that evaluates if a termination condition is met. If the termination condition is not met, method 200 proceeds to step 240. If the termination condition is met, method 200 may proceed to perform one or both of steps 250 and 252. Step 230 may be performed by iterative reshaping-function optimizer 114. In one embodiment, the termination condition is that a deviation measure (hereafter referred to as the HDR-only deviation measure) between reconstructed HDR image 186 and input HDR image 182 is no greater than a threshold amount. In another embodiment, the termination condition is that a composite deviation measure is met, wherein the composite deviation measure is a weighted combination of (i) the deviation between reconstructed HDR image 186 and input HDR image 182 and (ii) the deviation between reshaped SDR image 184 and input SDR image 180. Either of these two embodiments may be extended to further place a limit on the number of iterations, such that the termination condition is met when either one of the following two situations occurs: (1) the deviation measure (HDR-only or composite) is no greater than an associated threshold amount, and (2) a predefined maximum number of iterations have been performed.
Input SDR image 180 and input HDR image 182 may be color images having a plurality of color-component channels. In one embodiment pertaining to color images, the deviation measure (e.g., either one of the HDR-only deviation measure and the composite deviation measure) is based on all of the plurality of color-component channels. In another embodiment pertaining to color images, the deviation measure is based on only a subset of the plurality of color-component channels. For example, when the color-component channels are a luma channel, a first chroma channel, and a second chroma channel, the deviation measure may be based on the only luma channel such that the luma channel is used as a guide for the iterative process, deviation measure may be based on only the chroma channels such that the chroma channels are used as guides for the iterative process. In another embodiment of method 200, pertaining to color images having a luma channel and two chroma channels, iterative optimization is applied to the chroma channels, whereas the forward and backward reshaping functions for the luma channel are optimized in only a single iteration of steps 210 and 220.
Step 240 applies a correction to input SDR image 180. In one example of step 240, iterative reshaping-function optimizer 114 applies a correction to input SDR image 180 before feeding input SDR image 180, as corrected, to a next iteration of steps 210, 220, and 240. In an embodiment, step 230 evaluates a deviation measure (such as the HDR-only deviation measure), and step 240 corrects input SDR image 180 by an amount that is a function of the deviation measure. For example, step 240 may correct input SDR image 180 by an amount that is proportional to the HDR-only deviation measure, optionally limited to less than a predefined maximum correction amount to prevent too large changes to the input SDR image. In an alternative example, step 240, the function relating the correction amount to the deviation measure is more complex in order to guide the iterative optimization in a desired manner. Step 240 may correct input SDR image 180 on a pixel-by-pixel basis, with each pixel of input SDR image 180 being corrected by an amount that depends on the deviation measure pertaining to that pixel.
Step 250 outputs the reshaped SDR image generated in a last iteration of step 210 using the forward reshaping function optimized in this iteration. In one example of step 250, iterative reshaping-function optimizer 114 outputs reshaped SDR image 184. Step 252 outputs one or more parameters characterizing the backward reshaping function as optimized in last iteration of step 220. In one example of step 252, iterative reshaping-function optimizer 114 outputs these one or more parameters as part of metadata 190.
Although not shown in
When method 200 is utilizes to process a video, method 200 may be repeated for each frame of the video. Alternatively, method 200 may reoptimize the forward and backward reshaping function less frequently and, for example, use the same forward and backward reshaping functions for several successive frames.
Mathematical Description of One Embodiment of Method 200:
Here, notation is introduced for the purpose of discussing one embodiment of method 200 as applied to an input signal having t frames, wherein t is greater than or equal to one and each frame is represented by input SDR image 180 and input HDR image 182. Let vt,ich denote the ith pixel at t frame at ch channel for the input HDR signal. Let st,ich denote the ith pixel at t frame at ch channel for the input SDR signal. For the kth iteration, let the kth iteration of the forward reshaping function be denoted as FRtch,(k) ( ), let the kth iteration of a parameter set used in FRtch,(k) ( ) be denoted as FPtch,(k), let the kth iteration of the backward reshaping function be denoted as BRtch,(k) ( ) let the kth iteration of a parameter set used in BRtch,(k) ( ) be denoted as BPtch,(k) let ŝt,ich,(k) denote the ith pixel at t frame at ch channel for the reshaped SDR signal, let {circumflex over (v)}t,ich,(k) denote the ith pixel at t frame at ch channel for the reconstructed HDR signal, and let {tilde over (s)}t,ich,(k) denote the ith pixel at t frame at ch channel for the corrected input SDR signal.
At the first iteration (k=0), this embodiment of method 200 sets the corrected input SDR signal as the original input SDR signal, i.e.,
{tilde over (s)}t,ich,(0)=st,ich.
In each iteration k of this embodiment of method 200, step 210 finds the parameters FPtch,(k) of the forward reshaping function that minimize the difference between the reshaped SDR signal and the corrected input SDR signal:
FPtch,(k)=arg min Σi∥{tilde over (s)}t,ich,(k)−ŝt,ich,(k)∥2=arg min Σi∥{tilde over (s)}t,ich,(k)−FRtch,(k)(vt,ich)∥2,
wherein
ŝt,ich,(k)=FRtch,(k)(vt,ich)
In each iteration k of this embodiment of method 200, step 210 finds the parameters, i.e., BPtch,(k), of the backward reshaping function that minimize the difference between reconstructed HDR signal and input HDR signal:
wherein
{circumflex over (V)}t,ich,(k)=BRtch,(k)(ŝt,ich,(k))
At the end of each iteration k, based on the reshaped HDR signal and the input HDR signal, step 240 in this embodiment of method 200 determines the amount Δ{tilde over (s)}t,ich,(k) by which to correct the input SDR signal {tilde over (s)}t,ich,(k+1) to be used in iteration k+1. This amount is a function of {circumflex over (v)}t,ich,(k) and vt,ich:
Δ{tilde over (s)}t,ich,(k)=g({circumflex over (v)}t,ich,(k),vt,ich).
The g function may be a simple subtraction function
Δ{tilde over (s)}t,ich,(k)=vt,ich−{circumflex over (v)}t,ich,(k),
optionally scaled by a correction-scaling factor and/or further incorporating additional consideration of a gradient of the backward reshaping function. Step 240 then sets the corrected input SDR signal for the next iteration as:
{tilde over (s)}t,ich,(k+1)={tilde over (s)}t,ich,(k)+Δ{tilde over (s)}t,ich,(k)
This embodiment of method 200 defines deviation measures in HDR and SDR as
respectively. In this embodiment, step 230 continues iteration of method 200 until reaching either one of a pre-set maximum iteration count and the composite deviation measure being no greater than a threshold:
Dch,(k)=wv·Dvch,(k)+ws·Dsch,(k)≤δ,
wherein wv and ws are combination weights. In one implementation, wv equals one and ws equals zero, such that the composite deviation measure collapses to an HDR-only deviation measure. In another implementation, combination weights wv and ws are set to define a desired tradeoff between HDR distortion and SDR distortion, as discussed in further detail below in “Example 1: Polynomial Reshaping Functions” which presents results obtained with method 200 in an example using 4th-order polynomial reshaping functions.
In each of steps 310 and 320, the prediction model may incorporate MMR or TPBS approximation.
When method 300 is implemented in method 200, step 252 of method 200 (if included) may output parameter(s) 392. In one implementation, parameter(s) 390, generated in each iteration of step 310, are not used in any subsequent steps. In another embodiment, parameter(s) 390, generated in an iteration of step 310, may be used as a starting point for a subsequent iteration of step 310. The reconstructed HDR image 186 generated in each iteration of step 320 may be used in (a) step 230 to evaluate if the termination condition is met and (b) step 240 to determine the amount by which input SDR image 180 is corrected.
Encoder 400 receives input SDR image 180 and input HDR image 182 via interface 412. Encoder 400 may store input SDR image 180 and input HDR image 182 in data storage 480. Once encoder 400 has performed method 200, encoder 400 may output reshaped SDR image 184 and metadata 190 via interface 412, wherein metadata 190 includes a specification of the backward reshaping function as optimized by encoder 400. Interface 412 may include both an input interface and an output interface.
Forward optimization instructions 440 are configured to, when executed by processor 410, control processor 410 to perform step 210 of method 200. To perform step 210, processor 410 may retrieve input SDR image 180 and input HDR image 182 from data storage 480, and store or update a resulting reshaped SDR image 184 in data storage 480. Processor 410 may further use data storage 480 for temporary storage of forward reshaping parameters 390 during their optimization in step 210.
Backward optimization instructions 450 are configured to, when executed by processor 410, control processor 410 to perform step 220 of method 200. To perform step 220, processor 410 may retrieve reshaped SDR image 184 and input HDR image 182 from data storage 480, and store or update resulting backward reshaping parameters 392 and reconstructed HDR image 186 to data storage 480.
Iteration control instructions 470 are configured to, when executed by processor 410, control processor 410 to perform step 230 of method 200. Iteration control instructions 470 may include a termination condition 472. When executing iteration control instructions 470, processor 410 may retrieve one or more of input HDR image 182, reconstructed HDR image 186, input SDR image 180, and reshaped SDR image 184 from data storage 480 to determine if termination condition 472 is met. Termination condition 472 may include a deviation threshold 474 (e.g., δ). Termination condition 472 may also include a maximum iteration number 476.
SDR-image correction instructions 460 are configured to, when executed by processor 410, control processor 410 to perform step 240 of method 200. Processor 410 may retrieve input HDR image 182 and reconstructed HDR image 186 from data storage 480 to determine the amount by which input SDR image 180 is to be corrected, and then update input SDR image 180 in data storage 480.
Instructions 430 may further include one or both of MMR instructions 442 and TPBS instructions 444. MMR instructions 442 may be called by forward optimization instructions 440 and/or backward optimization instructions 450 to perform the associated optimization based on MMR. TPBS instructions 444 may be called by forward optimization instructions 440 and/or backward optimization instructions 450 to perform the associated optimization based on TPBS approximation.
For the luma component of a color-version of input SDR image 180 and input HDR image 182, method 500 includes steps 510L and 520L, luma-channel-specific embodiments of steps 210 and 220, respectively. For the luma channel, method 500 takes an input SDR image 580 and an input HDR image 582 as input. Input SDR image 580 and an input HDR image 582 are color images. Input SDR image 580 has a luma component, referred to as input SDR image luma component 580L, and two chroma components, referred to as input SDR image chroma components 580C0 and 580C1. Input HDR image 582 has a luma component, referred to as input HDR image luma component 582L, and two chroma components, referred to as input HDR image chroma components 582C0 and 582C1. Step 510L processes input SDR image luma component 580L and input HDR image luma component 582L. Step 510L optimizes a forward reshaping function, characterized by forward reshaping parameters 590L for the luma component, to minimize deviation between a reshaped SDR image luma component 584L and input SDR image luma component 580L. Step 520L optimizes the backward reshaping function, characterized by backward reshaping parameters 592L for the luma component, to minimize deviation between a reconstructed HDR image luma component 586L and input HDR image luma component 582L.
Once step 510L is completed, method 500 may perform a step 550L of outputting reshaped SDR image luma component 584L. Once step 520L is completed, method 500 may perform a step 552L of outputting backward reshaping parameters 592L for the luma component.
For the associated chroma channels, method 500 includes steps 510C, 520C, 530C, and 540C, chroma-specific embodiments of steps 210, 220, 230, and 240, respectively.
Step 510C utilizes MMR to optimize a forward reshaping functions, characterized by forward reshaping parameters 590C0 for the first chroma component and 590C1 for the second chroma component, to minimize (a) deviation between a reshaped SDR image chroma component 584C0 and input SDR image chroma component 580C0 and (b) deviation between a reshaped SDR image chroma component 584C1 and input SDR image chroma component 580C1. The MMR process of step 510C takes all color-components of input HDR image 582 as input to (a) predict both reshaped SDR chroma components 584C0 and 584C1 and (b) optimize the forward reshaping functions for the chroma channels characterized by respective forward reshaping parameters 590C0 and 590C1.
Next, step 520C utilizes MMR to optimize the backward reshaping functions, characterized by backward reshaping parameters 592C0 for the first chroma component and backward reshaping parameters 592C1 for the second chroma component, to minimize (a) deviation between a reconstructed HDR image chroma component 586C0 and input HDR image chroma component 582C0 and (b) deviation between a reconstructed HDR image chroma component 586C1 and input HDR image chroma component 582C1. As input to predict the reconstructed HDR chroma components and generate the backward reshaping functions for both chroma channels, the MMR process of step 520 takes reshaped SDR chroma components 584C0 and 584C1 (generated in step 510C of the same iteration) and reshaped SDR luma component 584L generated in step 510L.
Step 530C is a decision step that evaluates if a termination condition is met for the chroma components. If the termination condition is not met, method 500 proceeds to step 540C and a next iteration of steps 510C, 520C, and 530C. Step 540C applies a correction to one of both of input SDR image chroma components 580C0 and 580C1. Steps 530C and 540C are chroma-specific embodiments of steps 230 and 240, respectively. If step 530C finds that the termination condition is met, method 500 may perform steps 550C and 552C. Step 550C outputs reshaped SDR image chroma components 584C0 and 584C1 as generated in the last iteration of step 510C. Step 552C outputs backward reshaping parameters 592C0 and 590C1 for the chroma components, as optimized in last iteration of step 520C.
In one class of embodiments, optimization in steps 510C, 520C, 510L, and 520L is based on individual pixels of the involved images. One such embodiment is discussed below in reference to
For the luma component, method 600 includes steps 610L and 620L, embodiments of steps 510L and 520L, respectively.
Step 610L optimizes the forward reshaping function for the luma component and includes steps 612L and 614L. Step 612L processes individual pixels of a reshaped SDR image luma component 684L and of input SDR image luma component 580L to determine a first collective (across all pixels) deviation between reshaped SDR image luma component 684L and input SDR image luma component 580L. Step 614L optimizes pixel-based parameters 690L of the forward reshaping function for the luma channel to minimize the first collective deviation. Step 614L may determine the luma-channel forward reshaping function according to cumulative distribution function (CDF) matching between a histogram of reshaped SDR image luma component 684L and a histogram of input SDR image luma component 580L.
Step 620L optimizes the backward reshaping function for the luma component and includes steps 622L and 624L. Step 622L processes individual pixels of a reconstructed HDR image luma component 686L and of input HDR image luma component 582L to determine a second collective (across all pixels) deviation between reconstructed HDR image luma component 686L and input HDR image luma component 582L. Step 624L optimizes pixel-based parameters 692L of the backward reshaping function for the luma channel to minimize the second collective deviation. Step 624L may determine the luma-channel forward reshaping function according to CDF matching between a histogram of reconstructed HDR image luma component 686L and a histogram of input HDR image luma component 582L.
Once step 610L is completed, method 600 may perform step 550L to output reshaped SDR image luma component 684L. Once step 620L is completed, method 600 may perform step 552L to output pixel-based parameters 692L.
For the chroma channels, method 600 includes steps 610C, 620C, 630C, and 640C, which are embodiments of steps 510C, 520C, 530C, and 540C, respectively.
Step 610C optimizes the forward reshaping functions for the chroma components and includes steps 612C and 614C. For each chroma component, step 612C processes individual pixels of a reshaped SDR image chroma component 684C0/C1 and of input SDR image chroma component 580C0/C1 to determine a first collective (across all pixels) deviation between reshaped SDR image chroma component 684C0/C1 and input SDR image chroma component 580C0/C1. Step 612C thus computes a first collective deviation for each chroma channel. Step 614C utilizes MMR to optimize (a) pixel-based parameters 690C0 of the forward reshaping function for the first chroma channel to minimize the first collective deviation for the first chroma channel and (b) pixel-based parameters 690C1 of the forward reshaping function for the second chroma channel to minimize the first collective deviation for the second chroma channel.
Step 620C optimizes the backward reshaping function for the chroma component and includes steps 622C and 624C. For each chroma component, step 622C processes individual pixels of a reconstructed HDR image chroma component 686C0/C1 and of input HDR image chroma component 582C0/C1 to determine a second collective (across all pixels) deviation between reconstructed HDR image chroma component 686C0/C1 and input HDR image chroma component 582C0/C1. Step 622C thus computes a second collective deviation for each chroma channel. Step 624C utilizes MMR to optimize (a) pixel-based parameters 692C0 of the backward reshaping function for the first chroma channel to minimize the second collective deviation for the first chroma channel, and (b) pixel-based parameters 692C1 of the backward reshaping function for the second chroma channel to minimize the second collective deviation for the second chroma channel.
Step 630C is a decision step that evaluates if a termination condition is met for the chroma components. If the termination condition is not met, method 600 proceeds to step 640C and a next iteration of steps 610C, 620C, and 630C. Step 640C applies a correction to one or both of input SDR image chroma components 580C0 and 580C1. Steps 630C and 640C are embodiments of steps 530C and 540C, respectively, based on pixel-by-pixel processing. If step 630C finds that the termination condition is met, method 600 may proceed to steps 550C and 552C to output reshaped SDR image chroma components 684C0 and 684C1 and pixel-based parameters 692C0 and 692C1, respectively.
One example of method 600 is discussed in further detail below in “Example 2: Pixel-Based MMR”.
Method 700 includes steps 702 and 704 (see
In summary, step 704 generates input HDR 3D-binned image data 782 and input SDR 3D-binned image data 780. Input HDR 3D-binned image data 782 includes an input HDR 3D-binned luma component 782L, an input HDR 3D-binned chroma component 782C0, and an input HDR 3D-binned chroma component 782C1. The term input HDR 3D-binned chroma component 782C refers non-specifically to one of input HDR 3D-binned chroma components 782C0 and 782C1. Input SDR 3D-binned image data 780 includes an input SDR 3D-binned luma component 782L, an input SDR 3D-binned chroma component 780C0, and an input SDR 3D-binned chroma component 780C1. The term input SDR 3D-binned chroma component 780C refers non-specifically to one of input SDR 3D-binned chroma components 780C0 and 780C1.
As compared to pixel-based method 600, 3D-bin based method 700 may save memory space, and offer faster processing and/or reduce the requirements to computational resources, since all similarly appearing pixels are grouped together. Steps 702 and 704 may be encoded in machine-readable instructions 430 of encoder 400 of
After completion of step 704, method 700 proceeds to optimization of the luma channel as shown in
For the luma component, method 700 includes steps 710L and 720L, embodiments of steps 510L and 520L, respectively.
Step 710L optimizes the forward reshaping function for the luma component and includes steps 712L and 714L. Steps 712L and 714L are similar to steps 612L and 614L except for operating on 3D-bins instead of individual pixels. As a result, step 710L generates 3D-binned reshaped SDR luma component data 784L in place of reshaped SDR image luma component 684L, and optimizes 3D-bin-based parameters 790L of the forward reshaping function for the luma channel.
Step 720L optimizes the backward reshaping function for the luma component and includes steps 722L and 724L. Steps 722L and 724L are similar to steps 622L and 624L except for operating on 3D-bins instead of individual pixels. As a result, step 720L generates 3D-bin-based parameters 792L of the backward reshaping function for the luma channel, and operates with 3D-binned reconstructed HDR luma component data 786L in place of reconstructed HDR image luma component 686L.
Once step 710L is completed, method 700 may perform step 550L to output reshaped 3D-binned reshaped SDR chroma component data 784C. Without departing from the scope hereof, method 700 may include converting reshaped 3D-binned reshaped SDR luma component data 784L to a pixel-based representation of the reshaped SDR luma component and outputting this pixel-based representation in step 550L. Once step 720L is completed, method 700 may perform step 552L to output 3D-bin-based parameters 792L.
For the chroma channels, method 700 includes steps 710C, 720C, 730C, and 740C, which are embodiments of steps 510C, 520C, 530C, and 540C, respectively.
Step 710C optimizes the forward reshaping functions for the chroma components and includes steps 712C and 714C. Steps 712C and 714C are similar to steps 612C and 614C except for operating on 3D-bins instead of individual pixels. As a result, step 710C generates 3D-binned reshaped SDR chroma component data 784C0 and 784C1 in place of reshaped SDR image chroma components 684C0 and 684C1, and optimizes 3D-bin-based parameters 790C0 and 790C1 of the forward reshaping function for the first and second chroma channels, respectively.
Step 720C optimizes the backward reshaping functions for the chroma components and includes steps 722C and 724C. Steps 722C and 724C are similar to steps 622C and 624C except for operating on 3D-bins instead of individual pixels. As a result, step 720C generates 3D-bin-based parameters 792C0 of the backward reshaping function for the first chroma channel and 3D-bin-based parameters 792C1 of the backward reshaping function for the second chroma channel, and operates with 3D-binned reconstructed HDR chroma component data 786C0 and 786C1 in place of reconstructed HDR image chroma components 686C0 and 686C1.
Step 730C is a decision step that evaluates if a termination condition is met for the chroma components. If the termination condition is not met, method 700 proceeds to step 740C and a next iteration of steps 710C, 720C, and 730C. Step 740C applies a correction to one or both of input SDR 3D-binned chroma components 780C0 and 780C1. Steps 730C and 740C are similar to steps 630C and 640C, respectively, except for being based on 3D-bin-by-3D-bin processing rather than pixel-by-pixel processing. If step 730C finds that the termination condition is met, method 700 may proceed to steps 550C and 552C to output reshaped 3D-binned reshaped SDR chroma component data 784C0 and 784C1 and 3D-bin-based parameters 792C0 and 782C1, respectively. Without departing from the scope hereof, method 700 may include converting reshaped 3D-binned reshaped SDR chroma component data 784C0 and 784C1 to pixel-based representations of the reshaped SDR chroma components and outputting these pixel-based representations in step 550C.
One example of method 700 is discussed in further detail below in “Example 3: 3D-Bin-Based MMR”.
Without departing from the scope hereof, the chroma channels may be processed by 3D-bin-based method 700 (according to the flow charts outlined in
Each color-component channel, method 800 takes input SDR image 580 and input HDR image 582 as input, and performs one or more iterations of steps 810, 820, 830, and 840 to generate the forward and backward reshaping functions, as well as associated reshaped SDR image data, for the color-component channel under consideration. If, for a color-component channel, only one iteration is performed, step 840 is omitted from the processing of this color-component channel.
Step 810 optimizes the forward reshaping function for the color-component channel under consideration and includes steps 812 and 814. Step 812 processes individual pixels of a reshaped SDR image color component 784 and of the associated color component of input SDR image 580, i.e., one of color components 580L, 580C0, and 580C1, to determine a first collective (across all pixels) deviation between reshaped SDR image color component 784 and the color component of input SDR image 580. Step 814 utilizes TPBS approximation to optimize pixel-based parameters 790 of the forward reshaping function for the color-component channel to minimize the first collective deviation.
Step 820 optimizes the backward reshaping function for the color-component channel under consideration and includes steps 822 and 824. Step 822 processes individual pixels of a reconstructed HDR image color component 786 and of the associated color component of input HDR image 582, i.e., one of color components 582L, 582C0, and 582C1, to determine a second collective (across all pixels) deviation between reconstructed HDR image color component 786 and the color component of input HDR image 582. Step 824 utilizes TPBS approximation to optimize pixel-based parameters 792 of the backward reshaping function for the color-component channel to minimize the second collective deviation.
Step 830 is a decision step that evaluates if a termination condition is met for the color component. If the termination condition is not met, method 800 proceeds to step 840 and a next iteration of steps 810, 820, and 830. Step 840 applies a correction to the color component of input SDR image 580 under consideration. Steps 830 and 840 are embodiments of steps 230 and 240, respectively. If step 830 finds that the termination condition is met, method 800 may proceed to steps 850 and 852. Step 850 outputs reshaped SDR image color component 784 for the color-component channel under consideration, as optimized in a last iteration of step 810. Step 852 outputs pixel-based parameters 792 for the color-component channel under consideration, as optimized in a last iteration of step 820.
One example of method 800 is discussed in further detail below in “Example 4: Pixel-Based TPBS Approximation”.
In the same manner that method 700 is 3D-bin-based alternative to pixel-based method 600, method 800 may be modified to instead process 3D-binned image data. One example of such a method is discussed in further detail below in “Example 5: 3D-Bin-Based TPBS Approximation”.
In this example, method 200 is performed with each of the forward reshaping function FRtch,(k) ( ) and the backward reshaping function BRtch,(k) ( ) being a 4th-order polynomial, such that each of steps 210 and 220 optimizes five polynomial coefficients. This optimization may be done according to a least-squares minimization. While more advanced optimization algorithms are expected to provide better performance, this example is used here as an illustration and may be implemented in scenarios where computational ease outweighs performance. The input data processed in this example of method 200 has only a single color-component, equivalent to a luma component, a single chroma component, or grayscale image data.
An input HDR image 182 is generated with random pixel values. An input SDR image 180 is generated from input HDR image 182 by clipping the dynamic range, introducing polynomial distortion, and adding noise. The resulting correspondence 900 between HDR pixel values and SDR pixel values is plotted in
In the following, we outline one example of method 600.
Luma Channel:
In this example, the luma channel (denoted y) is a single-channel predictor. The forward reshaping function in step 620L is a one-dimensional (1D) lookup table (LUT) and the backward reshaping function in step 610L is an 8-piece 2nd-order polynomial predictor. The backward reshaping function is also expressed as a 1D-LUT. Note that since the predictor is 1D-LUT and can be easily obtained in an invertible fashion, there is no iteration involved in the luma channel. We define the following notations:
Chroma Channel:
The chroma channels (c0 and c1) use MMR prediction. In MMR forward reshaping in step 610C, we take all three color-component channels from the HDR domain (vt,iy, vt,ic0, vt,ic1) as input to predict the SDR chroma-components st,ic0 st,ic1. The predicted (reshaped) value is (ŝt,iy,ŝt,ic0,(k), ŝt,ic1,(k)) at the kth iteration. We use these three SDR predicted values as input for the backward reshaping optimization in step 620C to predict the HDR chroma value (vt,ic0, vt,ic1). The predicted (reconstructed) value at the kth iteration is ({circumflex over (v)}t,ic0,(k), {circumflex over (V)}t,ic1,(k). Note that there are some matrices used in each iteration that will not change and can be pre-computed.
Initialization:
At the first iteration (k=0), we set the corrected input SDR image as the original input SDR image, i.e., {tilde over (s)}t,ich,(0)=st,ich for all 3 channels (ch=y, c0, and c1). Denote the MMR expanded form for the ith input HDR pixel as:
Collect all P pixels to get an expanded matrix:
We can also define the following matrix, which will be used in the coefficient optimization process:
BtV=VtT·Vt.
We also prepare the input HDR chroma components in vector form:
Note that vtc0, vtc1, Vt, BtV, and (BtV)−1 will not change during the iteration since they represent the input HDR image. They can be pre-computed and stored in memory without re-computing.
Forward Reshaping:
With forward MMR coefficients mtF,c0,(k) and mtF,c1,(k) in the c0 and c1 channels, the predicted SDR values at the kth iteration can be found via
ŝtc0,(k)=Vt·mtF,c0,(k), and
Ŝtc1,(k)=Vt·mtF,c1,(k),
where
At each iteration, we collect the corrected SDR values from the previous iteration as vectors
These two vectors will be the target values the forward prediction to achieve at iteration k. Note that luma predicted values won't change at each iteration.
Define the following two vectors to facilitate the discussion of coefficient optimization:
atV,c0,(k)=VtT·{tilde over (s)}tc0,(k), and
atV,c1,(k)=VtT·{tilde over (s)}tc1,(k).
The goal of coefficient optimization is to find the MMR coefficients to minimize the mean-square error (MSE) between the corrected input SDR signal and the predicted (reshaped) SDR signal at iteration k:
mtF,c0,(k),opt=arg min∥ŝtc0,(k)−ŝtc0,(k)∥2=arg min∥Vt·mtF,c0,(k)−{tilde over (s)}tF,c0,(k)∥2
mtF,c1,(k),opt=arg min∥ŝtc1,(k)−ŝtc1,(k)∥2=arg min∥Vt·mtF,c1,(k)−{tilde over (s)}tF,c1,(k)∥2
The optimal MMR coefficients in forward reshaping can be found at the least squared solutions:
mtF,c0,(k),opt=(BtV)−1atV,c0,(k), and
mtF,c1,(k),opt=(BtV)−1atV,c1,(k).
The optimal predicted SDR values at iteration k can be expressed as
ŝtc0,(k)=Vt·mtF,c0,(k)opt, and
ŝtc1,(k)=Vt·mtF,c1,(k)opt.
Backward Reshaping:
At the kth iteration, denote the MMR expanded form for a predicted SDR pixel
We can collect all P pixels together to construct a matrix
Define the following matrix which will be used in coefficient optimization process:
BtS,(k)=(St(k))T·St(k).
With backward MMR coefficients mtB,c0,(k) and mtB,c1,(k) in the c0 and c1 channels, the predicted HDR values at the kth iteration can be found via
{circumflex over (v)}tc0,(k)=St(k)·mtB,c0,(k), and
{circumflex over (v)}tc1,(k)=St(k)·mtB,c1,(k),
where
We also define the following vectors to facilitate the coefficient optimization:
atS,c0,(k)=(St(k))T·vtc0,
atS,c1,(k)=(St(k))T·vtc1.
The optimization goal is to find the MMR coefficients to minimize the MSE between the original input HDR signal and the predicted (reconstructed) HDR signal at iteration k:
mtB,c0,(k),opt=argmin∥{circumflex over (v)}tc0,(k)−Vtc0∥2=argmin∥St(k)·mtB,c0,(k)·vtc0∥2,
mtB,c1,(k),opt=argmin∥{circumflex over (v)}tc1,(k)−Vtc1∥2=argmin∥St(k)·mtB,c1,(k)·vtc1∥2.
The optimal MMR coefficients in backward reshaping can be found as the least squared solutions:
mtB,c0,(k),opt=(BtS,(k))−1atS,c0,(k), and
mtB,c1,(k),opt=(BtS,(k))−1atS,c1,(k).
The predicted HDR values can be expressed as
{circumflex over (V)}tc0,(k)=St(k)·mtB,c0,(k),opt, and
{circumflex over (V)}tc1,(k)=St(k)·mtB,c1,(k),opt.
Iteration Control and Computation of Corrected Input SDR Signal in Steps 630C and 640C:
At the end of each iteration, based on the reconstructed HDR signal and the input HDR signal, we determine the amounts Δ{tilde over (s)}t,ic0,(k) and Δśt,ic1,(k) by which to correct the input SDR signals {tilde over (s)}t,ic0,(k+1) and {tilde over (s)}t,ic1,(k+1) respectively. These correction amounts are a function of {circumflex over (v)}t,ich,(k) and vt,ich:
ε is a threshold to determine whether to proceed to input SDR image correction in step 640C. αch is a convergence scaling factor and serves as tradeoff between convergence speed and distortion. To avoid too much change in the input SDR signal, we can set a correction upper bound as
Δ{tilde over (s)}t,ich,(k)=min(Δ{tilde over (s)}t,ich,(k),λ).
The corrected input SDR value is updated, in step 640C, for the next iteration as:
{tilde over (s)}t,ich,(k+1)=min({tilde over (s)}t,ich,(k)+Δ{tilde over (s)}t,ich,(k).
Iteration Termination:
Iteration termination is carried out as discussed above in reference to
In the following, we outline one example of method 700.
Basis of 3D-Binning:
Let vt,i=[vt,iy vt,ic0 vt,ic1]T denote the three color values of the ith pixel from the input HDR signal for the tth frame. Let st,i=[st,iy st,ic0 st,ic1]T denote the corresponding pixel in the input SDR image at the ith pixel for the tth frame. We quantize the HDR image with three channel values (Y, C0 and C1) using a fixed number of bins Qy, QC
dtch=(vt,ich,max−vt,ich,min)/Qch.
Denote the 3D histogram bin as ΩlQ,v, where Q=[(Qy, QC
q=qy·QC
We also compute the sum of each color component in the SDR for each 3D-bin. Let Ψt,yQ,s, Ψt,C
Assume we have P pixels, the operations can be summarized as follows in pseudo code:
The next step is to find the 3D histogram bins that have non-zero number of pixels. In other words, we collect all non-zero entry to set ΦtQ. We compute the averages of HDR (
Let Pt denote the number of elements in ΦtQ.
and
Then, we have the mapping pair {
Optimization:
This example of method 700 then applies iterative MMR-based optimization to 3D mapping pairs {
Initialization:
At the first iteration (k=0), we set the corrected input 3D-binned SDR as the original input SDR. i.e.,
{tilde over (Ψ)}t,ch,qQ,s=
Denote the MMR expanded form for the ith input HDR entry as
Collect all Pt entries to get an expanded matrix
We also define the following matrix, which will be used in coefficient optimization process:
BtV=VtT·Vt
We also prepare the input 3D-binned HDR chroma value in vector forms:
Note that vtc0, vtc1, Vt, BtV, and (BtV)−1 will not change during the iteration since they represent the HDR input value. They can be pre-computed and store in memory without re-computing.
Forward Reshaping:
With forward MMR coefficients mtF,c0,(k) and mtF,c1,(k) in the c0 and c1 channels, the predicted SDR values at the kth iteration can be found via
Ŝtc0,(k)=Vt·mtF,c0,(k),
Ŝtc1,(k)=Vt·mtF,c1,(k),
where
At each iteration, we collect the corrected SDR values from previous iteration as vectors
These two vectors will be the target values for the forward prediction process to achieve at iteration k. Note that luma predicted value will not change at each iteration.
Define the following two vectors to facilitate the discussion of coefficient optimization:
atV,c0,(k)=VtT·{tilde over (s)}tc0,(k), and
atV,c1,(k)=VtT·{tilde over (s)}tc1,(k).
The goal of coefficient optimization is to find the MMR coefficients to minimize the MSE between the corrected input SDR signal and the predicted SDR signal at iteration k:
mtF,c0,(k),opt=arg min∥ŝtc0,(k)−ŝtc0,(k)∥2=arg min∥Vt·mtF,c0,(k)−{tilde over (s)}tc0,(k)∥2,
mtF,c1,(k),opt=arg min∥ŝtc1,(k)−ŝtc1,(k)∥2=arg min∥Vt·mtF,c1,(k)−{tilde over (s)}tc1,(k)∥2.
The optimal MMR coefficients in forward reshaping can be solved via least squared solution:
mtF,c0,(k),opt=(BtV)−1atV,c0,(k)
mtF,c1,(k),opt=(BtV)−1atV,c1,(k)
The optimal predicted SDR value at iteration k can be expressed as
ŝtc0,(k)=Vt·mtF,c0,(k)opt, and
ŝtc1,(k)=Vt·mtF,c1,(k)opt.
Backward Reshaping:
At the kth iteration, denote the MMR expanded form for a predicted SDR pixel as
We can collect all P pixels together to construct a matrix
Define the following matrix which will be used in coefficient optimization process:
BtS,(k)=(St(k))T·St(k)
With backward MMR coefficients mtB,c0,(k) and mtB,c0,(k) in the c0 and c1 channels, the predicted HDR values at the kth iteration can be found via
{circumflex over (v)}tc0,(k)=St(k)·mtB,c0,(k)
{circumflex over (v)}tc1,(k)=St(k)·mtB,c1,(k)
where
We also define the following vectors to facilitate the coefficient optimization:
atS,c0,(k)=(St(k))T·vtc0
atS,c1,(k)=(St(k))T·vtc1
The optimization goal is to find the MMR coefficients to minimize the MSE between the original input HDR entry and the predicted HDR entry at iteration k:
mtB,c0,(k),opt=argmin∥{circumflex over (v)}tc0,(k)−vtc0∥2=argmin∥St(k)·mtB,c0,(k)−vtc0∥2,
mtB,c1,(k),opt=argmin∥{circumflex over (v)}tc1,(k)−vtc1∥2=argmin∥St(k)·mtB,c1,(k)−vtc1∥2.
The optimal MMR coefficients in backward reshaping can be found as the least squared solutions:
mtB,c0,(k),opt=(Bts,(k))−1atS,c0,(k), and
mtB,c1,(k),opt=(Bts,(k))−1atS,c1,(k).
The predicted HDR values can be expressed as
{circumflex over (V)}tc0,(k)=St(k)·mtB,c0,(k),opt, and
{circumflex over (V)}tc1,(k)=St(k)·mtB,c1,(k),opt.
Iteration Control and Computation of Corrected Input SDR Signal:
At the end of each iteration, based on the reshaped HDR signal and original input HDR signal, we determine the amounts Δ{tilde over (Ψ)}t,C
The corrected input SDR value is updated for the next iteration as:
{tilde over (Ψ)}t,ch,iQ,s,(k+1)={tilde over (Ψ)}t,ch,iQ,s,(k)+Δ{tilde over (Ψ)}t,ch,iQ,s,(k).
Iteration Termination:
Iteration termination is carried out as discussed above in reference to
In the following, we outline one example of method 800.
Initialization:
At the first iteration (k=0), we set the corrected input SDR as the original input SDR, i.e.,
{tilde over (s)}t,ich,(0)=st,ich for all 3 channels (ch=y, c0, and c1).
Recall that the TPBS is defined for channel ch at iteration k as
where Bt,kFTPB,ch(vt,iy, vt,ic0, vt,ic1) is the forward TPBS basis function, and mt,kF,ch,(k) is the corresponding forward coefficient. With all P pixels in one frame, we can construct the matrix form for the design matrix:
We can also define the following matrix, which will be used in coefficient optimization process:
BtTPB,V,ch=(VtTPB,ch)T·VtTPB,ch.
We also prepare the input HDR chroma values in vector form:
Note that vtch, VtTPB,ch, BtTPB,V,ch, and (BtTPB,V,ch)−1 will not change during the iteration since they represent the HDR input value. They can be pre-computed and store in memory without re-computing.
Forward Reshaping:
With forward TPBS coefficients mtF,ch,(k) in channel ch, the predicted SDR values at the kth iteration can be found via
stch,(k)=VtTPB,ch·mtF,ch,(k),
where
At each iteration, we collect the corrected SDR value from previous iteration as a vector for each channel:
These three vectors will be the target value the forward prediction to achieve at iteration k. Define the following vectors for the three channels to facilitate the discussion of coefficient optimization:
atV,ch,(k)=(VtTPB,ch)T·{tilde over (s)}tch,(k).
The coefficient optimization goal is to find the TPBS coefficients to minimize the MSE between the corrected input SDR signal and the predicted SDR signal at iteration k. For each channel, we solve the following optimization problem:
mtF,ch,(k),opt=arg min∥ŝtch,(k)−{tilde over (s)}tch,(k)∥2=arg min∥vtTPB,ch·mtF,ch,(k),opt−{tilde over (s)}tch,(k)∥2.
The optimal TPB coefficients in forward reshaping can be found at the least squared solution:
mtF,ch,(k),opt=(BtTPB,V,ch)−1atV,ch,(k).
The optimal predicted SDR value at iteration k can be expressed as
ŝtch,(k)=VtTPB,ch·mtF,ch,(k),opt.
Backward Reshaping:
At the kch iteration, denote the predicted TPBS value in HDR as
where Bt,kBTPB,ch(Ŝt,iy,Ŝt,ic0,Ŝt,ic0) is the backward TPBS basis function, and mt,kB,ch is the corresponding backward coefficient. With all P pixels in one frame, we can construct the matrix form for the design matrix:
Define the following matrix which will be used in coefficient optimization process:
BtTPB,S,ch,(k)=StTPB,ch,(k))T·StTPB,ch,(k)
With backward TPBS coefficients mtB,ch,(k) in channel ch, the predicted HDR values at the kth iteration can be found via
{circumflex over (v)}tch,(k)=StTPB,ch,(k)·mtB,ch,(k),
where
We also define the following vectors to facilitate the coefficient optimization:
atS,ch,(k)=(StTPB,ch,(k))T·Vtch
The optimization goal is to find the TPBS coefficients to minimize the MSE between the original input HDR signal and the predicted HDR signal at iteration k:
mtB,ch,(k),opt=argmin∥{circumflex over (v)}tch,(k)−vtch∥2=argmin∥StTPB,ch,(k)·mtB,ch,(k)−vtch∥2.
The optimal MMR coefficients in backward reshaping can be found as the least squared solution:
mtB,ch,(k),opt=(BtTPB,ch,(k))−1atS,ch,(k)
The predicted HDR value can be expressed as
{circumflex over (v)}tch,(k)=StTPB,ch,(k)·mtB,ch,(k),opt.
Iteration Control and Computation of Corrected Input SDR Signal:
At the end of each iteration, based on the reshaped HDR signal and original input HDR signal, we determine the amount {tilde over (Δ)}st,ich,(k) by which to correct the input SDR signal {tilde over (s)}t,ich,(k+1). This correction amount is a function of {circumflex over (v)}t,ich,(k) and vt,ich:
To avoid too much change in the input SDR signal, we can set a correction upper bound as
Δ{tilde over (s)}t,ich,(k)=min(Δ{tilde over (s)}t,ich,(k),λ).
The corrected input SDR value is updated for the next iteration as:
{tilde over (s)}t,ich,(k+1)={tilde over (s)}t,ich,(k)+Δ{tilde over (s)}t,ich,(k).
Iteration Termination:
Iteration termination is carried out as discussed above in reference to
In the following, we outline one example of method 800 modified to be based on processing of 3D-bins. The 3D-binning is the same as in “Example 3: 3D-Bin-Based MMR”. Following this 3D-binning procedure, we have mapping pairs {
Initialization:
At the first iteration (k=0), we set the corrected 3D-binned input SDR signal as the original 3D-binned input SDR signal, i.e.,
{tilde over (Ψ)}t,ch,iQ,s=
The predicted value is
With all Pt entries in one frame, we can construct the matrix form for the design matrix:
We can also define the following matrix, which will be used in the coefficient optimization process:
BtTPB,V,ch=(VtTPB,ch)T·VtTPB,ch
We also prepare the input HDR ch channel in vector forms:
Note that vtch, VtTPB,ch, BtTPB,V,ch, and (BtTPB,V,ch)−1 will not change during the iteration since they represent the input HDR value. They can be pre-computed and store in memory without re-computing.
Forward Reshaping:
With forward TPBS coefficients mtF,ch,(k), in channel ch, the predicted SDR values at the kth iteration can be found via
Ŝtch,(k)=VtTPB,ch·mtF,ch,(k),
where
At each iteration, we collect the corrected SDR value from previous iteration as a vector
These three vectors will be the target values for the forward prediction to achieve at iteration k. Define the following vectors for the three channels to facilitate the discussion of coefficient optimization:
atTPB,V,ch,(k)=(VtTPB,ch)T·{tilde over (s)}tch,(k).
The coefficient optimization goal is to find the TPBS coefficients to minimize the MSE between the corrected input SDR signal and the predicted SDR signal at iteration k:
mtF,ch,(k),opt=arg min∥ŝtch,(k)−{tilde over (s)}tch,(k)∥2=arg min∥VtTPB,ch·mtF,ch,(k)−{tilde over (s)}tch,(k)∥2.
The optimal TPBS coefficients in forward reshaping can be found as the least squared solution:
mtF,ch,(k),opt=(BtTPB,V,ch)−1atTPB,V,ch,(k).
The optimal predicted SDR value at iteration k can be expressed as
Ŝtch,(k)=VtTPB,ch·mtF,ch,(k),opt.
Backward Reshaping:
At the kth iteration, denote the predicted TPB value in HDR as
where Bt,kBTPB,ch({circumflex over (Ψ)}t,y,iQ,s,(k), {circumflex over (Ψ)}t,C
Define the following matrix which will be used in coefficient optimization process:
BtTPB,S,ch,(k)=(StTPB,ch,(k))T·StTPB,ch,(k).
With backward TPBS coefficients mtB,ch,(k) in channel ch, the predicted HDR values at the kth iteration can be found via
{circumflex over (v)}tch,(k)=StTPB,ch,(k)·mtB,ch,(k).
where
We also define the following vectors to facilitate the coefficient optimization:
atS,ch,(k)=(StTPB,ch,(k))T·vtch
The optimization goal is to find the TPBS coefficients to minimize the MSE between the original input HDR signal and the predicted HDR signal at iteration k:
mtB,ch,(k),opt=argmin∥{circumflex over (v)}tch,(k)−vtch∥2=argmin∥StTPB,ch,(k)·mtB,ch,(k)−vtch∥2
The optimal TPBS coefficients in backward reshaping can be solved via least squared solution:
mtB,ch,(k),opt=(BtTPB,S,ch,(k))−1atS,ch,(k)
The predicted HDR value can be expressed as
{circumflex over (v)}tch,(k)=StTPB,ch,(k)·mtB,ch,(k),opt.
Iteration Control and Computation of Corrected Input SDR Signal:
At the end of each iteration, based on the reshaped HDR signal and original input HDR signal, we determine the amount Δ{tilde over (Ψ)}t,ch,iQ,s,(k) by which to correct the input SDR signal {tilde over (Ψ)}t,ch,iQ,s,(k+1). This correction amount is a function of {circumflex over (Ψ)}t,ch,iQ,v,(k) and
The corrected input SDR value is updated for the next iteration as:
{tilde over (Ψ)}t,ch,iQ,s,(k+1)={tilde over (Ψ)}t,ch,iQ,s,(k)+Δ{tilde over (Ψ)}t,ch,iQ,s,(k).
Iteration Termination:
Iteration termination is carried out as discussed above in reference to
Combinations of Features
Features described above as well as those claimed below may be combined in various ways without departing from the scope hereof. For example, it will be appreciated that aspects of one image reshaping method or product, described herein, may incorporate or swap features of another image reshaping method or product described herein. The following examples illustrate some possible, non-limiting combinations of embodiments described above. It should be clear that many other changes and modifications may be made to the methods, products, and systems herein without departing from the spirit and scope of this invention:
(A1) One method, for generating (a) a forward reshaping function for compressing an input high-dynamic range (HDR) image into a reshaped standard-dynamic-range (SDR) image and (b) a backward reshaping function for decompressing the reshaped SDR image into a reconstructed HDR image, includes (i) optimizing the forward reshaping function to minimize a deviation between the reshaped SDR image and an input SDR image corresponding to the input HDR image, (ii) optimizing the backward reshaping function to minimize a deviation between the reconstructed HDR image and the input HDR image, and (iii) until a termination condition is met, applying a correction to the input SDR image and reiterating, based on the input SDR image as corrected, the steps of optimizing the forward and backward reshaping functions.
(A2) In the method denoted as (A1), the step of applying the correction may include applying, to each pixel of each of one or more color-component channels of the input SDR image, a deviation-based correction value that is proportional to the deviation between the reconstructed HDR image and the input HDR image for a corresponding pixel and a corresponding color-component channel, at least as long as the deviation-based correction value is less than a predefined maximum correction value.
(A3) In the method denoted as (A2), for each pixel of each of the one or more color-component channels of the input SDR image, the deviation-based correction value may be proportional to the deviation between the reconstructed HDR image and the input HDR image for a corresponding pixel and a corresponding color-component channel.
(A4) In either of the methods denoted as (A2) and (A3), the termination condition may, at least until reaching a predefined maximum number of iterations of the steps of optimizing the forward and backward reshaping functions, be that a deviation measure at least between the reconstructed HDR image and the input HDR image is no greater than a threshold amount.
(A5) In the method denoted as (A4), the deviation measure may be a weighted combination of (i) the deviation between the reconstructed HDR image and the input HDR image and (ii) a deviation between the reshaped SDR image and the input SDR image.
(A6) In either of the methods denoted as (A4) and (A5), the deviation measure may be based on only a subset of the color-component channels defining each of the input SDR and input HDR images.
(A7) In the method denoted as (A6), the color-component channels may include a luma channel and two chroma channels, and the subset may be the luma channel.
(A8) In the method denoted as (A6), the color-component channels may include a luma channel and two chroma channels, and the subset may be the two chroma channels.
(A9) In any of the methods denoted as (A1) through (A8), the step of optimizing the backward reshaping function may include applying the backward reshaping function to the reshaped SDR image as generated by the optimized forward reshaping function.
(A10) In any of the methods denoted as (A1) through (A9), the steps of optimizing the forward reshaping function and applying a correction to the input SDR image may utilize a local copy of the input SDR image, and the method may further include maintaining an uncorrected version of the input SDR image.
(A11) Any of the methods denoted as (A1) through (A10) may include performing the steps of optimizing the forward reshaping function and optimizing the backward reshaping function for each of a plurality of color-component channels.
(A12) In the method denoted as (A11), the color-component channels may include a luma channel, a first chroma channel, and a second chroma channel, wherein (a) for the luma channel, the method optimizes the forward and backward reshaping functions in only a single iteration of the steps of optimizing the forward and backward reshaping functions, and (b) for each of the first and second chroma channels, the steps of optimizing the forward and backward reshaping functions utilize multivariate multiple regression.
(A13) A method for encoding an HDR image may include performing any of the methods denoted as (A1) through (A12) to generate an iteratively optimized forward reshaping function and an iteratively optimized backward reshaping function, after receiving the input HDR image and the input SDR image. The method may further include outputting (i) the reshaped SDR image generated in a last iteration of the step of optimizing the forward reshaping function and (ii) parameters characterizing the backward reshaping function as optimized in a last iteration of the step of optimizing the backward reshaping function.
(B1) One software product, for generating (a) a forward reshaping function for compressing an input high-dynamic range (HDR) image into a reshaped standard-dynamic-range (SDR) image and (b) a backward reshaping function for decompressing the reshaped SDR image into a reconstructed HDR image, includes non-transitory, computer-readable media storing machine-readable instructions. The instructions include (i) forward optimization instructions that, when executed by a processor, cause the processor to optimize the forward reshaping function to minimize deviation between the reshaped SDR image and an input SDR image corresponding to the input HDR image, (ii) backward optimization instructions that, when executed by the processor, cause the processor to optimize the backward reshaping function to minimize deviation between the reconstructed HDR image and the input HDR image, (iii) SDR-image correction instructions that, when executed by the processor, cause the processor to apply a correction to the input SDR image, and (iv) iteration control instructions that, when executed by the processor, cause the processor to evaluate if a termination condition has been met and, as long as the termination condition has not been met, execute the SDR image correction instructions and re-execute the forward optimization instructions and the backward optimization instructions based on the input SDR image as corrected.
(B2) In the software product denoted as (B1), the SDR-image correction instructions may be configured to, when executed by the processor, cause the processor to apply, to each pixel of each of one or more color-component channels of the input SDR image, a deviation-based correction value that is proportional to a deviation between the reconstructed HDR image and the input HDR image for a corresponding pixel and a corresponding color-component channel.
(B3) In either of the software products denoted as (B1) and (B2), the termination condition may include a threshold amount, and the iteration control instructions may be configured to, when executed by the processor, cause the processor to evaluate if a deviation measure at least between the reconstructed HDR image and the input HDR image is less than the threshold amount.
(B4) In the software product denoted as (B3), the termination condition may further include a maximum iteration number, and the iteration control instructions may be configured to, when executed by the processor, cause the processor to, if the maximum iteration number has been reached, (a) stop re-execution of the SDR-image correction instructions, the forward optimization instructions, and the backward optimization instructions, and (b) output parameters characterizing the backward reshaping function and output the reshaped SDR image.
(B5) In either of the software products denoted as (B3) and (B4), the deviation measure may be a weighted combination of (i) the deviation between the reconstructed HDR image and the input HDR image and (ii) a deviation between the reshaped SDR image and the input SDR image.
(B6) An encoder may include any of the software products denoted as (B1) through (B5), a processor for executing the machine-readable instructions of the software product, an input interface for receiving the input HDR image and the input SDR image, and an output interface for outputting (i) the reshaped SDR image generated using the forward reshaping function as optimized, and (ii) parameters characterizing the backward reshaping function as optimized.
Changes may be made in the above systems, products, and methods without departing from the scope hereof. It should thus be noted that the matter contained in the above description and shown in the accompanying drawings should be interpreted as illustrative and not in a limiting sense. The following claims are intended to cover generic and specific features described herein, as well as all statements of the scope of the present systems, products, and methods, which, as a matter of language, might be said to fall therebetween.
Number | Date | Country | Kind |
---|---|---|---|
20170837 | Apr 2020 | EP | regional |
This application claims priority to U.S. Provisional Application No. 63/013,807 and European Patent Application No. 20170837.7, both filed on Apr. 22, 2020, each of which is incorporated by reference in its entirety.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/US2021/028475 | 4/21/2021 | WO |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2021/216767 | 10/28/2021 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
9648351 | Su | May 2017 | B2 |
9716887 | Atkins | Jul 2017 | B2 |
10080026 | Su | Sep 2018 | B2 |
10223774 | Kadu | Mar 2019 | B2 |
10244244 | Piramanayagam | Mar 2019 | B2 |
10264287 | Wen | Apr 2019 | B2 |
11257195 | Andrivon | Feb 2022 | B2 |
20140029675 | Su | Jan 2014 | A1 |
20170308996 | Kadu | Oct 2017 | A1 |
20180007356 | Kadu | Jan 2018 | A1 |
20180242006 | Kerofsky | Aug 2018 | A1 |
20180278967 | Kerofsky | Sep 2018 | A1 |
20180309995 | He | Oct 2018 | A1 |
20190110054 | Su | Apr 2019 | A1 |
20190164262 | Cellier | May 2019 | A1 |
20190208173 | Kadu | Jul 2019 | A1 |
20190222866 | Song | Jul 2019 | A1 |
20190281325 | Song | Sep 2019 | A1 |
20190349607 | Kadu | Nov 2019 | A1 |
20210183028 | Cellier | Jun 2021 | A1 |
20220408081 | Su | Dec 2022 | A1 |
Number | Date | Country |
---|---|---|
20190086053 | Jul 2019 | KR |
2666234 | Sep 2018 | RU |
2687267 | Jan 2019 | RU |
2014204865 | Dec 2014 | WO |
2016153896 | Sep 2016 | WO |
2017024042 | Feb 2017 | WO |
2017157977 | Sep 2017 | WO |
WO-2018202744 | Nov 2018 | WO |
2018231968 | Dec 2018 | WO |
2019160986 | Aug 2019 | WO |
2019217751 | Nov 2019 | WO |
Entry |
---|
Methods for conversion of high dynamic range content to standard dynamic range content and vice-versa, BT Series Broadcasting service (television), Report ITU-R BT.2446-0, Apr. 2019, 33 pages. |
Number | Date | Country | |
---|---|---|---|
20230164366 A1 | May 2023 | US |
Number | Date | Country | |
---|---|---|---|
63013807 | Apr 2020 | US |