This application claims priority from European Patent Application No. 18305091.3, entitled “METHOD AND APPARATUS FOR GENERATING HDR IMAGES WITH REDUCED CLIPPED AREAS”, filed on Jan. 31, 2018, the contents of which are hereby incorporated by reference in its entirety.
The present invention relates generally to the field of high dynamic range imaging (HDR) and addresses the way of expanding the dynamic range of low or standard dynamic range images. More particularly, the invention relates to a method and an apparatus for generating HDR images with reduced clipped areas.
Recent advancements in display technology are beginning to allow for an extended dynamic range of color, luminance and contrast in images to be displayed. The term “images” refers to an image content that can be for example a video or a still picture.
Technologies allowing for an extended dynamic range in luminance or brightness of images are known as high dynamic range imaging, or HDR imaging. A number of display devices having the capability of processing and displaying HDR images with an extended dynamic range are already available for customers. Image capturing devices capable of capturing images with such an increased dynamic range are also being developed. However, HDR images are not yet well widespread and there exists many existing images that are SDR (for Standard Dynamic Range) or LDR (for Low Dynamic Range) images. Therefore, there is a need to visualize low or standard dynamic range (LDR or SDR) images on the recent high dynamic range (HDR) devices.
For this purpose, reverse or inverse tone mapping (ITM) algorithms have been developed. They allow to generate HDR images from conventional (LDR or SDR) images, by using algorithms that process the luminance information of pixels in the images with the aim of recovering or recreating the appearance of the original scene.
Most of the time, the ITM consists in defining a global transfer function which, applied to the LDR input images, increases the dynamic and contrast and generates HDR output images.
A simple ITM could be to apply to the LDR (or SDR) video signal a gamma curve with a constant coefficient, for example equal to 2.4. Some of the other algorithms are more complex and propose to adapt the curve locally and temporally depending on picture characteristics.
Another known solution proposed combines a dynamic expansion while considering local characteristics of the images to avoid noise expansion. The expansion curve is adapted to the content itself meaning that the shape of the curve depends on LDR video features, like a histogram of the grey level of the LDR images for instance, and could be refreshed for each frame.
The transfer function results very often from a trade-off between contrast improvement and the fact to preserve the low and mid-tones of the LDR images and consequently preserve the artistic intent (intent of the colorist or the film director). This trade-off leads sometimes to clipped areas in the bright parts of the images while increasing the contrast. The clipped areas are areas of the image close to the maximum encodable luminance where the detail level is low. Clipped areas can be present in the LDR image and in the HDR image. The clipped areas in the HDR image can correspond to clipped areas already present in the LDR image or clipped areas generated by the inverse tone mapping process. In the present patent application, clipped areas designate more specifically bright areas wherein the detail level is low.
The present disclosure proposes to reduce the clipped areas in the HDR image, and more specifically the clipped areas generated by the ITM process. This reduction of clipped areas is achieved by applying a post-processing to the HDR images obtained by ITM or by adapting the ITM process.
The present disclosure proposes a method for processing images, comprising:
receiving a low dynamic range image, called LDR image, and a high dynamic range image, called HDR image, of a scene, said HDR image being obtained by inverse tone mapping applied to the LDR image,
detecting bright areas in said LDR and HDR images,
determining bright areas of the HDR image, called new bright areas, that are non-present in the LDR image,
modifying the luminance of the new bright areas of the HDR image in order to reduce the number of clipped areas in said new bright areas.
According to this method, the variation of bright areas between the LDR image and the HDR image is determined and the new introduced bright areas are modified in order to limit the clipped areas in these new bright areas.
According to an embodiment, the bright areas in said LDR image and in said HDR image are detected by converting color values of the LDR image and the HDR image into luminance values and by thresholding the luminance values of the LDR image with a first threshold ThLDR and the luminance values of the HDR image with a second threshold ThHDR.
According to an embodiment, if the luminance values of the LDR image are encoded on n bits and the luminance values of the HDR image are encoded on m bits, with m and n being integers such that m>n, ThHDR≥2m−n×ThLDR.
According to another embodiment, ThHDR=f(ThLDR) where f is a transfer function of the inverse tone mapping applied to the LDR image to obtain the HDR image.
According to an embodiment, the new bright areas are determined by subtracting the bright areas of the LDR image from the bright areas of the HDR image.
According to an embodiment, the luminance of the new bright areas of the HDR image is modified by mixing luminance values of said new bright areas of the HDR image with luminance values of areas of the LDR image corresponding to said new bright areas of the HDR image.
According to an embodiment, the luminance of the new bright areas of the HDR image is modified as follows:
Y
HDRmodified
=α*Y
HDR+(1−α)*YLDR
wherein—α is a predetermined parameter;
YHDRmodified is the modified luminance of a pixel of the new bright areas of the HDR image,
YHDR is the luminance of the corresponding pixel of the HDR image, and
YLDR is the luminance of the corresponding pixel of the LDR image.
According to an embodiment, the parameter α is equal to:
According to an embodiment, the method further comprises obtaining said HDR image by inverse tone mapping applied to the LDR image and the luminance of the new bright areas of the HDR image is modified by modifying at least one parameter of the transfer function of the inverse tone mapping.
According to an embodiment, at least one parameter of the transfer function of the inverse tone mapping is modified until the number of pixels of the new bright areas is greater than a predetermined number N.
The present disclosure also relates to a device for processing images, said device being configured to:
receive a low dynamic range image, called LDR image, and a high dynamic range image, called HDR image, of a scene, said HDR image being obtained by inverse tone mapping applied to the LDR image,
detect bright areas in said LDR and HDR images,
determine bright areas of the HDR image, called new bright areas, that are non-present in the LDR image, and
modify the luminance of the new bright areas of the HDR image in order to reduce the number of clipped areas in said new bright areas.
According to an embodiment, the luminance of the new bright areas of the HDR image is modified by mixing luminance values of said new bright areas of the HDR image with luminance values of areas of the LDR image corresponding to said new bright areas of the HDR image.
According to an embodiment, the device is further configured to obtain said HDR image by inverse tone mapping applied to the LDR image and wherein the luminance of the new bright areas of the HDR image is modified by modifying at least one parameter of the transfer function of the inverse tone mapping.
The present disclosure also relates to a computer program product comprising program code instructions for implementing the above-defined method, when said program is executed on a computer or a processor
The present disclosure also relates to a non-transitory computer-readable storage medium storing the above-mentioned computer program product.
The invention can be better understood with reference to the following description and drawings, given by way of example and not limiting the scope of protection, and in which:
The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.
While example embodiments are capable of various modifications and alternative forms, embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit example embodiments to the particular forms disclosed, but on the contrary, example embodiments are to cover all modifications, equivalents, and alternatives falling within the scope of the claims. Like numbers refer to like elements throughout the description of the figures.
Before discussing example embodiments in more details, it is noted that some example embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations as sequential processes, many of the operations may be performed in parallel, concurrently or simultaneously. In addition, the order of operations may be re-arranged. The processes may be terminated when their operations are completed, but may also have additional steps not included in the figures. The processes may correspond to methods, functions, procedures, subroutines, subprograms, etc.
Methods discussed below, some of which are illustrated by the flow charts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium such as a storage medium. A processor(s) may perform the necessary tasks. Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments of the present invention. This invention may, however, be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.
In the following paragraphs, referring to the drawings, an implementation of the method and the related devices according to an embodiment will be described. A HDR image will designate an image having a higher dynamic range than a LDR or SDR image. The expression “LDR image” will be used to designate a LDR (Low Dynamic Range) image or a SDR (Standard Dynamic range) image. In a more general manner, in the present patent application, a LDR or SDR image designates an image having a dynamic range lower than the one of a HDR image. By contrast, a HDR image designates an image having a dynamic range higher than the one of a LDR or SDR image.
The method according to the invention comprises a plurality of steps and supposes that, for each frame of a video or each image, a LDR version and a HDR version are available, the HDR version resulting from an operation of ITM applied on the LDR version.
Compared to the LDR image (coded for example in 8 bits) of
The steps of the method according to a first embodiment of the invention are shown in
This Figure shows a preliminary step 10 of inverse tone mapping (ITM) for generating a HDR image (to be processed) from a LDR image. This step of ITM may be part of the method of the invention. That is the reason why this step is shown dashed lines.
The ITM process may be of any type. The transfer function of the ITM process is for example a simple gamma curve or a function using an expansion exponent map as disclosed in WO2015/096955. The LDR image may be the LDR image of
According to this embodiment, bright areas are detected in the LDR image and in the HDR image in steps 11 and 12 respectively. The detected bright areas of the HDR image are potential clipped areas.
Each of these two steps may be performed by segmentation. This segmentation can be done using a thresholding operation. Let's take the example of a LDR image in which the RGB values are encoded on 8 bits for each color. The step 11 may consist in first converting the RGB values of the LDR image into luminance values and then thresholding the luminance values with a first threshold ThLDR. Likewise, the step 12 may consist in first converting the RGB values of the HDR image into luminance values and then thresholding the luminance values with a second threshold ThHDR.
The conversions may be performed by the following by the following equation:
Y
LDR=0.299*RLDR+0.587*GLDR+0.114*BLDR [1]
Y
HDR=0.299*RHDR+0.587*GHDR+0.114*BHDR [2]
where
YLDR is the luminance value of a pixel of the LDR image,
RLDR is the value of the red component of said pixel of the LDR image,
GLDR is the value of the green component of said pixel of the LDR image,
BLDR is the value of the blue component of said pixel of the LDR image,
and
YHDR is the luminance value of a pixel of the HDR image,
RHDR is the value of the red component of said pixel of the HDR image,
GHDR is the value of the green component of said pixel of the HDR image,
BHDR is the value of the blue component of said pixel of the HDR image.
The thresholding operation is then applied to the luminance values resulting from the conversion operation. The results of the thresholding operation are illustrated by the masks MaskLDR and MaskHDR of
The masks Mask LDR and Mask HDR are obtained as follows:
If YLDR>ThLDR, MaskLDR=white else MaskLDR=black in
If YHDR>ThHDR, MaskHDR=white else MaskHDR=black in
The thresholds ThLDR and ThHDR are defined such that, if the luminance values YLDR are encoded on n bits and the luminance values YHDR are encoded on m bits, with m and n being integers and m>n, we have ThHDR≥2m−n×ThLDR.
The thresholds ThLDR and ThHDR may be constant (fixed) or may be dependent from the image content or from the transfer function of the ITM.
The threshold should be adapted to the number of encoding bits.
If the values YLDR are encoded on 8 bits (n=8), the threshold, ThLDR is for example equal to 230. As mentioned herein above, this value may depend from the type of image (landscape, indoor, . . . ). The threshold ThLDR could be closer to 255. But the threshold ThLDR is preferably not equal to 225 because, due to the inevitable presence of noise, clipped areas does not mean only pixels at 255.
If the values YHDR are encoded on 12 bits (m=12), the associated threshold ThHDR is for example equal to 230*(24)=3680 whatever the transfer function of the ITM is. The threshold ThLDR can be comprised between 3680 and 4095.
In another example, the threshold ThLDR may depend from the transfer function f of the ITM used for generating YHDR. For instance, ThHDR=f(230).
Referring again to
This step aims to separate the clipped areas already present in the LDR image and the new clipped areas that have been “added” by the ITM process in HDR. The underlying idea is to consider that the already clipped areas in the LDR image (advantageously graded by a colorist) can be kept as they are in the HDR image (no details is lost). At the opposite, the “new clipped areas” of the HDR image present in the MaskHDR-LDR (white areas of
Thus, in a step 14, the luminance values of the new bright areas (visible on the mask MaskHDR-LDR) are adapted to reduce the luminance levels in these areas and, as a consequence, reduce the number of clipped areas in these areas.
According to an embodiment, in step 14, the luminance of the new bright areas of the HDR image is modified by mixing luminance values of the new bright areas of the HDR image with luminance values of the corresponding areas in the LDR image.
For example, the luminance of the new bright areas of the HDR image is modified as follows:
and—YHDRmodified is the modified luminance of a pixel of the new bright areas of the HDR image,
YHDR is the luminance of the corresponding pixel of the HDR image, and
YLDR is the luminance of the corresponding pixel of the LDR image.
According to this formula, the closest the LDR pixel value is from the maximum luminance value (255 in 8 bits) the more the HDR value is mixed with the LDR one.
The equation [3] is applied in a linear domain. The luminance values are then converted in to RGB values. Of course, new corresponding RGB values are calculated to preserve the pixel color.
The results of this process, applied to the images of
The above described embodiment is a post-processing method which is applied after the ITM process.
In a variant, the ITM process is part of the inventive method and the ITM process is amended in order to reduce the clipped areas in the HDR image.
This variant is illustrated by
Unlike the previous embodiment, the HDR image is modified by modifying at least one parameter of the transfer function of the inverse tone mapping (step 10).
According to this variant, the number of pixels of new bright areas (=number of white pixels of the mask MaskHDR-LDR) is determined and compared to a predetermined number N in a step 15. The number N may be fixed or depending on the type of image (mindscape, indoor, . . . ). This number can possibly be equal to zero.
If the number of pixels of the new bright areas is greater than N, at least one parameter of the transfer function f of the ITM step 10 is modified in a step 16 in order to reduce the new bright areas.
In one example, the transfer function f can be of the type:
Y
HDR(p)=YLDR(p)E(p)
where—YHDR(p) is the luminance value of a pixel p in the HDR image;
The parameter of f to be modified may be a parameter of the quadratic function.
The transfer function of the ITM is modified until the number of pixels of the new bright areas is lower than or equal to N. When this condition is met, the new HDR image (delivered by the ITM process) is the modified HDR image HDRmodified. This modified image is delivered at step 17.
The device 800 comprises one or more processor(s) 810, which is(are), for example, a CPU, a GPU and/or a DSP (English acronym of Digital Signal Processor), along with internal memory 820 (e.g. RAM, ROM, EPROM). The device 800 comprises one or several Input/Output interface(s) 830 adapted to display output information and/or allow a user to enter commands and/or data (e.g. a keyboard, a mouse, a touchpad, a webcam); and a power source 840 which may be external to the device 800. The device 800 may also comprise network interface(s) (not shown). If the device implements the ITM process, the device can receive only LDR images. And if the device does not implement the ITM process, it receives LDR images and HDR images.
According to an exemplary and non-limitative embodiment of the invention, the device 800 further comprises a computer program stored in the memory 820. The computer program comprises instructions which, when executed by the device 800, in particular by the processor 810, make the device 800 carry out the method described with reference to
According to exemplary and non-limitative embodiments, the device 800 is a device, which belongs to a set comprising:
a mobile device;
a communication device;
a game device;
a tablet (or tablet computer);
a laptop;
a still image camera;
a video camera;
an encoding chip;
a still image server;
a video server (e.g. a broadcast server, a video-on-demand server or a web server);
a video uploading platform; and
a display or a decoding chip.
The implementations described herein may be implemented in, for example, a method or a process, a device, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). A device may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, a device such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.
Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications. Examples of such equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a camera, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices. As should be clear, the equipment may be mobile and even installed in a mobile vehicle.
Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions (and/or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD”), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory (“RAM”), or a read-only memory (“ROM”). The instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of the two. A processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.
As will be evident to one of skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax-values written by a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.
Number | Date | Country | Kind |
---|---|---|---|
18305091.3 | Jan 2018 | EP | regional |