Patent Application
20240281941
In general, the invention concerns the field of high dynamic range (HDR) imaging. More particularly, the invention is related to a method for tone mapping a high dynamic range (HDR) input image into a low dynamic range (LDR) output image.
Moreover, the invention is related to a computer program having program code means adapted to perform such a method.
Furthermore, the invention is related to an electronic device adapted to perform such a method.
HDR images typically offer a dynamic range of 14-16 bits to 20-24 bits, while traditional LDR images (or standard dynamic range images) typically only offer a dynamic range of 8-10 bits. Therefore, HDR images allow preserving the details of real-world scenes that contain very bright, but also very dark areas much better than conventional LDR images. Luminance is a photometric quantity that is used in photo and video technology to measure the brightness of pixels. HDR images allow capturing dynamic ranges similar to the human eye, which can detect a luminance range of up to approximately 1014.
However, common displays are only able to reproduce the luminance range of conventional LDR images. Currently, there are no displays available that can reproduce the luminance range of HDR images, but only very few expensive displays are able to reproduce a luminance range of approximately 12 bits.
Due to these limitations of display contrast, the luminance range of input HDR images has to be reduced in order to be displayed on a regular display. This process of converting an HDR image to an LDR image is commonly referred to as tone mapping.
In the context of this application, which is related to a method for tone mapping a high dynamic range input image into a low dynamic range output image, the term “low dynamic range output image” can generally refer to any image with a dynamic range that is smaller than the dynamic range of the high dynamic range input image. In particular, in certain embodiments, the low dynamic range can be a conventional low dynamic range image, e. g. with a dynamic range of 8 to 10 bits and/or a standard dynamic range (SDR) image.
The input image and the output image can be part of an input video sequence and an output video sequence, respectively.
The objectives of tone mapping methods can be different depending on the particular application. For example, different criteria have to be met to produce images matching the Human Visual System (HVS) on the one hand and for Machine Vision System (MVS) applications on the other hand.
With regard to the HVS, for example, in order to display HDR images on a standard display with a displayable dynamic range of 8 to 10 bits, the HDR images, which may comprise a luminance range of 20 to 24 bits, have to be compressed in a way that all dark, mid-tone and bright image contents are clearly distinguishable by the naked human eye. Moreover, at the same time, the reproduction of the image on the display shall look as realistic and natural as possible to the human observer.
On the other hand, with regard to MVSs, tone mapping methods that compress the HDR image (with a luminance range of, for example, 20 to 24 bits) to an LDR image (with a luminance range of, for example, 8 to 10 bits) should allow the MVS to process the tone-mapped image as effectively as possible. In most cases, this means that all structures (edges, textures) in the image have to be preserved as good as possible.
In general, a common tone mapping objective in both HVS and MVS applications is to minimize loss of image information. For this purpose, it is essential to preserve contrast, in particular local contrast, appropriately with regard to the intended application.
From the state of the art, a variety of different tone mapping operators is known that have been developed to meet these challenges. They can be divided into two main types: global tone mapping operators and local tone mapping operators.
On the one hand, global tone mapping operators (or global tone mapping methods) map every pixel of the image in the same way, independent of its position or surrounding pixels in the image. In general, global tone mapping operators are non-linear functions (also called tone mapping curves) that are based on a global luminance and/or other global properties of the particular image. As the same function is applied to every pixel of the particular image, global tone mapping methods are simple and fast and require only a small amount of processing power. Therefore, they can be implemented, for example based on look-up tables, using low-cost hardware. However, these global methods often cause a significant loss of contrast, in particular with regard to local image details, as only global properties of the image are considered.
On the other hand, local tone mapping operators (or local tone mapping methods) map each pixel of the image differently, in particular depending on local image properties, for example depending on surrounding pixels. When designed correctly, these methods can achieve very good results in terms of preserving local contrast as it is beneficial with regard to the intended application. However, these local tone mapping methods are prone to artifacts like halo effect and ringing and their output may appear unrealistic to the human observer. Moreover, these methods are generally more complicated than global tone mapping methods and hence require significantly more processing power in most cases.
From US 2018/0097992 A1, for example, a global tone mapping method and a corresponding system are known that involve computing a first histogram for luminance values of the input image, accessing a target histogram for the image, and computing a transfer function based on the first histogram and the target histogram. The tone-mapped image is produced by applying the transfer function to the pixel values of the input image.
The present invention provides a method for tone mapping of HDR input images into LDR output images that offers an improved preservation of the image's details and contrast compared to conventional global tone mapping methods, but requires a smaller amount of processing power than complicated local tone mapping methods.
According to the invention, the method comprises at least the following steps:
The steps of the method do not have to be executed in the specified order and the invention is not limited accordingly, i. e. the alphabetic order of the letters does not imply a specific sequence of steps a) to f). For example, as matter of course, step b) could be executed after steps c) and d), or some of the method's steps could be executed in parallel.
In step a), a luminance component of the input image is obtained. Such a luminance component can be, for example, the luma component (luma channel) of a color space in which luminance intensity values and color tone values are separated, for example the luma component Y of the YCbCr color space (or Y′ of the Y′CbCr color space). If the luminance component is a luma component of a color space in which luminance intensity values and color tone values are separated, as the luma component Y of the YCbCr color space, it can be necessary to transform the input HDR image to such a color space first. For example, if the input image is an RGB image, the RGB input image could be transformed to the YCbCr color space first.
However, the luminance component can generally also be any other component or combination of components of an arbitrary color space that represents the luminance of the image. For example, the luminance component could also be the green component G of the RGB color space, as it can represent the image's luminance sufficiently well. As a matter of course, the luminance component comprises a plurality of luminance values. In particular, the luminance component of the input image can comprise one luminance value for each pixel of the input image.
The terms “dynamic range” and “luminance range” are used equivalently in the context of this application.
In step b), an initial tone mapping curve is obtained, which is a global tone mapping curve. In other words, in step b), a global tone mapping function (or global tone mapping operator) is obtained that maps the input luminance values of an HDR input image to output luminance values of an LDR output image.
In an advantageous embodiment of the invention, the initial tone mapping curve and/or the adapted tone mapping curve can be implemented as a look-up table, respectively. Such embodiments of the invention provide the advantage that the tone mapping method can be implemented in a computationally efficient manner.
Moreover, in another advantageous embodiment of the invention, the initial tone mapping curve and/or the adapted tone mapping curve can be constructed by subsampling the HDR input image and interpolating between the subsampled values. As the dynamic range of the HDR input image can be very large, such embodiments of the invention provide the advantage that the required processing power can be further reduced.
In step c), a luminance histogram is obtained that represents the luminance distribution of the input image. In general, this luminance histogram can be obtained from the original HDR input image. However, as will be explained in greater detail below, the luminance histogram can also be obtained from a compressed version of the input image in order to reduce the processing power that is required for step c).
In step d), a plurality of clusters of the luminance distribution is determined from the luminance histogram. The number of clusters to be created in step d) can be a predetermined number. In particular, the number of clusters may be greater than 2 and smaller than 12. In particular, the number of clusters may be greater than 2 and smaller than 6. In particular, the number of clusters may be greater than 2 and smaller than 5. The number of clusters, however, can also be not predetermined. For example, the number of clusters can be determined by the applied clustering algorithm.
In step e), an adapted tone mapping curve is generated from the initial tone mapping curve. To this end, for each of the plurality of clusters that have been determined in step d), the slope of the initial tone mapping curve is adapted depending on a concentration of luminance values in that cluster, wherein a higher concentration of luminance values in the respective cluster results in a greater slope of the adapted tone mapping curve for that cluster. The concentration of luminance values in the respective cluster can be determined based on different metrics, including an amount of variation of the luminance values in the cluster and/or an absolute or relative number of luminance values in that cluster, which will be explained in more detail below.
In step f), the output image is generated by applying the adapted tone mapping curve to at least the luminance component of the input image.
The method according to the invention effectively avoids loss of relevant image information, e. g. object contours and textures. For this purpose, it is essential to preserve local contrast, which also comprises textural structures of the image. The invention is based on the insight that in order to avoid loss of local contrast—and hence loss of local image information—as far as possible, it is important that the tone mapping curve is sufficiently steep, i. e. has a sufficiently large slope, for those parts of the luminance range (HDR) that are dominant in the individual HDR input image. These parts of the luminance range are also referred to as dominant tonal range. The inventors have found that these dominant parts of the luminance range can be determined based on the concentration of luminance values from the luminance histogram. The reasons for this are twofold. On the one hand, if the luminance histogram shows a large concentration of luminance values in a certain part of the luminance range, this means that compressing the dynamic range in these parts would affect particularly large areas of the input image. On the other hand, if the luminance histogram locally indicates a large concentration of luminance values, this indicates that in these parts of the luminance range, the input image contains areas with low local contrast, where the luminance values of a pixel and its surroundings are particularly close to each other. These areas of the input image are particularly sensitive to compression.
In order to identify the different parts of the luminance range that are dominant in the input image, a clustering method is applied to determine a plurality of clusters of the luminance distribution from the luminance histogram. Known clustering algorithms, for example k-means clustering, can be used for this purpose. A large concentration of luminance values in that cluster indicates that the corresponding part of the luminance range is dominant in the input image. By increasing the slope of the tone mapping curve for those parts of the luminance range with a large concentration of luminance values in the input image, the input image's dominant tonal range is compressed to a lesser extent and hence contrast can be preserved. In other words, those parts of the luminance range that carry a particularly large amount of information in the input image are preserved from being compressed too heavily.
As a result, the invention effectively avoids loss of relevant image information, e. g. object contours, textures and details, by preserving contrast in the image's dominant tonal range, which is determined based on the concentration of luminance values for each cluster of the luminance distribution.
This effect is achieved by means of global tone mapping, as the adapted tone mapping curve-like the initial tone mapping curve—is a global tone mapping curve. Compared to local tone mapping methods, this results in significantly lower processing power requirements. Therefore, the invention can be advantageously be implemented using low-cost hardware.
According to an advantageous embodiment of the invention, it is proposed that step c) comprises
Such embodiments of the invention provide the advantage that the processing power necessary for processing the histogram and determining the clusters of luminance values in step d) can be greatly reduced.
In general, the luminance histogram can also be directly obtained from the HDR input image. However, due to the enormous size of the HDR tonal range (which may use 20 bits, 24 bits or even more bits), a large amount of processing power would be necessary for processing the histogram—in particular for determining the clusters of luminance values from the histogram in step d). Such processing power might not be available in common image processing pipelines and it may be undesirable or even impossible to equip image pipelines with such processing power.
Nevertheless, the scope of the application also comprises obtaining the luminance histogram from the uncompressed input image. The scope of the application hence also comprises obtaining the luminance histogram directly from the HDR input image.
According to another advantageous embodiment of the invention, it is proposed that the input image is compressed by applying a global tone mapping curve. In particular, the input image can be compressed by applying the initial tone mapping curve. In this case, the same tone mapping curve is used to compress the input image that is also used as a basis for generating the adapted tone mapping curve in step e).
Such embodiments of the invention provide the advantage that the input image can be compressed in a particularly efficient and computationally inexpensive manner, as the global tone mapping curve can be implemented, for example, as a look-up table. As a result, the required processing power can be reduced.
According to another advantageous embodiment of the invention, it is proposed that the initial tone mapping curve is based on the global tone mapping operator by Reinhard.
The global tone mapping operator by Reinhard (also referred to as Reinhard's tone mapping operator) is given by the equation
L
out(x,y)=1+Lin(x,y)(1+Lin(x,y)/Lwhite2)/1+Lin(x,y),
wherein Lout is the displayable output luminance for pixel (x,y), Lin(x,y) is the (scaled) input luminance and Lwhite is the smallest luminance that will be mapped to pure white. This global tone mapping operator has been presented in the paper “Photographic Tone Reproduction for Digital Images” by E. Reinhard, J. Ferwerda and P. Shirley in ACM Transactions on Graphics in May 2002. The term “Reinhard curve” refers to a global tone mapping curve that is constructed by applying Reinhard's global tone mapping operator.
According to another advantageous embodiment of the invention, it proposed that the initial tone mapping curve is a Reinhard curve.
Reinhard's global tone mapping operator is comparably simple and yet, for a conventional global tone mapping operator, provides good results with regard to the output image's quality. Therefore, such embodiments of the invention that use Reinhard's global tone mapping operator for the initial tone mapping operator, which forms the basis for the adapted tone mapping curve that is used to generate the output image, provide the advantage that they are simple to implement and at the same time result in a particularly good quality of the output image, especially in terms of preserving image contrast and relevant details.
According to another advantageous embodiment of the invention, it is proposed that in step d), the plurality of clusters is determined by k-means clustering.
Such embodiments of the invention provide the advantage that computationally efficient implementations of the well-known k-means clustering algorithms can be used to effectively determine a number of clusters of the input image's luminance distribution from the luminance histogram.
According to another advantageous embodiment of the invention, step d) comprises merging two or more clusters depending on the distance between these clusters and/or the centroids of these clusters, in particular if the distance is lower than a threshold.
In such embodiments, the merging of two or more clusters results in reducing the number of clusters if the clusters are too close to each other. Such embodiments of the invention hence provide the advantage that the number of clusters can be adapted to the individual input image and its luminance distribution. As a result, an inappropriately large number of clusters can be avoided.
According to another advantageous embodiment of the invention, the concentration of luminance values in the cluster is determined at least based on
As explained above, the invention is based on the finding that in order to preserve important image details and local contrast, a sufficiently large slope of the tone mapping curve has to be ensured for those parts of the input image's luminance histogram that show a high concentration of luminance values.
For this purpose, the concentration of luminance values can be determined, for example, based on an amount of variation of the luminance values in the respective cluster. The concentration can also be determined based on the dispersion (also referred to as variability, scatter or spread) of the luminance values in that cluster, as this reflects the amount of variation. The amount of variation and/or the dispersion of the luminance values can be quantified, for example, using common measures of statistical dispersion. Examples for such measures include standard deviation, variance and coefficient of variation.
The amount of variation and/or the dispersion of the luminance values can be estimated, e. g. by empirically determining typical amounts of variations from a plurality of images beforehand. In particular, different amount of variations can be estimated for different cluster centroids, i. e. for different positions of clusters within the luminance rage. Alternatively or additionally, the amount of variation and/or the dispersion can also be determined analytically from the input image (or from a compressed version of the input image, as explained above).
Alternatively or additionally, the concentration of luminance values can be determined based on the number of luminance values in the respective cluster. In particular, the concentration of luminance values can be determined based on the number of luminance values in that cluster in relation to the total number of luminance values in the luminance histogram. In other words, the concentration of luminance values can be determined based on the ratio of luminance values in the respective cluster.
Such embodiments of the invention provide the advantage that the concentration of luminance values can be reliably and efficiently determined and hence the slope of the adapted tone mapping curve can be set appropriately in step e).
According to another advantageous embodiment of the invention, it is proposed that in step e), for each cluster, the slope of the initial tone mapping curve is adapted within an adaptation area depending on the concentration of luminance values in that adaptation area, wherein the adaptation area includes a range of luminance values around the centroid of the cluster.
In other words, it is proposed that the concentration of luminance values is determined and the slope of the tone mapping curve is adapted in an area around the centroid of the cluster.
According to another advantageous embodiment of the invention, it is proposed that the size and/or the position of the adaptation area is limited by a lower boundary and an upper boundary both representing luminance values, wherein the lower boundary and/or the upper boundary are determined depending on
The aforementioned embodiments of the invention, which rely on determining the concentration of luminance values and adapting the slope of the tone mapping curve in an adaptation area around the centroid of the cluster, provide the advantage that the tone mapping curve can be adapted specifically for those parts of the input image's luminance range that represent the dominant tonal range, which can be expected to be located around the cluster centroids. As a result, a particularly effective preservation of image contrast can be achieved.
According to another advantageous embodiment of the invention, it is proposed that step e) comprises determining a plurality of luminance zones, wherein each luminance zone covers a fraction of the luminance range, and adapting the slope of the initial tone mapping curve by adapting its output values for each luminance zone.
Such embodiments of the invention provide the advantage that they simplify the handling of the input image's luminance range, which can be very large, and yet allow adapting the slope of the tone mapping curve in a flexible manner, as the luminance zones can be defined depending on individual characteristics of the input image.
According to another advantageous embodiment of the invention, it is proposed that step a) comprises obtaining a luminance component and a number of chrominance components of the input image and step f) comprises applying the adapted tone mapping curve to the luminance component and the number of chrominance components.
In other words, it is proposed that the method according to the invention is not only applied to the luminance component, but also to at least one chrominance component of the input image. In particular, the number of chrominance components can comprise all chrominance components of the input image.
Such embodiments of the invention provide the advantage that the above-mentioned effects, in particular preserving important image details and local contrast, can be used for the tone mapping of the input image's color information. As the same adapted tone mapping curve can be used for the chrominance components that is also used for the luminance component, this allows processing all components of the input image very efficiently.
According to another advantageous embodiment of the invention, it is proposed that step f) additionally comprises desaturating the resulting chrominance components of the output image, in particular by applying a desaturation scaling factor to the chrominance components.
In general, tone mapping of colored HDR input images into LDR output images can have the undesired effect that certain areas of the output image may appear oversaturated, which may particularly apply to bright areas of the input image. To compensate this, the output image's chrominance components can be desaturated, for example, by applying a desaturation scaling factor that can be empirically determined. For example, the scaling factor can be implemented by means of a look-up table.
Therefore, such embodiments of the invention provide the advantage that an undesired oversaturation of the output image's chrominance components can be compensated.
According to another advantageous embodiment, it is proposed that step e) comprises, after adapting the slope of the initial tone mapping curve, smoothing the resulting adapted tone mapping curve, in particular by applying Bernstein polynomials and/or Bernstein-Bézier polynomials and/or Bézier curves.
Such embodiments of the invention provide the advantage that a smooth adapted tone mapping curve can be generated and applied to the output image.
The object of the invention is further achieved by a computer program having program code means adapted to perform a method as described above when the computer program is executed on a computer.
The object of the invention is further achieved by an electronic device that is adapted to perform a method as described above.
The electronic device can be, for example, a stand-alone integrated circuit (IC) or a part thereof. The electronic device can also be a system on a chip (SoC) or a part thereof. The electronic device can also be a part of an image processing pipeline and/or an image processing chain. The electronic device can also be a camera or a display or a part of a camera or a display. The electronic device can also be a system that comprises a camera and/or a display. The electronic device can also be a part of such a system.
In the following, the invention will be explained in more detail using the exemplary embodiments schematically shown in the attached drawings. The drawings show the following:
In steps 101 and 102, which correspond to step a) explained above, a luminance component and a number of chrominance components—in this exemplary embodiment two chrominance components—of the input image are obtained. For this purpose, in step 101, a HDR input image with an exemplary dynamic range of 16 bits is read, which initially has an RGB format. In the exemplary embodiment shown in
In step 103 shown in
wherein Lout is the displayable output luminance for pixel (x,y), Lin(x,y) is the (scaled) input luminance and Lwhite is the smallest luminance that will be mapped to pure white.
The following steps 104 and 105 shown in
Afterwards, in step 105, a luminance histogram that represents the luminance distribution of the input image is obtained from the compressed input image.
Referring to
In this exemplary embodiment, the k-means clustering algorithm is initially started with a predetermined number of four clusters. However, an additional constraint has been defined to prevent determining clusters that are too close to each other. For this purpose, in each iteration of the algorithm, it is checked whether the distance between any pair of centroids of the clusters is smaller than a predetermined threshold. In this case, it is assumed that the two clusters (or their centroids, respectively) are too close to each other and the two clusters are merged. For this purpose, the new centroid that results from the merging is calculated as follows:
wherein cij is the centroid of the merged cluster, ci and cj are the centroids of the clusters i and j, respectively, which are merged as they are too close to each other, ni and nj are the numbers of luminance values (pixels) in clusters i and j, respectively.
As a result, in the exemplary embodiment described here, the number of clusters that are determined from the luminance histogram is reduced from the initial number of four clusters to a final number of three clusters, as otherwise two of the cluster centroids would have been too close to each other. Therefore, after completion of step 106, three clusters of the input image's luminance distribution have been determined that are defined by the cluster centroids.
In step 107 of the exemplary embodiment shown in
In this exemplary embodiment, the lower boundary and the upper boundary are determined depending on the centroid of the respective cluster (the centroid's luminance value) and an estimated amount of variation of the luminance values in the respective cluster. For this purpose, the following closeness estimation metric is defined for each luminance value Y and each cluster i:
wherein Y is a luminance value from the input image's luminance histogram, ci is the centroid of cluster i, n is the number of clusters and σi2 is the estimated variation of the luminance values in cluster i, which serves as a metric for the estimated amount of variation. In this exemplary embodiment, Y is a luminance value of the compressed input image as explained above. In alternative embodiments, however, the original uncompressed HDR input image can also be used for this purpose. Those luminance values for which the closeness estimation metric fclose is greater than a predefined threshold are included in the adaptation area. In particular, this threshold can be defined depending on the number of clusters. For example, the threshold can defined as 0.5 for a number of clusters greater than three and the threshold can be defined as 0.1 if the number of clusters is three and the threshold can be defined as 0.001 for a number of clusters smaller than three.
In alternative embodiments, the lower boundary and the upper boundary can be determined depending on the centroid of the respective cluster (the centroid's luminance value), an estimated or real amount of variation of the luminance values in the respective cluster and a distance between the centroids of the respective cluster and a neighboring cluster. For example, the upper and the lower boundary of the cluster's adaptation area could also be determined based on the following equations:
wherein bu,i and bl,i are the upper boundary and the lower boundary for cluster i, respectively, ci is the cluster centroid, σi is the estimated standard deviation of the luminance values in cluster i, and di is the distance between the centroids of cluster i and a neighboring cluster.
The result of steps 101 to 107 is schematically illustrated in
Referring to
In other embodiments, the concentration of luminance values can additionally or alternatively be determined, as described above already, based on an amount of variation of the luminance values in that cluster, in particular based on an amount of variation of the luminance values in the cluster's adaptation area.
Afterwards, in step 109 shown in
In step 110 shown in
As can be seen in
This increase of the tone mapping curve's slope around the cluster centroids c1, c2, c3, i. e. in those parts of the luminance range where the input image's concentration of luminance values is particularly high, results in an improved quality of the generated LDR output image, as local contrast and relevant image details of the input image can be effectively preserved.
Referring to
Finally, in step 112, the resulting chrominance components of the output image are desaturated by applying a desaturation scaling factor to the chrominance components.
Steps 111 and 112 shown in
Furthermore, the image processing system 211 shown in
The HDR video camera 201 generates a sequence of HDR input images. The HDR video camera 201 is directly or indirectly connected with the image processing unit 203, which allows the image processing unit 203 to read the HDR input images generated by video camera 201. Each HDR input image read by the image processing unit 203 can be stored in memory 207.
The image processing unit 203 is adapted to perform the method as described above for tone mapping the HDR input image into a LDR output image.
After tone mapping the HDR input image into an LDR output image, i. e. after generating the LDR output image as previously explained, the generated LDR output image can be transmitted to the display unit 209, where the output image can be displayed.
This procedure can be repeated for each HDR input image of the HDR video sequence generated by the HDR video camera 201. This results in a generation of a LDR output video sequence, which is a sequence of LDR output images. The LDR output video sequence can be displayed on display unit 209.
Additionally or alternatively, the output image and/or the output video sequence can be stored in a memory and/or stored on a data storage unit and/or can be transmitted via a data transmission link.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/073182 | 8/20/2021 | WO |
