Patent Application
20240233679
This application claims the benefit of Japanese Patent Application No. 2023-001914, filed Jan. 10, 2023, which is hereby incorporated by reference herein in its entirety.
The present invention relates to an image processing apparatus and an image processing method, and in particular to an image processing apparatus and an image processing method capable of handling high dynamic range (HDR) signals.
The Hybrid Log Gamma (HLG) and the Perceptual Quantization (PQ) have been standardized for video signals that support a wider dynamic range than conventional video signals. In the HLG standard, an opto-electronic transfer function (OETF) indicating the relationship between the display luminance and the video signal level, that is, a characteristic on the image capture apparatus side, is specified. On the other hand, in the PQ standard, an electro-optical transfer function (EOTF) indicating the relationship between the video signal level and the display luminance, that is, a characteristic on the display device side, is specified. Also, while luminance is treated as a relative value in the HLG standard, luminance is treated as an absolute value in the PQ standard.
Japanese Patent Laid-Open No. 2021-90109 discloses that the maximum luminance value of the output dynamic range (units: [nits] or [cd/m2]) or the tone value (signal level) corresponding to the maximum luminance value is recorded, as a parameter called “maxDRL”, in association with an HDR signal conforming to the PQ standard. Using the maxDRL makes it possible to appropriately map the luminance of an HDR signal conforming to the PQ standard (PQ HDR signal) to the luminance of an HDR signal conforming to the HLG standard (HLG HDR signal) or a standard dynamic range (SDR) signal.
An PQ HDR signal can be converted into an HLG HDR signal through a method such as that described in Report ITU-R BT.2408-5, “Guidance for operational practices in HDR television production”, March 2022, ITU-R, <https://www.itu.int/dms_pub/itu-r/opb/rep/R-REP-BT.2408-5-2022-PDF-E.pdf>. In this case, the maxDRL originally recorded for the PQ HDR signal is not applicable to a converted HLG HDR signal. However, no proposal has been made regarding how to handle an accompanied parameter other than the HDR signal when converting a PQ HDR signal into an HLG HDR signal.
In one aspect, the present invention provides an image processing apparatus and an image processing method that allow a parameter related to an PQ HDR signal to be applicable to an HLG HDR signal converted from the PQ HDR signal.
According to an aspect of the present invention, there is provided an image processing apparatus, comprising: one or more processors that execute a program stored in a memory and thereby function as: an acquisition unit configured to acquire a Perceptual Quantization (PQ) image comprised of a High Dynamic Range (HDR) signal conforming to Perceptual Quantization (PQ), and information relating to a maximum luminance value of the PQ image, which is associated with the PQ image; a conversion unit configured to convert the PQ image into a Hybrid Log Gamma (HLG) image comprised of an HDR signal conforming to Hybrid Log Gamma (HLG), and convert the information relating to the maximum luminance value in the same manner as in the conversion of the PQ image to the HLG image; and an association unit configured to associate the HLG image with the converted information relating to the maximum luminance value.
According to another aspect of the present invention, there is provided an image processing method performed by an image processing apparatus, comprising: acquiring a Perceptual Quantization (PQ) image comprised of a High Dynamic Range (HDR) signal conforming to Perceptual Quantization (PQ), and information relating to a maximum luminance value of the PQ image, which is associated with the PQ image; converting the PQ image into a Hybrid Log Gamma (HLG) image comprised of an HDR signal conforming to Hybrid Log Gamma (HLG); converting the information relating to the maximum luminance value in the same manner as in the conversion of the PQ image to the HLG image; and associating the HLG image with the converted information relating to the maximum luminance value.
According to a further aspect of the present invention, there is provided a non-transitory computer-readable medium that stores a program executable by a computer, which, when executed by the computer, causes the computer to function as an image processing apparatus, comprising: an acquisition unit configured to acquire a Perceptual Quantization (PQ) image comprised of a High Dynamic Range (HDR) signal conforming to Perceptual Quantization (PQ), and information relating to a maximum luminance value of the PQ image, which is associated with the PQ image; a conversion unit configured to convert the PQ image into a Hybrid Log Gamma (HLG) image comprised of an HDR signal conforming to Hybrid Log Gamma (HLG), and convert the information relating to the maximum luminance value in the same manner as in the conversion of the PQ image to the HLG image; and an association unit configured to associate the HLG image with the converted information relating to the maximum luminance value.
Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings).
Hereafter, embodiments will be described in detail with reference to the attached drawings. Note, the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but limitation is not made to an invention that requires all such features, and multiple such features may be combined as appropriate. Furthermore, in the attached drawings, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted.
Note that, in the following embodiments, a case will be described in which the present invention is implemented in a computer device (a personal computer, a tablet computer, a media player, a PDA, etc.) serving as an example of an image processing apparatus. However, the present invention can be implemented with any electronic device that can handle HDR signals. Such electronic devices include image capture apparatuses, smartphones, game consoles, display devices, robots, drones, and drive recorders. These are just examples, and the present invention can be implemented with other electronic devices as well.
The present invention is applicable to image data constituted by pixel data having a plurality of color components. For example, the image data may be image data generated by an image capture apparatus that uses an image sensor with a color filter. Note that a PQ HDR signal requires a depth of 10 bits or more per component. Accordingly, a data format that supports a depth of 10 bits or more is used.
Here, as an example, a data format compliant with the High Efficiency Image File Format (HEIF) standardized by ISO/IEC 23008-12 is used. In the HEIF, data for related images other than the main image and its thumbnail can be stored in one data file. For example, as related image data, it is possible to store still image or moving picture data with a depth of 10 bits, which is encoded using an encoding method compliant with the H.265 or High Efficiency Video Codec (HEVC) standard.
In the following, unless otherwise specified, it is assumed that an HEIF-format data file in which a PQ-format HDR signal is recorded as main image data and a maxDRL relating to the main image is recorded in metadata is handled.
The CPU 1 implements the operations of the image processing apparatus 10 described below by loading programs stored in the ROM 2 and the external storage device 4 into the RAM 3 and executing them. Note that although one CPU 1 is shown in
The ROM 2 is an electrically-rewritable nonvolatile memory. The ROM 2 stores control programs such as a BIOS, which is required for starting up the image processing apparatus 10, as well as programs, parameters, and data that do not require modification.
The RAM 3 has a work region for the CPU 1, a primary storage region for temporarily storing various types of data, a load region for various programs, a video memory region for the display unit 6, and the like.
The external storage device 4 stores basic software (OS), various control programs, various application programs executable on the OS, various types of data, and the like. The external storage device 4 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a storage device using removable media, or the like. The external storage device 4 may also be removable.
The external storage device 4 stores a plurality of application programs including, for example, the following application programs.
An application program for applying so-called developing processing to RAW-format image data in which pixels have one color component. An application program for converting the format of an HDR signal from the PQ format to the HLG format. The operation unit 5 is one or more input devices that can be operated by the user, such as a keyboard, a mouse, and a touch panel. The CPU 1 detects an operation on the operation unit 5 and executes processing according to the detected operation.
The display unit 6 is, for example, a liquid crystal display (LCD) or an organic EL display. The display unit 6 may also be an external device. The display unit 6 displays various types of information through the user interface of the OS and application programs running on the image processing apparatus 10.
The system bus 7 communicably connects each of the functional blocks described above.
Since the display luminance reflects the intention of the production, the display luminance is converted into scene luminance (Scene Light) by applying HLG OOTF−1 34. Thereafter, by applying the HLG OETF 35, an HLG image 36 is obtained. Note that the combination of the HLG OOTF−1 34 and the HLG OETF 35 corresponds to the HLG EOTF−1.
Report ITU-R BT.2408-5 describes γ=1.2, the maximum display light Lw=1000 [nit], and black display light LB=0 [nit], as standard parameter values. These parameter values are set in the HLG OOTF−1 34. With this conversion method, changes in luminance due to conversion can be avoided by matching the maximum luminance of the PQ image to the maximum luminance of the HLG image (1000 [nit] in this case).
However, the unconverted PQ image and the converted HLG image have different signal values corresponding to the maximum luminance. For example, if the maximum luminance of the PQ image is 650 [nit] and the signal value is 721, the corresponding signal value in the converted HLG image will be 955. Accordingly, if the signal level corresponding to the maximum luminance for the PQ image is recorded as maxDRL, it cannot be used for the converted HLG image.
Accordingly, in this embodiment, when converting a PQ image to an HLG image, the maxDRL for the PQ image is also converted to a value suitable for the HLG image.
Here, it is assumed that an HEIF-format PQ image is generated by an image capture apparatus, and the subject optical image is converted into a signal value according to the OETF of the image capture apparatus. It is also assumed that a parameter maxDRL regarding the maximum luminance of the PQ image is recorded as metadata in the data file in which the PQ image is stored.
Also, information on the transfer functions (OOTF, OETF, EOTF, and their inverse functions) conforming to the PQ standard and the HLG standard can be stored in the ROM 2 or the external storage device 4. Note that as long as PQ EOTF, PQ OOTF, HLG OETF, and HLG OOTF are stored, the other transfer functions can be obtained. For example, PQ OETF can be obtained from PQ OOTF and PQ EOTF−1.
In step S1, the CPU 1 acquires the PQ image 31 and loads it to the RAM 3. Although it is assumed here that the PQ image 31 is acquired from an HEIF-format data file stored in the external storage device 4, the PQ image 31 may also be acquired from an external device that can communicate through an external interface that the image processing apparatus 10 has.
In step S2, the CPU 1 acquires the maxDRL for the PQ image 31 from the data file in which the PQ image 31 is stored, and stores the acquired maxDRL in the RAM 3.
In step S3, the CPU 1 converts the signal values of the PQ image 31 into display luminance (Display Light) by applying the PQ EOTF 32 shown in
In step S4, the CPU 1 converts Display Light 33 to Scene Light by applying HLG OOTF−1 to the Display Light 33. Scene Light is a signal corresponding to the subject luminance. The OOTF−1 can be obtained by combining (multiplying) the EOTF−1 and the OETF−1, but may also be stored in the ROM 2 or the external storage device 4 in advance. Also, the system parameters (γ, LW, LB) used for the OOTF−1 may be those described above.
Then, the CPU 1 converts the Scene Light to the signal values of the HLG image 36 by applying the HLG OETF 35 as shown in
Note that in step S4, the Display Light 33 may be regarded as the Scene Light and the HLG OETF 35 may be applied without converting the Display Light 33 to the Scene Light using the OOTF−1. In this case, the HLG image 36 is obtained through the conversion procedure shown in
In steps S5 and S6, the CPU 1 converts the maxDRL of the PQ image to the maxDRL of the HLG image in the same manner as in steps S3 and S4. Specifically, in step S5, the CPU 1 applies the PQ EOTF 32 to the maxDRL of the PQ image acquired in step S2 to convert it to the value of the Display Light.
Then, in step S6, the CPU 1 applies the HLG OOTF−1 to the maxDRL converted to the Display Light value, to convert it to the Scene Light value. Furthermore, the CPU 1 applies the HLG OETF 35 to the maxDRL converted to the Scene Light value to convert it to the maxDRL of the HLG image 36. Here as well, the HLG OETF 35 may be applied directly to the maxDRL converted to the Display Light value to convert it to the maxDRL of the HLG image 36.
In step S7, the CPU 1 generates an HEIF-format data file in which the HLG image 36 obtained in step S4 is stored, and includes the maxDRL obtained in step S6 as metadata.
In step S8, the CPU 1 records the data file of the HLG image in the external storage device 4. Note that although the HLG image and the converted maxDRL are included and recorded in the same data file here, they do not need to be recorded. For example, the CPU 1 may store the HLG image and the converted maxDRL in association with each other in the RAM 3, without recording them in the external storage device 4.
Thus, in this embodiment, when converting a PQ image into an HLG image, the maxDRL is also converted in the same manner as the image and is associated with the HLG image. Accordingly, even if the maximum signal value available in the HLG image (1023 in the case of a 10-bit depth) does not correspond to the maximum luminance value, appropriate processing can be executed based on the maxDRL.
For example, a case is considered in which a PQ image having a histogram as shown in
In HLG images in which luminance values are handled in a relative manner, the saturated luminance level is usually equal to the maximum signal value corresponding to the bit depth (1023 in the case of a 10-bit depth). However, when a PQ image is converted into an HLG image, the signal value corresponding to the saturated luminance level of the HLG image becomes smaller than the maximum signal value according to the maximum luminance of the PQ image. According to the present embodiment, it is possible to associate the maxDRL corresponding to the converted HLG image, and therefore it is possible to correctly understand the signal level corresponding to the maximum luminance value of the HLG image based on the maxDRL. For this reason, as shown in
Note that the method of using the maxDRL for the converted HLG image is not limited to the display of a luminance histogram. For example, the method can be used for any processing that depends on the maximum luminance, such as the display of a highlight region of the HLG image.
According to this embodiment, when converting a PQ image into an HLG image, parameters recorded for the PQ image are converted so that they are applicable to the converted HLG image. For this reason, even if the signal value corresponding to the maximum luminance of the HLG image is not the maximum signal value, processing depending on the maximum luminance of the HLG image can be appropriately executed.
Next, a second embodiment of the present invention will be described. In the first embodiment, a case has been described in which the maximum luminance of the HLG image used for the HLG OOTF−1 is 1000 [nit] and the system gamma γ=1.2. This embodiment relates to a configuration in which the user can select these values.
When the configuration screen 101 is closed, the CPU 1 acquires the configuration value of the system gamma (or maximum luminance value) and stores it in the ROM 2, for example.
The PQ-HLG conversion processing in this embodiment will be described below with reference to the conversion model shown in
In step S11, the CPU 1 refers to the ROM 2 and acquires the system gamma configuration value (or maximum luminance value).
In step S12, the CPU 1 applies the PQ EOTF 102 to the PQ image 101 stored in the RAM 3, as in the first embodiment. Then, the CPU 1 applies the HLG OOTF−1 103. In this way, in this embodiment, the PQ image 101 is converted into the Display Light 105 by applying the PQ EOTF 102 and the HLG OOTF−1 103 to the PQ image 101.
Then, the system parameters 104 including the system gamma (or maximum display luminance LW) acquired in step S11 are applied to the HLG OOTF−1. Note that the black display luminance LB is set to 0 here as well.
Thereafter, the processing of step S4 and onward is the same as in the first embodiment. That is, as described with reference to
According to this embodiment, it is possible to execute PQ-HLG conversion with consideration given to the gamma of the display device that displays the converted HLG image, and it is possible to perform appropriate display according to the gamma of the display device that displays the HLG image. Also, as in the first embodiment, it is possible to realize the effect of obtaining an appropriate maxDRL for the converted HLG image.
Next, a third embodiment of the present invention will be described. In the second embodiment, PQ-HLG conversion was performed with consideration given to the configured system gamma. In this embodiment, an HLG image is generated for each of a plurality of predetermined system parameters.
However, in this embodiment, since HLG images and maxDRLs equal to the number of types of system parameters are generated from a PQ image of 1 frame, the configurations of the MetaData 805 and the Image Data 809 of the data file in which HLG images are stored are different from each other. HEIF-format data files can store multiple images in one file, and therefore the number of data files will not increase.
Similarly, for the MetaData 805, maxDRLs corresponding to the system parameters are sequentially stored.
When displaying an HLG image from the HLG image data file generated in this embodiment, for example, the display program acquires HLG image data that matches the gamma or maximum display luminance of the display device from the Image Data 809 and uses the acquired HLG image data for display. Also, when using a maxDRL as well, a maxDRL that matches the gamma or maximum display luminance of the display device is acquired from the MetaData 805 and used. Note that the display program may be, for example, an application program running on a computer device. Alternatively, if the HLG image data file is recorded on a recording medium such as an optical disc, the display program may be a program that runs on a drive that plays back the recording medium.
In this embodiment, when converting a PQ image into an HLG image, HLG images corresponding to each of a plurality of system parameters having different values are generated. For this reason, it is possible to display an HLG image that is suitable for the system parameters of the environment in which the HLG image is displayed. Also, since a maxDRL is also generated and recorded for each system parameter value, it is possible to use a maxDRL that is suitable for the HLG image used for display.
Instead of applying a transfer function, a lookup table in which the discrete signal values of the PQ image are associated with the values of the Display Light, and a lookup table in which discrete values of the Scene Light and signal values of the HLG image are associated with each other may also be used. For example, instead of applying PQ EOTF, a value of the Display Light corresponding to the signal value of each pixel forming the PQ image is determined by referring to a lookup table. A signal value not stored in the lookup table can be obtained by interpolating the value of the Display Light according to the difference from a stored signal value. Similarly, instead of applying the HLG OETF, the value of an individual pixel of the Scene Light (Display Light) can also be converted to a signal value of a pixel of an HLG image using a lookup table. The maxDRL can also be converted in the same way. For example, a lookup table can be used when processing speed is more important than conversion accuracy.
Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2023-001914 | Jan 2023 | JP | national |
