Patent Application
20240314451
The present invention relates to a technique of processing combination of a plurality of images.
When imaging of a subject is performed using an imaging device, imaging conditions are determined on the basis of photometry information and an imaging operation is performed. Patent Literature 1 discloses a technique of performing image combination when the luminance of a subject is equal to or less than a predetermined value. It is possible to capture a moving image with a smooth motion at a low cost while correcting camera shake using an electronic image segmentation system when a subject with low luminance/low illuminance is imaged.
In the technique disclosed in Patent Literature 1, when image combination is started after the luminance becomes equal to or less than the predetermined value and a long time is taken for image processing, a combined image is output with a delay with respect to a change in luminance. For example, in an application to an onboard camera, there is a likelihood that an image from which a subject cannot be recognized due to a phenomenon called blackening or halation will be temporarily output at a place with a great change in luminance such as an entrance/exit of a tunnel.
The present invention provides an image processing device, an image processing method, an imaging device, and a recording medium that allow a recognizable image to be more rapidly output with respect to a change in luminance.
An image processing device according to an embodiment of the present disclosure comprises a memory storing instructions, and a processor executing the instructions causing the image processing device to acquire outside information, acquire luminance information of an image, combine a plurality of images and to generate a combined image, perform a process of outputting the acquired image or the combined image, and start a process of generating the combined image in response to acquisition of the outside information, and perform a process of outputting an uncombined image not combined or the combined image on the basis of the luminance information.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the embodiments, application examples with respect to an image processing device and an imaging device that can acquire outside information and perform a process of outputting a recognizable image with respect to a change in luminance will be described.
A camera that is mounted in a mobile object such as a vehicle will be exemplified as an imaging device according to a first embodiment.
The imaging unit 101 includes an imaging element and acquires an image of a subject. The image combining unit 102 combines a plurality of images acquired by the imaging unit 101 and generates a combined image. An image combining method is arbitrary and can employ any known method. For example, an image combining method using addition can be employed. In the image combining method using addition, a method of adding all images at a fixed combination proportion, a method of adding images at different combination proportions according to segments of the images, a method of adding different numbers of images to be combined according to segments of the images, and the like can be employed.
The outside information acquiring unit 103 acquires outside information through wireless communication or wired communication with an external device. For example, the outside information is as follows.
The outside information acquiring unit 103 outputs the acquired outside information to the control unit 106.
The photometry unit 104 measures luminance information of an imaging area associated with the imaging unit 101. Data of the measurement results is output from the photometry unit 104 to the control unit 106.
The image output unit 105 performs a process of outputting an image not combined (hereinafter referred to as an uncombined image) acquired by the imaging unit 101 or a combined image combined by the image combining unit 102. The uncombined image or the combined image is subjected to a process of being displayed on a display unit, a process of being recorded on a recording medium, or a process of being output to an external device.
The control unit 106 includes, for example, a central processing unit (CPU) and comprehensively controls the constituents of the imaging device 100. The control unit 106 controls the imaging unit 101, the image combining unit 102, and the image output unit 105 on the basis of the outside information acquired by the outside information acquiring unit 103 and the luminance information measured by the photometry unit 104.
In S101, the imaging unit 101 acquires an image through imaging. In this case, the image combining unit 102 does not perform a combination process on the acquired image. The image output unit 105 performs a process of outputting the image acquired by the imaging unit 101 and not subjected to the combination process. For example, the vehicle having the imaging device 100 mounted thereon is traveling on a road before entering a tunnel in the daytime, and the imaging unit 101 performs a process of imaging an external environment. At this time, the imaging unit 101 is subjected to exposure control based on an external environment, and an exposure time is set to, for example, less than 1/30 seconds. In a period in which the subsequent processes of S102 to S107 are performed, the exposure time is set to be less than 1/30 seconds such that an imaging period of the image acquired by the imaging unit 101 is not greater than a predetermined value (for example, a value corresponding to 30 fps). Then, the process flow proceeds to S102.
In S102, the control unit 106 determines whether predetermined outside information (hereinafter referred to as first outside information) has been acquired by the outside information acquiring unit 103. The first outside information is information for determining that the position is close to, for example, an entrance of a tunnel using position information of the mobile object, map information, road information, identification information, distance measurement information, or the like. When it is determined that the outside information acquiring unit 103 has acquired the first outside information, the process flow proceeds to S103. When it is determined that the outside information acquiring unit 103 has not acquired the first outside information, the process flow proceeds to S101.
In S103, the image combining unit 102 starts the process of generating a combined image. The process of generating a combined image is performed on the basis of predetermined combination conditions (a combination ratio or the number of combined images). For example, the combined image is generated such that the luminance of a low-luminance part in an uncombined image is higher than that before combination. In S103, the image output unit 105 performs a process of outputting the uncombined image acquired by the imaging unit 101. Then, the process flow proceeds to S104.
In S104, the control unit 106 compares a value of luminance information measured by the photometry unit 104 with a predetermined value. The predetermined value is a first threshold value of luminance information and is set to, for example, a value enabling a great decrease in luminance at an entrance of a tunnel to be detected. When it is determined that the value of luminance information is equal to or less than the predetermined value, the process flow proceeds to S105. When it is determined that the value of luminance information is greater than the predetermined value, the process flow proceeds to S103.
In S105, the control unit 106 performs a process of switching the uncombined image output by the image output unit 105 to the combined image which is generated by the image combining unit 102 of which generation has been started in S103. In this embodiment, the image combining unit 102 starts the combination process at a time point at which the first outside information is acquired before a time point at which the value of luminance information becomes equal to or less than the predetermined value. In comparison with a case in which the combined image generating process is started at the time point at which the value of luminance information becomes equal to or less than the predetermined value and the combined image is output, a recognizable combined image can be output without a delay after the value of the luminance information has become equal to or less than the predetermined value. For example, an image without blackening can be output when the mobile object enters a tunnel. In the daytime, an external environment may be brighter than the inside of a tunnel and the luminance may decrease greatly at an entrance of the tunnel. In the nighttime, the inside of a tunnel may be brighter than an external environment, and luminance at an exit of the tunnel may decrease greatly. In consideration of such cases, the processes of S101 to S104 may be performed in the tunnel, and information enabling determination of the vicinity of the exit of the tunnel in S102 may be acquired.
According to this embodiment, the image combining unit 102 can start the combined image generating process in response to acquisition of the outside information and more rapidly output a recognizable image with change in luminance.
Modified examples of this embodiment will be described below. In a plurality of modified examples described below and embodiments described later, description of the same details as in the first embodiment and constituents corresponding to the first embodiment will be omitted, and differences therefrom will be mainly described.
A process flow according to Modified Example 1 of the first embodiment will be described below with reference to
In S106, the control unit 106 compares a value of luminance information measured by the photometry unit 104 with a predetermined value. The predetermined value is a second threshold value associated with the luminance information and is set to a value enabling detection of a great increase in luminance at an exit of a tunnel. When it is determined that the value of luminance information is equal to or greater than the predetermined value (equal to or greater than the threshold value), the process flow proceeds to S107. When it is determined that the value of luminance information is less than the predetermined value, the process flow proceeds to S105.
In S107, the control unit 106 performs a process of switching a combined image to an uncombined image. The image combining unit 102 stops the combined image generating process. The image output unit 105 performs an uncombined image outputting process.
According to this modified example, when a change in luminance is detected and an exit of a tunnel is recognized, the combined image can be immediately switched to the uncombined image and the uncombined image can be output. Accordingly, it is possible to output an image without halation.
When the process flow proceeds from S101 to S102 and it is determined that the first outside information has not been acquired, the process flow proceeds to S108. In S108, the control unit 106 compares a value of luminance information measured by the photometry unit 104 with a predetermined value. In this case, the predetermined value is the first threshold value mentioned in S104.
When it is determined in S108 that the value of luminance information is equal to or less than the predetermined value (equal to or less than the threshold value), the process flow proceeds to S105. In this case, the image combining unit 102 immediately starts the image combining process, and the image output unit 105 performs the combined image outputting process. When it is determined in S108 that the value of luminance information is greater than the predetermined value, the process flow proceeds to S101.
According to this modified example, when the outside information acquiring unit 103 has not acquired the first outside information, comparison with a threshold value is performed with reference to luminance information. When it is determined that the vehicle is in a tunnel, it is possible to output the combined image. It is assumed that the outside information acquiring unit 103 has acquired the first outside information indicating that the vehicle is at an entrance of a tunnel in the tunnel after the combined image has been output. In this case, the image combining unit 102 continues to perform the combined image generating process, and the image output unit 105 continues to perform the combined image outputting process.
After the combined image has been output in S105, the process flow proceeds to S109. In S109, the control unit 106 determines whether the luminance information measured by the photometry unit 104 is outside of a predetermined range. When “luminance information is outside of a predetermined range,” the predetermined range means a range of luminance information in which a combined image of which an exposure intensity is not appropriate and in which halation or blackening is not generated is generated through the process of the image combining unit 102 in S105. The process flow proceeds to S106 when it is determined that the luminance information is outside of the predetermined range, and the process flow proceeds to S105 when it is determined that the luminance information is not outside of the predetermined range.
The process of S106 is the same as described above with reference to
The process of S107 is the same as described above with reference to
When exposure of the combined image is not appropriate in a tunnel, the process flow proceeds from S106 to S110. In S110, the control unit 106 changes settings of the number of images combined by the image combining unit 102 according to the luminance information in S109 such that an image with an appropriate exposure intensity can be combined. After combination conditions in the image combining unit 102 have been changed in S110, the process flow proceeds to S105. In S105, a combined image is generated under the combination conditions changed in S110, and the combined image is output.
According to this modified example, when luminance information changes after a combined image has been output, it is possible to output a combined image based on the changed luminance information.
In S101A, the imaging unit 101 performs imaging with a normal imaging period (hereinafter referred to as T0). The image combining unit 102 does not perform a process of generating a combined image, and the image output unit 105 performs a process of outputting an uncombined image captured in the imaging period T0.
When it is determined in S102 that the outside information acquiring unit 103 has not acquired the first outside information, the process flow proceeds to S101A. When it is determined in S102 that the outside information acquiring unit 103 has acquired the first outside information, the process flow proceeds to S103A.
In S103A, the imaging unit 101 performs imaging with an imaging period (hereinafter referred to as T1) shorter than the normal imaging period T0. Changing and setting of the imaging period are performed by the control unit 106. The image combining unit 102 starts the process of generating a combined image using an image captured with the imaging period T1. The image output unit 105 performs the process of outputting the image (an uncombined image) acquired by the imaging unit 101. The process flow proceeds to S105A when it is determined in S104 that the value of luminance information measured by the photometry unit 104 is equal to or less than the predetermined value, and the process flow proceeds to S103A when it is determined that the value of luminance information is not equal to or less than the predetermined value.
In S105A, the imaging period is T1, and the image combining unit 102 continues to perform the process of generating a combined image. The image output unit 105 switches the uncombined image to the combined image generated by the image combining unit 102 and performs the process of outputting the combined image.
When the measurement result of the photometry unit 104 changes and thus it is determined in S106 that the value of luminance information is equal to or greater than the predetermined value, the process flow proceeds to S107A. When it is determined that the value of luminance information is less than the predetermined value, the process flow proceeds to S105A.
In S107A, the control unit 106 performs a process of returning the imaging period T1 to the normal imaging period T0. The image combining unit 102 stops the process of generating a combined image. The image output unit 105 switches the combined image to the uncombined image and performs the process of outputting the uncombined image.
According to this modified example, by changing the imaging period in the imaging unit 101 in response to acquisition of outside information, it is possible to output a recognizable image without a delay with respect to a change in luminance.
An image combining method that is performed by the image combining unit 102 will be described below. When a method of adding a plurality of images is used and image blurring occurs between images, there is a likelihood that blurring will occur in a combined image through simple addition. In order to correct blurring between images, the outside information acquiring unit 103 can acquire second outside information. The second outside information is detection information from an angular velocity sensor, an acceleration sensor, or the like associated with a device in which the imaging device 100 is provided. The second outside information can be acquired and used to correct blurring between images. As the second outside information, detection information from an angular velocity sensor or an acceleration sensor provided in the imaging device 100 can be acquired and used to correct blurring between images.
As another method, a method of detecting a motion vector which is acquired from a plurality of images with different imaging times can be used. As the second outside information, a motion vector can be acquired and used to correct blurring between images. A configuration of an imaging device 100C will be described below with reference to
Blurring between images can be corrected using the detection information from the angular velocity sensor or the acceleration sensor and the motion vectors together as the second outside information. This is similarly applicable to embodiments and modified examples which will be described later.
A second embodiment will be described below with reference to
In S102B, the control unit 106 determines whether the outside information acquiring unit 103 has acquired first outside information. When it is determined that the outside information acquiring unit 103 has acquired first outside information, the process flow proceeds to S103B. When it is determined that the outside information acquiring unit 103 has not acquired first outside information, the process flow proceeds to S101B.
In S103B, the first image combining unit 102A performs a process of generating a first combined image. The second image combining unit 102B starts a process of generating a second combined image using images acquired by the imaging unit 101. At this time, the image output unit 105 performs the process of outputting the first combined image. Then, the process flow proceeds to S104B.
In S104B, the control unit 106 compares the value of luminance information measured by the photometry unit 104 with a predetermined value (a first threshold value). When it is determined that the value of luminance information is equal to or less than the predetermined value, the process flow proceeds to S105B. When it is determined that the value of luminance information is greater than the predetermined value, the process flow proceeds to S103B.
In S105B, the first image combining unit 102A stops the process of generating a first combined image, and the second image combining unit 102B performs the process of generating a second combined image. The image output unit 105 switches the first combined image to the second combined image and performs the process of outputting the second combined image. The number of combined images in the second image combining unit 102B is set to be larger than the number of combined images in the first image combining unit 102A. Accordingly, it is possible to generate a recognizable combined image even in a situation of a darker surrounding environment.
In this embodiment, it is possible to output a recognizable image without a delay with respect to a change in luminance by using a plurality of image combining units with different numbers of combined images. The configuration including two image combining units is an example, and this embodiment is applicable to a configuration including three or more image combining units with different combination conditions such as the number of combined images.
A process flow according to a modified example of the second embodiment will be described below with reference to
In S106B, the control unit 106 determines whether the outside information acquiring unit 103 has acquired third outside information. The third outside information is, for example, information for identifying the vicinity of an exit of a tunnel which is emitted from a beacon provided at an exit of the tunnel. Alternatively, detection information from an acceleration sensor, an angular velocity sensor and map information (GPS information) of a surrounding environment of a mobile object, or the like may be used as the third outside information. By acquiring the third outside information, it is possible to determine whether a vehicle approaches a specific place such as an exit of a tunnel. When it is determined in S106B that the outside information acquiring unit 103 has acquired the third outside information, the process flow proceeds to S107B. When it is determined that the outside information acquiring unit 103 has not acquired the third outside information, the process flow proceeds to S105B.
In S107B, both the first image combining unit 102A and the second image combining unit 102B perform a combined image generating process. When it is determined that the vehicle approaches an exit of a tunnel through acquisition of the third outside information, the first image combining unit 102A restarts the process of generating a first combined image. At this time, the image output unit 105 performs the process of outputting a second combined image. Then, the process flow proceeds to S108B.
In S108B, the control unit 106 compares the value of luminance information measured by the photometry unit 104 with a predetermined value (a second threshold value). The process flow proceeds to S109B when it is determined that the value of luminance information is equal to or greater than the predetermined value, and the process flow proceeds to S107B when it is determined that the value of luminance information is less than the predetermined value.
When the process flow proceeds from S108B to S109B, it is detected that the vehicle has exited the tunnel. In S109B, the first image combining unit 102A continues to perform the process of generating a first combined image, and the second image combining unit 102B stops the process of generating a second combined image. The image output unit 105 switches the second combined image to the first combined image and performs the process of outputting the first combined image.
According to this embodiment, it is possible to output a recognizable image without a delay with respect to a change in luminance at an entrance and an exit of a tunnel or the like.
A third embodiment will be described below with reference to
The image acquiring unit 301 acquires image data to be processed from an external device (such as an imaging device). The luminance acquiring unit 304 acquires luminance information from the image data acquired by the image acquiring unit 301. The image output unit 105 performs a process of outputting an uncombined image acquired by the image acquiring unit 301 or a combined image generated by the image combining unit 102.
The control unit 106 controls the image acquiring unit 301, the image combining unit 102, and the image output unit 105 on the basis of outside information acquired by the outside information acquiring unit 103 and luminance information acquired by the luminance acquiring unit 304.
In S303, the control unit 106 determines whether the outside information acquiring unit 103 has acquired first outside information. When it is determined that the outside information acquiring unit 103 has acquired first outside information, the process flow proceeds to S304. When it is determined that the outside information acquiring unit 103 has not acquired first outside information, the process flow proceeds to S302.
In S304, the image combining unit 102 performs the process of generating a combined image, and the image output unit 105 performs the process of outputting an uncombined image acquired by the image acquiring unit 301. Then, in S305, the control unit 106 compares a value of luminance information acquired by the luminance acquiring unit 304 with a predetermined value (a first threshold value). When it is determined that the value of luminance information is equal to or less than the predetermined value, the process flow proceeds to S306. When it is determined that the value of luminance information is greater than the predetermined value, the process flow proceeds to S304.
In S306, the image combining unit 102 performs the process of generating a combined image, and the image output unit 105 performs the process of outputting the combined image. The output process is performed with switching from an uncombined image output by the image output unit 105 to a combined image generated by the image combining unit 102 in S304.
In this embodiment, the process of generating a combined image is started, for example, at a time point at which outside information for identifying that a vehicle is in the vicinity of an entrance of a tunnel has been acquired. It is possible to output a combined image without a delay from a time point at which luminance information is equal to or less than the predetermined value and it is determined that a vehicle has entered a tunnel.
According to this embodiment, it is possible to provide an image processing device that can output a combined image without a delay with respect to a change in luminance by employing a configuration for acquiring image data from an external device.
The image processing device 300 has a configuration in which the image acquiring unit 301 acquires image data from an external device instead of causing the imaging unit 101 to perform imaging. The image processing device 300 includes the luminance acquiring unit 304 instead of the photometry unit. Without measuring luminance of an imaging target, the image acquiring unit 301 can acquire luminance information from image data acquired by the image acquiring unit 301.
In this regard, the configurations or methods described in the first embodiment, the second embodiment, and the modified examples are applicable by replacing the imaging unit with the image acquiring unit in the third embodiment and replacing the photometry unit with the luminance acquiring unit.
According to this embodiment, it is possible to provide an imaging device and an image processing device that can more rapidly output a recognizable image with respect to a change in luminance.
The present invention can be realized through a process of supplying a program for realizing one or more functions of the aforementioned embodiments to a system or a device via a network or a storage medium and causing one or more processors in a computer of the system or the device to read and execute the program. The one or more functions can be realized by a circuit (for example, an ASIC).
Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021-201737 | Dec 2021 | JP | national |
This application is a Continuation of International Patent Application No. PCT/JP2022/43593, filed Nov. 25, 2022, which claims the benefit of Japanese Patent Application No. 2021-201737, filed Dec. 13, 2021, both of which are hereby incorporated by reference herein in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2022/043593 | Nov 2022 | WO |
| Child | 18676677 | US |
