The present disclosure relates to an image processing algorithm evaluating apparatus.
Vehicles with driving aiding functions such as autonomous driving are currently under development. Provided onboard a vehicle with such a driving aiding function is an image processing algorithm for capturing an image of the environment around the vehicle using an onboard camera or the like, and determining a situation around the vehicle based on the captured image thus captured.
In order to implement appropriate driving aiding functions, such an image processing algorithms are subjected to various types of performance evaluation. For example, there is a known technology for evaluating the performance of an image processing algorithm by causing the image processing algorithm to apply image processing on a composite image resultant of superimposing a weather disturbance image created by computer graphics, over an actual image captured from the vehicle (see Patent Literature 1, for example).
However, images such as a weather disturbance image generated in the technology disclosed in Patent Literature 1 does not reflect the situation such as objects captured in the actual image. Therefore, if an image processing algorithm is evaluated using such a composite image, the evaluation may end up being inappropriate.
The present disclosure is made in consideration of the above, and an object of the present disclosure is to provide an image processing algorithm evaluating apparatus capable of evaluating the performance of an image processing algorithm for determining a situation around a vehicle, appropriately.
An image processing algorithm evaluating apparatus according to the present disclosure includes: an image storage unit that stores actual images captured from a vehicle; an image generating unit that acquires a target image, when receiving disturbance information representing a disturbance in the target image from among the actual images stored in the image storage unit, interprets the target image, and generates a composite image by manipulating the target image in such a manner that the disturbance is reflected to the target image, based on the interpretation; and an image processing unit that evaluates performance of an image processing algorithm for determining a situation around the vehicle based on the generated composite image.
According to the present disclosure, it is possible to evaluate the performance of an image processing algorithm for determining a situation around a vehicle, appropriately.
Some embodiments of an image processing algorithm evaluating apparatus according to the present disclosure will now be explained with reference to the drawings. These embodiments are, however, not intended to limit the scope of the present invention. The elements disclosed in the embodiments described below include those that are replaceable and easily replaceable, or those that are substantially identical.
The image storage unit 10 stores therein actual images captured from the vehicle. Examples of actual images includes a front image resultant of capturing an image ahead of the vehicle, a side image resultant of capturing an image from a side of the vehicle, and a rear image resultant of capturing an image behind the vehicle. Examples of actual images also includes an actual image captured in clear weather, an actual image captured in rainy weather, an actual image captured in daytime, and an actual image captured in nighttime, for example.
The image generating unit 20 generates a composite image by combining a disturbance that is based on disturbance information received from an input unit not illustrated, for example, with a target image received from the image storage unit 10. The image generating unit 20 includes an image interpreting unit 21 and an image manipulating unit 22. The image interpreting unit 21 interprets an image stored in the image storage unit 10. The image manipulating unit 22 manipulates a target image input to the image generating unit 20.
The image processing unit 30 evaluates the performance of an image processing algorithm performing image processing based on the generated composite image, and determining the situation around the vehicle. The image processing unit 30 applies image processing to the composite image using the image processing algorithm, and calculates determination information for determining the situation around the vehicle. The image processing unit 30 stores the calculated determination information in a storage unit, not illustrated, for example. An example of the determination information includes an approaching and crossing time that is the time for a traveling vehicle to move from the position at which the image is captured to a position where there is an object ahead of the vehicle. The image processing unit 30 can evaluate the performance of the image processing algorithm based on whether there is any large difference between the determination information resultant of processing an image without any disturbance, and determination information resultant of processing the image with a disturbance.
The image interpreting unit 21 includes operating units 21a, 21b. The operating unit 21a acquires the input target image. The operating unit 21a performs image processing to estimate a distance from the position at which the target image is captured to the object included in the target image, for each pixel. When the object included in the target image is the sky, the distance can be estimated as infinity (indicating the sky). The operating unit 21a then generates distance information in which the estimated distance is associated with the corresponding pixel, and output the distance information.
The operating unit 21b acquires the distance information output from the operating unit 21a. The operating unit 21b also acquires disturbance information entered via an input unit, not illustrated. An example of the disturbance information includes a fog image generated by computer graphics, for example. This image may be an image having the same resolution as that of the target image (actual image), for example.
The operating unit 21b calculates the intensity of disturbance corresponding to each pixel, based on the distance information. For example, in an environment in which fog appears, an object is affected more by the fog when the object is farther away from the position at which the image is captured. Therefore, by adjusting the intensity of disturbance for each pixel based on the distance included in the distance information, it is possible to adjust the disturbance to that close to the actual environment. After making the adjustment of the disturbance for each pixel, the operating unit 21b outputs the disturbance information having the disturbance adjusted.
The image manipulating unit 22 acquires the input target image. The image manipulating unit 22 also acquires the disturbance information output from the operating unit 21b. The image manipulating unit 22 generates a composite image by superimposing the disturbance included in the acquired disturbance information over each pixel of the target image. In other words, the disturbance information input to the image manipulating unit 22 is information resultant of adjusting, based on the distance, the intensity of the disturbance in each pixel of the disturbance information input from the input unit, not illustrated. By superimposing the adjusted disturbance over the target image, an appropriate composite image that is closer to the actual disturbance environment can be generated.
Based on the composite image, the image processing unit 30 evaluates the performance of an image processing algorithm for determining the situation around the vehicle. Because the performance of an image processing algorithm is evaluated based on an appropriate composite image that is closer to the actual surrounding environment, an appropriate evaluation result can be achieved, compared with when superimposed is disturbance generated simply by computer graphics. Furthermore, compared with a configuration in which a simulator is used to reproduce the actual surrounding environment three dimensionally in detail to determine the situation around the vehicle, it is possible to achieve an appropriate evaluation result without using a large complex system.
In the manner described above, the image processing algorithm evaluating apparatus 100 according to the present embodiment includes: the image storage unit 10 that stores therein a plurality of actual images captured from a vehicle; the image generating unit 20 that acquires a target image, when receiving disturbance information representing a disturbance in the target image, from among the plurality of actual images stored in the image storage unit 10, interprets the target image, and generates a composite image by manipulating the target image in such a manner that the disturbance is reflected to the target image based on the interpretation; and the image processing unit 30 that evaluates the performance of an image processing algorithm for determining the situation around the vehicle based on the generated composite image.
With this configuration, because a composite image is generated by manipulating the target image in such a manner that disturbance is reflected to the target image, based on the result of the interpretation of the target image, made by the image generating unit 20, a composite image close to the actual surrounding environment can be generated, compared with when superimposed is disturbance generated merely by computer graphics, for example. In this manner, it is possible to evaluate the performance of an image processing algorithm for determining the situation around the vehicle, appropriately.
Furthermore, in the image processing algorithm evaluating apparatus 100 according to the present embodiment, the image generating unit 20 estimates a distance from a position at which the target image is captured to an object included in the target image, calculate an intensity of disturbance resultant of being subjected to the estimated distance, and generates the composite image based on the calculation result. Therefore, a composite image even closer to the actual surrounding environment can be generated.
A second embodiment will now be explained.
In the present embodiment, the training image storage unit 40 stores therein a reference image that is an actual image not including any disturbance, and a disturbance image that is an actual image including disturbance, as training images. An example of the reference image includes an actual image captured under a condition resulting in a least disturbance, e.g., during the daytime under clear weather. An example of the disturbance image includes an actual image captured under a condition where a larger amount of disturbance than that in the reference image is introduced, e.g., at the time of rain or snow, with fog, at the time of dawn, in the evening, or during the nighttime. The training image storage unit 40 may store therein a reference image and a disturbance image of the same subject or corresponding subjects, in a manner associated with each other. For example, the training image storage unit 40 may store therein a reference image of a predetermined location captured during the daytime under clear weather, and a disturbance image of the predetermined location captured at the time of rain or snow, with fog, at the time of dawn, in the evening, or during the nighttime, in a manner associated with each other.
In the present embodiment, the image learning unit 50 can generate, when receiving a target image and disturbance information representing the disturbance in the target image, a training composite image, through learning with a neural network, for example. As a method for implementing predetermined image manipulation using a neural network in the image learning unit 50, a technology referred to as a generative adversarial network (GAN) or a technology referred to as a cycle GAN may be used, for example.
In the cycle generative adversarial network, any reference image not including any disturbance and a disturbance image including disturbance may be used, and it is not necessary for the reference image and the disturbance image to be associated with each other. Therefore, compared with the generative adversarial network, training data can be collected easily.
Furthermore, when the cycle generative adversarial network is used, a plurality of disturbance images with different degrees of disturbance can be stored in the training image storage unit 40.
By using the cycle generative adversarial network in a configuration in which a plurality of disturbance images with different degrees of disturbance are stored in the training image storage unit 40, when information designating a degree of disturbance (brightness) and information designating the type of disturbance (nighttime) are received as the disturbance information, for example, the image generating unit 20 can generate a composite image having a brightness corresponding to the designated brightness. In the example illustrated in
After the training has finished, when receiving a target image and disturbance information, the image generating unit 20 acquires the target image from the image storage unit 10 (Step S207). The image interpreting unit 21 interprets the image based on the training result (Step S208). The image manipulating unit 22 generates a composite image that is the disturbance added to the target image, based on the result of the interpretation (Step S209).
As described above, in the image processing algorithm evaluating apparatus 100 according to the present embodiment, the training image storage unit 40 stores therein a reference image that is an actual image not including any disturbance and a disturbance image that is an actual image including a disturbance; the image learning unit 50 performs training using a generative adversarial network or a cycle generative adversarial network, based on the reference image and the disturbance image; and the image generating unit 20 interprets the target image based on the training result of the image learning unit 50, and generates a composite image.
In this configuration, the image learning unit 50 is trained with an actual image using a generative adversarial network or a cycle generative adversarial network, and the image generating unit 20 interprets the target image based on the training result and generates a composite image. Therefore, it is possible to generate a composite image closer to the actual surrounding environment. In this manner, it is possible to evaluate the performance of an image processing algorithm for determining the situation around the vehicle, appropriately.
Furthermore, the training image storage unit 40 stores therein a plurality of the disturbance images including the disturbance of a same type with different degrees; the image learning unit 50 carries out training based on the plurality of disturbance images including the disturbance with different degrees; and when receiving an input of the disturbance information including a degree of the disturbance, the image generating unit 20 interprets the target image based on the training result of the image learning unit 50, and generates the composite image in such a manner that the disturbance is reflected to the target image by a degree corresponding to the disturbance information. In this manner, because a composite image with a different degree of disturbance can be generated appropriately, a wide range of performance evaluation of an image processing algorithm can be performed.
A third embodiment will now be explained. The image processing algorithm evaluating apparatus 100 according to the third embodiment has a configuration including the image storage unit 10, the image generating unit 20, the image processing unit 30, the training image storage unit 40, and the image learning unit 50, in the same manner as that according to the second embodiment. In the present embodiment, the type of the actual image stored in the training image storage unit 40, and processing performed by the image generating unit 20 and the image learning unit 50 are different from those according to the second embodiment described above.
The training image storage unit 40 stores therein actual images including a reference image and a disturbance image as the training images, in the same manner as in the second embodiment. In the present embodiment, the training image storage unit 40 stores therein actual images with different attributes. Examples of the actual images with different attributes include a plurality of actual images with different attributes such as the location where the image is captured, e.g., an actual image captured in a shopping district, an actual image captured in a residential area, and an actual image captured on a mountain road. The training image storage unit 40 can store therein label information indicating the attribute, in a manner associated with such an actual image.
In the present embodiment, when a target image and disturbance information for the target image are received, the image generating unit 20 extracts label information indicating the above described attribute, from the target image. The image generating unit 20 is capable of interpreting the target image based on the training result of the image learning unit 50, which is to be described later, and generating a composite image reflecting the attribute. As one example, when the target image is a reference image of a “shopping district” during the daytime, and “nighttime” is input as disturbance information, conversions for not only darkening the sky but also turning on the illumination of nearby buildings may be performed. As another example, when the target image is a reference image of a “mountain road” during the daytime, and “nighttime” is input as disturbance information, conversions for not only darkening the sky but also darkening the entire surroundings may be performed.
In this example, the image generating unit 51 generates a first composite image that is a reference image combined with disturbance. For example, when extracted is an attribute “shopping district”, and input is the disturbance “nighttime”, the image generating unit 51 generates a composite image in such a manner that the disturbance of nighttime is reflected to the reference image of the shopping district. The image generating unit 53 also generates a second composite image resultant of removing the disturbance of nighttime from the disturbance image during the nighttime in the shopping district. The authenticity determining unit 52 determines the authenticity of the first composite image based on the authentic disturbance image, and also determines whether the attribute “shopping district” is appropriately reflected to the first composite image. Furthermore, the authenticity determining unit 54 determines the authenticity of the second composite image based on the authentic reference image, and also determines whether the disturbance of the nighttime in the shopping district has been removed appropriately. In the manner described above, in the present embodiment, the authenticity determining units 52 and 54 make a determination on whether the attribute is reflected appropriately, in addition to the determination of the authenticity of the composite image.
In the present embodiment, the image generating units 51 and 53 generate a first composite image and a second composite image in such a manner that such composite images are determined by the authenticity determining units 52 and 54 as being closer to the authentic image and reflecting the attribute more appropriately. Furthermore, the authenticity determining units 52, 54 attempt to detect more differences with respect to the authentic image, and points where the attribute is not reflected appropriately, from the generated first composite image and second composite image. By proceeding the training by causing such two networks to compete with each other alternately, the image generating unit 51 is trained to become able to generate a composite image resulting in a higher determination accuracy ratio of the authenticity, as well as a higher determination accuracy ratio of the degree by which the attribute is reflected.
When receiving a target image and disturbance information, the image generating unit 20 acquires the input target image (Step S308). The image interpreting unit 21 extracts the label information indicating the attribute included in the target image (Step S309), and interprets the target image (Step S310). The image manipulating unit 22 generates a composite image in which the disturbance is added to the target image, based on the result of the interpretation (Step S311).
As described above, in the image processing algorithm evaluating apparatus 100 according to the present embodiment, the training image storage unit 40 stores therein a plurality of the actual images including different attributes in a manner associated with label information indicating the attributes; the image learning unit 50 carries out training using the cycle generative adversarial network in such a manner that a determination accuracy related to the attributes between the generated training composite image and the disturbance image is improved; and when receiving the target image and the disturbance information, the image generating unit 20 extracts the label information indicating the attribute of the target image, interprets the target image based on the training result of the image learning unit 50, and generates the composite image reflecting the attribute.
With such a configuration, training can be carried out using a cycle generative adversarial network to improve the determination accuracy ratio of the authenticity of the training composite image, as well as the determination accuracy ratio of the degree by which the attribute is reflected. Therefore, it is possible to generate a composite image in a manner suitable for the attribute of the actual image.
The technical scope of the present invention is not limited to the embodiments described above, and the embodiments may be modified as appropriate, within the scope not deviating the gist of the present invention.
Number | Date | Country | Kind |
---|---|---|---|
2021-083932 | May 2021 | JP | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/JP2022/020645 | 5/18/2022 | WO |