Patent Grant
10762606
The present disclosure relates to an image processing apparatus and method for generating a high quality image.
There is a technique for generating a high quality image from a plurality of images such as described in Japanese Patent No. 4942221. In the known technique, images from multiple viewpoints undergo a projective transformation, one by one, in such a manner as to cause the images to agree with an image from a virtual viewpoint, and the superimposed images are averaged to obtain a high quality image.
The known method requires to distort the images with appropriate projective transformation parameters in such a manner that the images are superimposed substantially at the pixel level. Moreover, if a subject has depth, it is required to distort the images according to the areas, considering parallax. Moreover, in the known method, improvement in image quality is based on the image averaging process; therefore, it is difficult to improve a cause of systematic degradation common among the images, such as chromatic aberration and spherical aberration.
An image processing apparatus includes: an acquisition unit configured to acquire a plurality of images each capturing an identical target and having a different attribute; a derivation unit configured to derive features from the plurality of images, using a first neural network; an integration unit configured to integrate the features derived from the plurality of images; and a generation unit configured to generate a higher quality image than the plurality of images from the feature integrated by the integration unit, using a second neural network.
Further features of the present disclosure will become apparent from the following description of exemplary embodiments (with reference to the attached drawings).
Embodiments of the present disclosure are described hereinafter with reference to the drawings.
The CPU 11 executes processes based on the program stored in the memory 12 to achieve the functional configurations of the image processing apparatus 10, the functional configurations being illustrated in
The purpose of the image processing apparatus 10 of the first embodiment is to generate a high quality image that can be obtained by an image capture device with a highly accurate configuration, from a plurality of low quality images of an image capture device with a simple configuration (hereinafter also referred to as the low quality image capture device). Resolution enhancement, noise reduction, color correction, sharpening, a change in the depth of field, and the like are conceivable for improvement in image quality. However, improvement in image quality is assumed to indicate resolution enhancement in the embodiment.
In the embodiment, low quality images are assumed to be different in image attribute such as a color image, an infrared image, an image captured with a flash. The viewpoint positions of the low quality images and the high quality image generated are assumed to be the same. A plurality of images with different attributes from the agreed viewpoint is obtained here by the following method: (1) A ray of light is split in the low quality image capture device, using a half mirror, and (2) Light-sensitive elements with different attributes are mixed and arranged in an image capture device. The low quality images acquired by the image processing apparatus 10 are assumed to be those captured by the low quality image capture device designed as described above.
The image processing apparatus 10 according to the embodiment is described with reference to the drawings. Those having the same reference numerals throughout the drawings are assumed to perform the same operations, and their repeated descriptions are omitted.
Firstly, an outline of the image processing apparatus 10 of the embodiment is described with reference to a functional configuration diagram of
Next, in order to clarify a difference between a technique obtained by applying the embodiment and a known technique, an outline of a technique described in Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang, Learning a Deep Convolutional Network for Image Super-Resolution. Proceedings of European Conference on Computer Vision (ECCV), 2014, which is a known technique, is described. This technique is for generating an enhanced resolution image from one image. The technique uses a convolutional neural network (convolutional neural network, hereinafter abbreviated to CNN) as illustrated in
The embodiment is described compared with the above known technique. As illustrated in a schematic diagram of
The processes of the above known technique and the embodiment are different in the respect that a high quality image is complementarily generated using the images with the plurality of attributes. Moreover, the processes of the known technique based on image averaging as in Japanese Patent No. 4942221 and the embodiment are different in the respect that not pixels of an image but features are used and integrated at the time of integration. Moreover, the processes of another known technique and the embodiment are still different in the respect that before integration, each of the images with the plurality of attributes undergoes a feature transformation through its own NN, and is transformed into features with abstractness appropriate for integration. In the process of the embodiment, the feature connection is illustrated as one implementation mode of the integration of the images. However, as illustrated in the following embodiments, the integration method is not limited to the feature connection.
The image processing apparatus 10 of the embodiment performs feature transformations 308a to 308c that transforms the low quality images 301a to 301c into features 303a to 303c of the CNN. The feature transformations 308a to 308c each include a two-stage convolution process.
Next, the image processing apparatus 10 is different from the known technique in the respect of including a process of feature integration 309. In the process of the feature integration 309, the features 303a to 303c of the CNN of the low quality images are connected to obtain one feature 304 of the CNN. After the features are integrated into one, the image processing apparatus 10 performs a process of an image quality improvement transformation 315 on the feature 304, and obtains a result of a high quality image as a final output.
Next, the operation of the image processing apparatus 10 is described in detail using a flowchart of
S103 to S107 are a loop process of performing the following process for each of the low quality images 301a to 301c. Firstly, in S104, the feature transformation unit 102 corrects a distortion of an image resulting from, for example, an optical configuration of the low quality image capture device. Furthermore, in S105, the feature transformation unit 102 adjusts the size and aspect in such a manner that the images agree in size and aspect ratio with each other. The feature transformation unit 102 may perform these correction processes using design values of an optical system and the low quality image capture device. Alternatively, intrinsic parameters of a camera may be estimated from, for example, a pre-captured pattern for calibration to make corrections.
Next, in S106, the feature transformation unit 102 transforms the low quality images 301a to 301c into the features 303a to 303c of the CNN. This is based on the following calculations of a convolution and an activation function in the embodiment.
fi+1=θ(f′i+1),
θ(x)=Max(0,x),
f′i+1(x,y,di+1)=ΣdiΣΔxΣΔyfi(x,y,di)wi(x+Δx,y+Δy,di,di+1)+bi(di)
fi and fi+1 are features before and after the transformation. f(x, y, d) is the d-th value of a multidimensional feature f related to a pixel (x, y). θ is an activation function of half-wave rectification. Δx and Δy are coordinates of each pixel of an image patch of size S×S, and Δx, Δy∈{−S/2, . . . , 0, . . . , S/2}. Moreover, w is a transformation parameter of a fourth-order tensor. b is a bias term. An area outside the image is filled with a value of 0. The size of the image is assumed to be unchanged before and after a convolution operation.
The parameters w and b of the convolution operation are acquired in advance in a case learning method illustrated below. The convolution operation can be qualitatively assumed to be a coding process of a local patch for associating a low quality image with a high quality image being a post-transformation target.
A more specific description is given. An image is considered to be a set of small local patches. The image has spatial continuity. The same subject such as a sky or a wall has a repetitive pattern and a similar pattern. In addition, each patch can be efficiently approximated by a combination of codebooks of a finite number of patterns. This is codebook-based image compression. Each patch of the image is compressed into a codebook and its coefficient (coding), and the image can be generated from the sum of products of the coefficient and the codebook (decoding).
The image processing apparatus 10 of the embodiment also performs a partially similar process to the above process. What is different is that an image targeted for decoding is not an input image itself but a high quality image. Moreover, an information representation of a codebook in a middle stage is required to sufficiently represent high definition information and be robust against noise and the like.
The above known technique has demonstrated that a feature representation of an NN is adequate for this. Various pieces of information such as a combination of colors and the shape of an object are represented in different hierarchical levels in the layers of a CNN where a convolution operation is repeated a plurality of times. Moreover, a necessary type of representation is heuristically acquired according to the learned task. The above description is the background of a convolution operation of an NN to achieve a feature representation (coding) for improvement in image quality.
Next, in S108, the feature integration unit 103 integrates the features 303a to 303c of the CNN obtained in the former stage, and generates the integrated feature 304. The integration process in the embodiment is a feature connection process as in the following equation:
f3(x,y)=[f2a(x,y)T,f2b(x,y)T,f2c(x,y)T]T
f2a(x, y), f2b(x, y), and f2c(x, y) are feature vectors related to the pixel (x, y) of the features 303a to 303c of the CNN, respectively, and T is the transpose.
In S109, the image quality improvement transformation unit 104 performs a process of a deconvolution 310 on the integrated feature 304. As a result, a 1×1× D3-dimensional feature of each pixel is transformed into an S× S× D4-dimensional feature. This calculation is indicated as follows:
fi+1=θ(f′i+1)
f′i+1(Sx+Δx,Sy+Δy,di+1)=Σdiw′i(Δx,Δy,di,di+1)fi(x,y,di)+b′i(Δx,Δy,di+1))
w′ is a transformation parameter, b′ is a bias term, Δx and Δy are elements of coordinates of an image patch obtained by transforming each pixel. The transformation of the above equation causes the feature 304 of the low quality image into a feature 305 with an S-times resolution.
In the above technique, a deconvolution operation is not performed and an input image is mechanically enlarged by the bicubic method. In contrast, the image processing apparatus 10 of the embodiment improves the resolution on the basis of case learning as described below, and weights are optimized through all the layers of the NN.
Next, in S110, the image quality improvement transformation unit 104 performs a process of a convolution 311 and a process of a convolution 312 on the high image quality feature 305 of
Parameters related to a series of transformations of the embodiment are acquired by learning in advance appropriate pre- and post-transformation relationships between an image and a feature from multiple case images. The operation at the time of learning is described below. In addition to the calculations of a convolution and an activation function described here, a process of normalizing the value of a feature may be combined with each stage of the NN. The detailed contents of variations of such a process are not an essential element for the embodiment. Therefore, they are not described in detail here.
(Learning Process)
Next, a description is given of the flow of the detailed process for acquiring weights of an NN of the embodiment, that is, appropriate transformation parameters by learning. A functional configuration diagram of the image processing apparatus 10 related to learning is as illustrated in
Firstly, a set of case images used for learning is assumed to be obtained by capturing in advance images of various subjects, and saving the images in the learning case image retaining unit 109. The case images here include a group of low quality images captured by the image acquisition units 101a to 101c and a high quality image being a final target to be generated from the group of low quality images. An image capture device different from a group of low quality image capture devices is prepared to capture the high quality image. It is assumed here that the position of the image capture device is adjusted in such a manner as to make the viewpoint and focal plane the same as the group of low quality images to capture the high quality image.
When the learning process of the image processing apparatus 10 is initiated, the transformation parameter retaining unit 106 initializes all the parameters {wi, bi, w′i, b′i} with random numbers first in S201. Next, in S202 and S203, the image acquisition units 101a to 101c read the low quality images of the image capture systems from the learning case image retaining unit 109, and perform a transformation in such a manner as to substantially align the subjects in the low quality images by correcting the distortion, size, and aspect of each image. The determined transformation parameters for image correction are saved in the transformation parameter retaining unit 106. The same values are also used at the time of recognition. The image acquisition units 101a to 101c return the corrected images to the learning case image retaining unit 109 and save them therein.
Unlike Japanese Patent No. 4942221 where low quality images are averaged, and the average is the final result, this image correction does not always need strict agreement at the pixel level. For example, if there is a displacement between images, when they are the same in size and aspect, the displacement manner is systematic. In other words, the same edge between the low quality images is always shifted by the same displacement. In this case, a filter whose center of the receptive field is displaced is learned for each of the feature transformations 308a to 308c of the low quality images in
Moreover, if a specific low quality image is distorted, or the low quality images are slightly different in aspect and size, a displacement occurs non-systematically. In such a case, filters whose centers of the receptive fields are blurred are learned for the feature transformations 308a to 308c. The size of the blur is proportional to a standard deviation of the non-systematic displacement between the low quality images. In this manner, the displacement between the low quality images is robustly absorbed by the filter of the NN.
The next processes from S204 to S210 are a loop to update a parameter to reduce an error in such a manner as to bring a transformation result close to a target image. The loop is repeated a predetermined number of times. Firstly, in S205, the image acquisition units 101a to 101c select n sets of low quality images. Next, in S206, the high quality image acquisition unit 107 acquires n high equality images on the same focal plane from the same viewpoint corresponding to each set of low quality images.
Next, in S207, the feature transformation units 102a to 102c, the feature integration unit 103, and the image quality improvement transformation unit 104 perform the transformation process in a predetermined order, using the values of the parameters for the transformation obtained up to this point. Consequently, the n high quality images are generated from the n sets of low quality images. Next, in S208, the image comparison unit 108 compares the generated n high quality images with n target high quality images, and calculates errors. A sum of the errors between the images is defined by the following squared error.
L=1/nΣj1/mjΣcΣxΣy∥I*jc(x,y)−I{circumflex over ( )}jc(x,y)∥2
I*j is a true value of the j-th high quality image among the n data sets. I{circumflex over ( )}j is a high quality image generated from the j-th set of low quality images. mj is a total number of pixels of the j-th image. c is an element representing three channels of RGB.
In S209, the image comparison unit 108 transmits error values to the feature transformation units 102a to 102c and the image quality improvement transformation unit 104. The parameters {wi, bi, w′i, b′i} are changed by the following equation in such a manner as to minimize the error values.
W(t+1)=W(t)−η∂L/∂W(t)
W represents any of the parameters. η is a learning coefficient. The error gradient term, ∂L/∂W, is sequentially calculated from the final layer, using the backpropagation technique. When there is a connection between a neuron i in a certain layer and a neuron j in a layer higher than the certain layer, the gradient of an error of a parameter Wij of a weight of a connection is obtained by the following equation.
∂L/∂Wij=(∂L/∂Zj)(∂Zj/∂Wij)
Zj is an output value of the neuron j.
The above update process is repeated a predetermined number of times. In S211, the feature transformation units 102a to 102c, the feature integration unit 103, and the image quality improvement transformation unit 104 then save all the parameters of the network in the transformation parameter retaining unit 106, and ends the process. There are also various derivative forms such as a type where a term called an inertial term or a weight attenuation term is added to the above learning update equation. The description of the process at the time of learning is finished here.
Next, a qualitative description is given of a mechanism for improvement in image quality of the embodiment. In a schematic diagram of
The image processing apparatus 10 transforms each image into features through the NN. An area of each image is decomposed into abstract high dimensional features related to the appearance of a subject, such as the shape of a contour and the texture of a surface. Next, the image processing apparatus 10 connects and integrates features 502a to 502c of the plurality of images. The images of the different attributes are then aggregated at a feature level. Next, the case-based learning allows the high dimensional feature vector and a pattern 504 of a local area of a high quality image 503 being a post-transformation target to be associated and stored. Furthermore, all the transformation parameters of the layers of the NN are optimized through the learning. In such a mechanism, the image processing apparatus 10 of the embodiment can generate the high quality image 503 using the group of low quality images of the different attributes.
In order to further clarify the mechanism of the embodiment, a known image quality improvement method is compared here. Known methods that use different attribute images complementarily include a method described in Qiong Yan, Xiaoyong Shen, Li Xu, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Jiaya Jia, Cross-Field Joint Image Restoration via Scale Map. IEEE International Conference on Computer Vision (ICCV), 2013. In this method, edges are extracted from an infrared image captured by an infrared flash, and edge-preserving smoothing are performed on a visible light image using the edges. Such a method requires to adjust the position of an edge and individually handle noise such as a missing edge and a pseudo-edge of the infrared image.
In contrast, in the embodiment, a case is considered in which there are a discrepancy in edge between low quality images and a discrepancy such as fringes of color due to aberration. Local patches of a low quality image include partial areas of various subjects. Many patches of similar contours and textures are extracted from the same subject such as a sky or wall. If a discrepancy between two different types of images is systematic, and the contours are always displaced by the same amount in the same direction, then there are multiple patches including similar displacements of contours. When an NN transforms local patches into features (coding), the weights of a convolution operation are learned also taking their displacements as a part of the pattern. Hence, the discrepancy between the images is absorbed by the feature transformation of the NN.
Furthermore, assume that the direction and amount of displacement between different types of images are not constant, and a displacement occurs by a different amount in a different direction according to the position in the image, the distance to the subject, and the like. In this case, it is difficult to absorb a discrepancy with one convolution operation. However, there is a possibility that a plurality of variations in the above displacement is individually learned, and coding can be appropriately performed as the contour through feature transformations in the plurality of layers of the NN.
As described above, in the embodiment, a discrepancy between different types of images is absorbed by learning to enable complementary use of each image. As in the above-mentioned paper of Qiong Yan et al., a user does not need to individually handle discrepancies between the different types of images.
Moreover, the known method such as Japanese Patent No. 4942221 is further compared. In Japanese Patent No. 4942221, a user gives the value of an image to be synthesized on a virtual focal plane. The positions of a group of images are displaced accordingly, and the images are superimposed and averaged. In contrast, in the embodiment, the above-mentioned process is not performed. The image processing apparatus 10 of the embodiment deals with an image having a different focal plane as another learning case even if a subject is the same. In learning, a correspondence relationship between a feature of a low quality image and a local area of a high quality image is stored according to the case where the focal plane is different. Hence, when a group of low quality images captured on a certain focal plane is provided as inputs, the image processing apparatus 10 transforms the images in such a manner as to reproduce a high quality image captured on the corresponding focal plane.
In this manner, the image processing apparatus 10 performs the image quality improvement process on a case basis. Hence, in terms of the types of image quality improvement process, not limited to resolution enhancement but various modes can be achieved. Various image quality improvement processes such as dark current noise reduction, color correction, sharpening, a change in depth of field, removal of a blur, and removal of a fog and a haze can be achieved.
However, in order to perform these processes with high accuracy, it is required that a low quality image carries necessary information to distinguish a signal and noise, and that a wide selection of cases is prepared to learn cases. In order to, for example, remove a fog and a haze, it is desired to have a multi-band (for example, infrared) image that can capture a subject hidden in the fog to some degree. Moreover, in order to reduce dark current noise, a wide selection of case sets that allow learning of a difference between a dark current and a normal pattern is required.
(Modifications)
The image processing apparatus 10 may be configured as in
Moreover, in the first embodiment, a deconvolution operation based on learning is used to obtain a high image quality feature. However, as illustrated in
Moreover, as another modification, as illustrated in
Moreover, as another modification, the image processing apparatus 10 may be in a mode that regards an image itself as features, connects all channels of images of different attributes in the first stage, and then performs the image quality improvement transformation by a convolution.
Moreover, as another modification, the image processing apparatus 10 may combine a pooling process. Pooling is a process of aggregating features of a plurality of pixels into a feature of one pixel, and has an effect that the response of a neuron is robust against the distortion and displacement of a pattern. However, the spatial resolution of a feature is reduced. In order to complement it, the image processing apparatus 10 may enlarge a feature again by linear interpolation or the like after pooling, and restore the resolution by a deconvolution operation.
Moreover, in the embodiment, the transformation parameters of a convolution and a deconvolution are constant irrespective of the position in an image. However, different parameters may be used according to the position in an image. In this modification, it can be expected to effectively learn and remove the causes of degradation such as specific aberration and reduction in peripheral brightness, which are likely to occur at an end of an image.
Moreover, as another modification, the image processing apparatus 10 may use, as low quality images, image data by special imaging techniques called, for example, coded imaging and compressed sensing. For example, Levin, A., Fergus, R., Durand, F., and Freeman, W. T., Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26, 3 (2007), 70 illustrates an example where when an image captured with an aperture of a special mask pattern called a coded aperture is used, the depth estimation of a subject is facilitated. As a realization example of a low quality image acquired by the image acquisition unit 101, image data as in the document may be used. When such image data and a normal luminance image are combined, images having different properties can be complementarily used. Accordingly, the accuracy can be expected to be improved.
Likewise, for example, Dengyu Liu, Jinwei Gu, Yasunobu Hitomi, Mohit Gupta, Tomoo Mitsunaga, Shree K. Nayar: Efficient Space-Time Sampling with Pixel-Wise Coded Exposure for High-Speed Imaging. IEEE Trans. Pattern Anal. Mach. Intell. 36(2): 248-260 (2014) illustrates a technology for controlling image capture timings individually according to image capture devices, capturing n images, and obtaining a moving image of n frames having different variations. It may be a mode that inputs such an image as image data having n channels as one of low quality images of the embodiment.
Moreover, in the embodiment, it is assumed that low quality images are adjusted in focal plane and captured. As a modification, it may be configured in such a manner that one of the low quality images is set as a reference image, the focal plane is operated to perform image capture, and the focal plane of the remaining low quality images is always fixed. Images other than the reference image are images captured at a large depth of field (what is called deep focus). In the modification, information on the focal plane is used as a clue from the reference image, and only information on appearance is used from the remaining images. Consequently, as in the previous modes, a high quality image such as captured on a focal plane corresponding to a focal plane of an input image is generated.
In the first embodiment, the configuration is described in which the viewpoint positions of each low quality image obtained by the image acquisition unit of the image processing apparatus 10 and a high quality image obtained by transforming the low quality images are the same. In a second embodiment, a configuration is described in which the viewpoint positions of each low quality image and a high quality image are different. The embodiment has a simpler configuration since the embodiment does not need the splitting of a light ray, and the like, unlike the first embodiment. On the other hand, parallax that does not occur in the first embodiment occurs between images. Accordingly, more complicated image transformation parameters need to be learned.
The image processing apparatus 10 of the embodiment is assumed to include an image capture unit 1000 as a hardware configuration.
Next, information processing at the time of learning in the embodiment is described using
Firstly, when the learning process is initiated, the learning case image retaining unit 109 captures an image of a subject for learning with the high quality image capture device, and saves the image in S301. Next, in S302, the learning case image retaining unit 109 captures images of the same subject with the two low quality image capture devices 1011a and 1001b, and saves the images. At this point in time, the learning case image retaining unit 109 performs image capture a plurality of times while displacing the positions of viewpoints of the low quality image capture devices 1011a and 1001b by a predetermined displacement in a horizontal direction. Consequently, high quality images of various subjects and a plurality of low quality images obtained by capturing the same subjects in different parallaxes are saved in the learning case image retaining unit 109.
Next, S303 to S307 are a loop to learn and update each transformation parameter. In the loop, a variable a is incremented by 0.1 from zero to one. In S304, the high quality image acquisition unit 107 selects and acquires n high quality images from the learning case image retaining unit 109. Next, in S305, the image acquisition units 101a and 101b each select and acquire n images corresponding to the high quality images from the learning case image retaining unit 109. At this point in time, the image acquisition units 101a and 101b each select images captured at a position of a displacement αρi according to the value of α, respectively, as illustrated in
Consequently, images from the same viewpoint positions are given as learning images in the early stage of learning, and images captured from true viewpoint positions are gradually given with the progress of learning. Consequently, learning is made simple to prevent falling into a local solution and the like. In S306, the feature transformation units 102a and 102b and the image quality improvement transformation unit 104 update the transformation parameters. This step includes S306a to S306c. This process is the same as the process operation (S207 to S209 of
Consequently, images from the same viewpoint positions are given as learning images in the early stage of learning. Images captured from true viewpoint positions of the image capture unit 1000 of the image processing apparatus 10 are gradually given with the progress of learning. Consequently, learning is made simple, and it is possible to prevent falling into a local solution and the like. When learning in α=1.0 is finished, the feature transformation units 102a and 102b and the image quality improvement transformation unit 104 save the transformation parameters of the NNs in the transformation parameter retaining unit 106 in S309. For example, the feature transformation units 102a and 102b and the image quality improvement transformation unit 104 then end the process of the flowchart illustrated in
Next, a qualitative description is given of the mechanism for improvement in image quality of the embodiment. When images are captured by a plurality of low quality image capture devices in different viewpoint positions, parallax occurs in an image group 511 and an image group 521 according to the distances to a subject as illustrated in
Let feature representation patterns obtained by transforming each image in the image group 511 into features be PL, Q, and RR. When images of the subject are captured in a near distance, further displaced patterns PLL, Q, and RRR are obtained as illustrated in
At the time of learning, the feature representation vectors (PL, Q, RR) are stored associated with an appearance 504 in a local area of a high quality image 513. The feature representation vectors (PLL, Q, RRR) are associated with an appearance 505 when the same local area is captured in a near distance. In this manner, each pattern is associated with the high quality image and stored.
When the feature representation vectors (P, Q, R), (PL, Q, RR), and (PLL, Q, RRR), . . . , of neurons indicate response patterns different from each other, it is difficult for a finite number of neurons to associate and store every low quality image and the high quality image.
However, the vectors of responses of neurons for the sets of patterns {P, PL, PLL} and {R, RR, RRR} are a one-dimensional manifold. A set of the vector groups {(P, Q, R), (PL, Q, RR), (PLL, Q, RRR)} is also a one-dimensional manifold. A response pattern of a neuron is a high-dimensional manifold also having variations in shape other than displacement. Here, the topic, however, is limited only to displacement for simplicity.
An NN is publicly known as a technique that allows a manifold pattern to be efficiently learned, and especially an autoencoder to reduce the dimension of a manifold pattern is widely known. The image processing apparatus 10 of the embodiment reduces the dimension of a high-dimensional input image pattern to features of the NN, ties the features to an output image pattern, and decodes them back to the image. This can regard the input and output data as a type of autoencoder that is asymmetrical between low image quality and high image quality.
As a result of the above, even if the subject in the images is displaced due to parallax as in
In the second embodiment, the mode is described in which viewpoint positions of each low quality image and a high quality image are not the same. In the above embodiment, if the viewpoint positions are apart, and parallax occurs to such an extent as to exceed the size of the receptive field of a neuron, images cannot be associated. Accordingly, there is a problem reducing the accuracy. When a CNN includes more layers, the receptive field of a neuron in a higher layer can be increased in size. However, an NN having multiple layers has a difficulty in learning. Hence, a process to solve such a problem, and the like are disclosed in the embodiment; meanwhile, it is illustrated that various modes including, but not limited to, the operation of connecting features, can be taken by the feature integration unit in the embodiment.
The image processing apparatus 10 of the embodiment includes a plurality of image acquisition units that acquire low quality images from different viewpoint positions as in the second embodiment. The purpose of the process of the image processing apparatus 10 is assumed to create an aesthetically pleasing image with a small depth of field such as captured with a large aperture lens, from a plurality of low quality images acquired. In the embodiment, such a high quality image can be obtained also from an optical system whose physical size is small or configuration is simple.
The functional configuration of the image processing apparatus 10 is as illustrated in
The information processing of the image processing apparatus 10 is described using
Next, S407 to S410 are a loop of a process of performing M different feature transformations on the set of low quality images obtained in the previous stage. Here, the M feature transformations correspond to cases of M different parallaxes, respectively. In S408, the feature transformation units 1002x to 1002z assume that there is a subject on a focal plane at a distance dj, and displace the images by a predetermined amount in a predetermined direction in such a manner that the positions of all the images agree on the focal plane, and superimpose the images. The N superimposed images are represented in a symbol I(j) in
Next, in S409, a convolution operation unit included in the feature transformation unit 1002 performs a convolution operation on the image I(j) to obtain a feature f1(j) of the NN. Next, in S411, the feature integration unit 1003 obtains a distance map D* from the subject distance acquisition unit 1007. In terms of the distance map, it is assumed that the value of a distance of each pixel is obtained separately by any means such as an active pattern irradiation method at the same time as image capture. The realization means of the subject distance acquisition unit 1007 does not matter here.
Next, in S412, the feature integration unit 1003 selects one of M types of features f1(1) (x, y) for each pixel on the basis of a value d*(x, y)∈D* of the distance map, and obtains a feature f2(x, y). A more specific description is given. The feature f1(j)(x, y) corresponding to the distance dj closest to d*(x, y) is set as a feature of f2(x, y). When expressed mathematically, it is as follows:
f2(x,y)=f1(j{circumflex over ( )})(x,y),
j{circumflex over ( )}=argminj|d*(x,y)−dj|
Next, in S413, the image quality improvement transformation unit 1004 performs a convolution operation on the feature f2, and generates a high quality image. Lastly, in S414, the image output unit 1005 outputs the generated image. The image output unit 1005 then ends the processes of flowcharts of
In the embodiment, a deconvolution operation is not used unlike the first and second embodiments. This is because the purpose of the embodiment is not to improve the resolution of an image but to obtain an aesthetically pleasing high quality image such as captured with a large aperture lens.
In the embodiment, the distance to a subject is obtained on a pixel by pixel basis. However, it is generally difficult to accurately measure the distance to a subject. Hence, a mode that does not require the estimation of the distance to a subject is disclosed next as a modification.
f2(x,y)=(1/Z)ΣjlCOR(j)(x,y)f1(j)(x,y),Z=ΣjlCOR(j)(x,y)
Here, lCOR(j) represents the likelihood of whether or not the same point of the subject at the distance dj is projected into the image set I(j).
The processes of S423 and S424 are similar to those of S413 and S414.
Next, the operation of the corresponding point determination unit 1008 that obtains lCOR(j) is described in detail. This is an extension of a method of J. Zbontar and Y. LeCun, Computing the Stereo Matching Cost with a Convolutional Neural Network. Proceedings of 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015 to a multi-view stereo mode. As illustrated in a schematic diagram of
Moreover, considering, here, that the determination of the likelihood of a corresponding point is not always successful, not the likelihood itself but the output of a previous layer (indicated by a symbol L5 in
Furthermore, some other modifications are described here. It is illustrated here that the mode of the third embodiment is applicable to various purposes.
As another modification, the image processing apparatus 10 that uses an image capture unit with a configuration such as illustrated in
As still another modification, for example, a transformation may be performed in such a manner as to obtain an RGB color image such as captured frontlit, from an RGB color image of a person captured backlit, and an infrared image captured using the infrared flash as illustrated in a schematic diagram of
In terms of the low quality images, as long as they agree with the purpose of the image quality improvement, images of various attributes can be used as follows.
As still another modification, for example, an image with a wide dynamic range of brightness (an HDR image) may be generated from a group of low quality images captured by low quality image capture devices from a plurality of viewpoints, the group of low quality images having been exposed to light for different periods of time, as illustrated in a schematic diagram of
As still another modification, an image captured by a low-field MRI apparatus and an image of a different attribute, such as an ultrasound image or X-ray image, obtained by capturing the same subject are combined, as illustrated in a schematic diagram of
As still another modification, a plurality of results may be output at the same time. As illustrated in a schematic diagram of
As described above, in the embodiment, the mode has been described in which low quality images from different viewpoint positions are integrated to generate a high quality image. Furthermore, it has been illustrated as the mode of the feature integration unit 103 that a selection on a pixel by pixel basis and a weighted sum (weighted average) of each pixel are possible other than the connection of features illustrated in the first embodiment. In this manner, various modes are conceivable for the feature integration unit 103, and the feature integration unit 103 is not limited to a specific mode.
In this embodiment, a mode is described in which a connectivity relationship of an NN is determined on the basis of a geometric positional relationship of low quality image capture devices. The functional configuration diagram is the same as
There is a geometric relationship called an epipolar constraint illustrated in
It is assumed here that objects a to c being subjects are arranged in front of the image capture unit. At this point in time, the objects a to c in low quality images are arranged on a straight line due to an epipolar constraint as illustrated in
In the image processing apparatus 10 of the embodiment, the connectivity relationship of each layer of the NN is designed as follows, using the correspondence relationships between image areas: As illustrated in
The connectivity relationship between a pixel (x′, y′) of each low quality image and a neuron (x, y) is set.
As described above, the input relationship between each neuron and an image can be uniquely determined by an epipolar constraint and parallax. However, a homography matrix is obtained as described below to accurately obtain a correspondence relationship in reality. Assuming a point Xπ and a plane π passing the point Xπ as illustrated in
x=H1Xπ,
x′=H2Xπ
x′=H2Xπ=H2H1−1x=Hx
It is known that when four or more points corresponding between two images are obtained, the homography matrix H is obtained. Hence, calibration boards or the like are presented vertically to the optical axis at positions of parallaxes 1 to M. Four or more corresponding points between the images are extracted to obtain the matrix H between the images. As a result, a connectivity relationship of the NN that associates each pixel of the low quality images from the plurality of viewpoints with each pixel of features f1(l) to f1(M) of the M parallaxes.
When the image processing apparatus 10 of the embodiment initiates its operation, N low quality images are acquired. The feature transformation units 1 to M transform the N low quality images into the features f1(l) to f1(M) of the NN. Furthermore, the feature transformation units 1 to M perform a convolution operation on the features f1(1) to f1(M) to obtain features f2(1) to f2(M) of the NN. f2(1) to f2(M) are features that represent what kind of appearance pattern such as an edge and texture is present at each position when the space is divided into M levels from near to far.
Furthermore, the feature integration unit 1003 of
As described above, the connectivity relationship between a low quality image and a first layer of the NN is determined on the basis of a geometric constraint condition between images. On the other hand, such a connectivity relationship may be acquired automatically. One example is cited below. (1) Firstly, the image processing apparatus 10 prepares fully connected weights between all pixels of an input image and neurons in the first layer of the NN to initialize them with random numbers. (2) The image processing apparatus 10 presents calibration patterns whose examples are illustrated in
Moreover, as still another modification, after acquiring the above connectivity relationship of the NN, the image processing apparatus 10 may further adjust weights in all the layers from the final output layer to the input layer by the backpropagation method on the basis of learning images. In this manner, the information processing of the image processing apparatus 10 of the embodiment indicates to determine the connectivity relationship of an NN from a geometric positional relationship of image capture devices, and to learn the weight of each connection with case images.
In a fifth embodiment, a description is given of a mode in which images acquired from multiple low quality image capture devices are integrated in stages. The purpose of the image processing apparatus 10 is to improve image quality as in the third embodiment. A wide selection of targets for improvement in image quality are applicable here as in the third embodiment. An image processing apparatus of the embodiment has an appearance of an image capture unit of a multi-lens camera such as illustrated in
In terms of learning in the embodiment, as in the previous learning, a high quality image captured from a virtual viewpoint position is provided as a target image, and transformation parameters in all the stages are learned in such a manner as to generate the target image from low quality images. This is simply required to learn while propagating an error through all the three layers from the final layer to the first layer by the backpropagation method.
As a modification of the embodiment, a description is given of a mode in which integration is performed in stages on images of an image capture device array as in
Furthermore, a description is given of a second derivative mode of stepwise integration of images of the image capture device array. A mode in which learning and stacking is performed in stages illustrated here. A description is given below using schematic diagrams of
The configuration of an NN equal to one stage for image integration is similar to the previous embodiments. As illustrated in
Moreover, still another modification is described here. What is described here is a modification in which a plurality of virtual viewpoint positions of high quality images can be set as illustrated in
Moreover, as still another modification, a configuration to which a measure against an occlusion upon imaging is added is disclosed. A manner for obtaining a high quality image robustly even in a situation where some cameras of an image capture device array are occluded by a finger of a photographer as illustrated in
For example, as one of the modes, the image processing apparatus 10 separately includes an occluded image determination unit that makes a determination on whether or not an image is occluded. Upon integration by the feature integration unit 103, features of images determined to be occluded are removed for integration. Calculations for the integration of features at this point in time are assumed to be the operation of averaging values according to the dimension. Moreover, if the occluded image determination unit outputs the value of likelihood of occlusion, the feature integration unit 103 may integrate features not by averaging but by the weighted sum method as in the following equation.
fi+1(k)(x,y,d)=(1/Σj∈NklOCCi(j)Σj∈NklOCCi(j)fi(j)(x,y,d),lOCCi+1(k)=Σj∈NklOCCi(j)
fi(j) is the j-th feature in the i-th stage, Nk is a set of features to be integrated into the k-th feature in each stage, and lOCCi(j) is the likelihood of whether or not the image feature j is an occluded image.
As an embodiment of the occluded image determination unit here, for example, discrimination learning is performed by a discriminator such as a support vector machine, using features of NNs of each image. Alternatively, an averaged image obtained by averaging all captured images and each image may be compared with a normalization function to determine that an image having a low similarity to the averaged image is occluded.
Moreover, as still another modification, it may be configured in such a manner that the determination unit for an occluded state is not provided. For example, the image processing apparatus 10 captures multiple low quality images occluded by a finger of a photographer. The image processing apparatus 10 then learns given a sufficient amount of such case data. As a result, a transformation parameter is obtained which does not respond to a feature obtained from an occluded image but responds to a feature obtained from a non-occluded image, and performs a transformation in each stage.
Up to this point the embodiment in which images acquired by image capture devices are integrated in stages in the image processing apparatus 10 of the embodiment including multiple low quality image capture devices has been described. The points illustrated in the embodiment are as follows: (1) The connectivity relationship of an NN is determined on the basis of an optical placement relationship of the low quality image capture devices, and (2) A transformation parameter of the determined connection is learned by case learning and adjusted. The above configuration is a novel configuration that does not included in a known image processing apparatus 10 for improvement in image quality.
Up to this point the preferred embodiments of the present disclosure have been described in detail. However, the present disclosure is not limited to such specific embodiments. For example, the whole or part of the functional configuration of the above-mentioned image processing apparatus 10 may be implemented as a hardware configuration on the image processing apparatus 10. Moreover, the above-mentioned embodiments and modifications of the embodiments may be combined as appropriate, and implemented.
As described above, according to the processes of the above-mentioned embodiments, features of abstract representations such as shape and texture are extracted from images. Moreover, image information from a plurality of viewpoints or of a plurality of attributes is integrated at a feature level, unlike the known methods. Furthermore, an image as a result of image quality improvement is estimated and synthesized from the integrated features, on the basis of the case learning method. As a result, there are the following effects as compared to the known methods: (1) A deformation and alignment is not required for causing a subject in an image to match substantially at the pixel level. (2) Images having different characteristics, such as an infrared image and a depth image, can be complementarily used. (3) Systematic noise that are difficult to be removed by image averaging can also be removed. (4) Not only resolution enhancement but also various types of image quality improvement processes can be achieved by preparing learning images adequate for the purpose and learning the cases.
Hence, a higher quality image can be obtained from a plurality of images.
Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present disclosure has been described with reference to exemplary embodiments, the scope of the following claims are to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2016-228042, filed Nov. 24, 2016, which is hereby incorporated by reference herein in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2016-228042 | Nov 2016 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20050069207 | Zakrzewski | Mar 2005 | A1 |
| 20060018566 | Coleman | Jan 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| 4942221 | May 2012 | JP |
| Entry |
|---|
| Jure Zbontar; Computing the Stereo Matching Cost with a Convolutional Neural Network. |
| Qiong Yan; Cross-Field Joint Image Restoration via Scale Map; The Chinese University of Hong Kong. |
| Dengyu Liu; Efficient Space-Time Sampling with Pixel-wise Coded Exposure for High Speed Imaging; IEEE Transactions on Pattern Analysis and Machine Intelligence pp. 1-14. |
| Anat Levin; Image and Depth from a Conventional Camera with a Coded Aperture; Massachusetts Institute of Technology, Computer Science and Artificial Intelligence Laboratory. |
| Chao Dong; Learning a Deep Convolutional Network for Image Super-Resolution; Department of Information Engineering, The Chinese University of Hong Kong Microsoft Research pp. 1-16. |
| Alexey Dosovitskiy; Learning to Generate Chairs with Convolutional Neural Networks; Department of Computer Science, University of Freiburg; pp. 1539-1546. |
| Number | Date | Country | |
|---|---|---|---|
| 20180144447 A1 | May 2018 | US |
