The present invention relates to the technical field of image processing, in particular to a binocular image quick processing method and apparatus and a corresponding storage medium.
Due to existence of disparity in binocular vision, images formed between monocles have displacement. For example, in a horizontal transverse direction, displacement of a pixel of one picture in a monocular image relative to a corresponding pixel of the other picture is a horizontal visual phase difference of binocular visual images.
Sizes of objects in existing binocular visual images may be different, which leads to a larger difference in feature precision of object features corresponding to objects of different sizes when analyzing image features, thus leading to a larger difference of an acquired binocular visual image phase difference. The accuracy of the binocular visual image phase difference is also lower, which makes it impossible to effectively process the corresponding images.
Therefore, it is necessary to provide a binocular image quick processing method and apparatus to solve problems existing in the prior art.
Embodiments of the present invention provide a binocular image quick processing method and apparatus that can quickly and accurately acquire a binocular visual image difference, to solve the technical problems of a larger difference and lower accuracy of the binocular visual image phase difference acquired by an existing binocular image quick processing method and apparatus.
An embodiment of the present invention provides a binocular image quick processing method, including:
An embodiment of the present invention further provides a binocular image quick processing apparatus, including:
An embodiment of the present invention further provides a computer readable storage medium, in which a processor executable instruction is stored. The instruction is loaded by one or more processors to execute any of the above binocular image quick processing methods.
Compared with a binocular image quick processing method and apparatus of the prior art, the binocular image quick processing method and apparatus of the present invention can acquire the difference features of the first-level left eye image and the first-level right eye image in different dimensions and the corresponding estimated phase differences through the next-level left eye images and next-level right eye images of the plurality of different dimensions, so that the estimated phase difference of the first-level left and right eye images can be quickly and accurately acquired, thereby improving a corresponding image processing efficiency; and the technical problems of a larger difference and lower accuracy of a binocular visual image phase difference acquired by the existing binocular image quick processing method and apparatus are effectively solved.
The technical solutions in the embodiments of the present invention will be described below clearly and completely with reference to accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, but not all the embodiments. On the basis of the embodiments in the present invention, all other embodiments obtained by those skilled in the art without inventive efforts fall within the protection scope of the present invention.
The binocular image quick processing method and apparatus of the present invention use an electronic device used for performing quick and accurate phase difference estimation of binocular images. The electronic device includes, but is not limited to a wearable device, a headset device, a medical and health platform, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant (PDA), and a media player), a multiprocessor system, a consumer electronic device, a minicomputer, a mainframe computer, a distributed computing environment including any of the above systems or devices, and so on. The electronic device is preferably an image processing terminal or an image processing server that performs image processing on the binocular images, so as to perform effective image processing by using an acquired binocular visual image phase difference.
Please refer to
An image processing flow of the binocular image quick processing method of the present embodiment is described in detail below.
In step S101, a binocular image quick processing apparatus (such as an image processing terminal) may acquire the first-level left eye image captured by a binocular camera and the corresponding first-level right eye image. The first-level left eye image and the corresponding first-level right eye image may be combined into a 3d scenario of the corresponding image.
In step S102, because sizes of scenario objects contained in the first-level left eye image and the first-level right eye image are different, in order to better perform feature recognition on the scenario objects with different sizes, the binocular image quick processing apparatus performs the folding dimensionality reduction operation on the first-level left eye image to acquire the plurality of next-level left eye images corresponding to the first-level left eye image, such as four second-level left eye images; and four third-level left eye images may be acquired if the folding dimensionality reduction operation is continued being performed on the second-level left eye image.
For details, please refer to
In a similar way, the binocular image quick processing apparatus may also perform the folding dimensionality reduction operation on the first-level right eye image to acquire the plurality of next-level right eye images corresponding to the first-level right eye image, such as four second-level right eye images; and four third-level right eye images may be acquired if the folding dimensionality reduction operation is continued being performed on the second-level right eye image.
The setting of left and right eye images with different levels or resolutions may better meet needs of object receptive fields in different scenarios.
In step S103, the binocular image quick processing apparatus performs feature extraction on the plurality of next-level left eye images (such as the second-level left eye images and the third-level left eye images) acquired in step S102 by using the first preset residual convolutional network to obtain the plurality of next-level left eye image features at different levels.
Meanwhile, the binocular image quick processing apparatus performs feature extraction on the plurality of next-level right eye images acquired in step S102 by using the first preset residual convolutional network to obtain the plurality of next-level right eye image features at different levels.
In step S104, the binocular image quick processing apparatus performs phase difference distribution estimation on the next-level left eye image feature and the next-level right eye image feature of each level. That is, the possible phase difference on each point in the next-level left eye image feature and the next-level right eye image feature is evaluated to obtain a possibility of a certain phase difference value appearing at this point, namely, feasible distribution of an effective phase difference interval on a certain feature point. Later, the most possible phase difference value of this feature point is obtained through the analysis of this distribution.
When a probability corresponding to the most possible phase difference value of each point in the next-level left eye image feature and the next-level right eye image feature is maximum, an image phase difference distribution estimation feature under this level is obtained.
In step S105, the binocular image quick processing apparatus performs fusion on the next-level image phase difference distribution estimation feature acquired in step S104 and the next-level left eye image feature of the corresponding level acquired in step S103 to obtain the next-level fusion feature. The fusion here may be feature superposition of the next-level image phase difference distribution estimation feature and the next-level left eye image feature of the corresponding level. The fusion operation of the next-level left eye image feature may reduce an impact of an initial difference of the next-level left eye image, improve the accuracy of the subsequent feature extraction operation, and thus improve the accuracy of the subsequent difference feature.
In step S106, the binocular image quick processing apparatus performs feature extraction on the next-level fusion feature acquired in step S105 by using the second preset residual convolutional network to acquire the difference feature of the next-level left and right eye images of the corresponding level.
In step S107, the binocular image quick processing apparatus obtains the estimated phase difference of the next-level left and right eye images based on the acquired difference feature of the next-level left and right eye images. That is, the estimated phase difference of the corresponding next-level left and right eye images is determined based on the preset estimated phase difference corresponding to the difference feature of the next-level left and right eye images. If the preset estimated phase difference corresponding to the difference feature of the next-level left and right eye images is larger, the estimated phase difference of the next-level left and right eye images obtained correspondingly is also larger. If the preset estimated phase difference corresponding to the difference feature of the next-level left and right eye images is smaller, the estimated phase difference of the next-level left and right eye images obtained correspondingly is also smaller. The preset estimated phase difference may be acquired through model training of positive and negative samples.
In step S108, the binocular image quick processing apparatus performs tiling dimensionality raising operation on the difference feature of the next-level left and right eye images acquired in step S106 to obtain the corrected difference feature of the first-level left and right eye images; and the binocular image quick processing apparatus performs the tiling dimensionality raising operation on the estimated phase difference of the next-level left and right eye images acquired in step S107 to obtain the corrected phase difference of the first-level left and right eye images.
For example, the binocular image quick processing apparatus may perform the tiling dimensionality raising operation on a difference feature of third-level left and right eye images to obtain a corrected difference feature of second-level left and right eye images, and the corrected difference feature of the second-level left and right eye images may be used to calculate a difference feature of the second-level left and right eye images; and then the binocular image quick processing apparatus performs the tiling dimensionality raising operation on the difference feature of the second-level left and right eye images to obtain the corrected difference feature of the first-level left and right eye images.
For details, please refer to
In a similar way, the binocular image quick processing apparatus may perform the tiling dimensionality raising operation on an estimated phase difference of the third-level left and right eye images to obtain a corrected phase difference of the second-level left and right eye images, and the corrected phase difference of the second-level left and right eye images may be used to calculate an estimated phase difference of the second-level left and right eye images; and then the binocular image quick processing apparatus performs the tiling dimensionality raising operation on the estimated phase difference of the second-level left and right eye images to obtain the corrected phase difference of the first-level left and right eye images.
In step S109, the binocular image quick processing apparatus performs feature fusion according to the first-level left and right eye feature data such as the first-level left eye image and the corresponding first-level right eye image acquired in step S101, the corrected difference feature of the first-level left and right eye images acquired in step S108, and the corrected phase difference of the first-level left and right eye images acquired in step S108, and obtains the estimated phase difference of the first-level left and right eye images based on the fusion feature. The corresponding relationship between the fusion feature and the estimated phase difference of the first-level left and right eye images may be acquired through model training of the positive and negative samples.
In step S110, the binocular image quick processing apparatus performs the image processing operation on the corresponding images by using the estimated phase difference of the first left and right eye images acquired in step S109, such as synthesizing the binocular images into a corresponding three-dimensional scenario image, or performing three-dimensional image change operation on a monocular image.
In this way, a binocular image quick processing process of the binocular image quick processing method of the present embodiment is completed.
According to the binocular image quick processing method of the present embodiment, the difference features of the first-level left eye image and the first-level right eye image in different dimensions and the corresponding estimated phase differences can be acquired through the next-level left eye images and next-level right eye images of the plurality of different dimensions, so that the estimated phase difference of the first-level left and right eye images can be quickly and accurately acquired, thereby improving a corresponding image processing efficiency.
Please refer to
An image processing flow of the binocular image quick processing method of the present embodiment is described in detail below.
In step S401, a binocular image quick processing apparatus may acquire the first-level left eye image captured by a binocular camera and the corresponding first-level right eye image. The first-level left eye image and the corresponding first-level right eye image may be combined into a 3d scenario of the corresponding image.
In step S402, the binocular image quick processing apparatus performs the folding dimensionality reduction operation on the first-level left eye image to acquire the plurality of next-level left eye images corresponding to the first-level left eye image, such as four second-level left eye images; four third-level left eye images may be acquired if the folding dimensionality reduction operation is continued being performed on the second-level left eye image; and in turn, the fourth-level left eye images and the fifth-level left eye images are acquired.
In a similar way, the binocular image quick processing apparatus may also perform the folding dimensionality reduction operation on the first-level right eye image to acquire the plurality of next-level right eye images corresponding to the first-level right eye image, such as four second-level right eye images; four third-level right eye images may be acquired if the folding dimensionality reduction operation is continued being performed on the second-level right eye image; and in turn, the fourth-level right eye images and the fifth-level right eye images are acquired.
In step S403, the binocular image quick processing apparatus sets a count value m, and the current count value m is the level number of the next-level image with the lowest resolution, that is, m=5.
In step S404, the binocular image quick processing apparatus performs the feature extraction on the fifth-level left eye images by using the first preset residual convolutional network to obtain a fifth-level left eye image feature. Meanwhile, the binocular image quick processing apparatus performs the feature extraction on the fifth-level right eye images by using the first preset residual convolutional network to obtain a fifth-level right eye image feature.
In step S405, since there is no corrected phase difference of fifth-level left and right eye images, the binocular image quick processing apparatus performs phase difference distribution estimation on the fifth-level left eye image feature and the fifth-level right eye image feature respectively. That is, the possible phase difference on each point in the fifth-level left eye image feature and the fifth-level right eye image feature is evaluated to obtain the possibility of a certain phase difference value appearing at this point, namely, feasible distribution of an effective phase difference interval on a certain feature point. Later, the most possible phase difference value of this feature point is obtained through the analysis of this distribution.
When a probability corresponding to the most possible phase difference value of each point in the fifth-level left eye image feature and the fifth-level right eye image feature is maximum, the fifth-level image phase difference distribution estimation feature is obtained.
In step S406, since there is no corrected difference feature of the fifth-level left and right eye images, the binocular image quick processing apparatus fuses the fifth-level image phase difference distribution estimation feature with the fifth-level left eye image feature to obtain a fifth-level fusion feature. The fusion here may be feature superposition of the fifth-level image phase difference distribution estimation feature and the fifth-level left eye image feature. The fusion operation of the fifth-level left eye image feature may reduce an impact of an initial difference of the fifth-level left eye image, improve the accuracy of the subsequent feature extraction operation, and thus improve the accuracy of the subsequent difference feature.
In step S407, the binocular image quick processing apparatus performs the feature extraction on the fifth-level fusion feature by using the second preset residual convolutional network to acquire the difference feature of the fifth-level left and right eye images.
In step S408, the binocular image quick processing apparatus obtains the estimated phase difference of the fifth-level left and right eye images based on the difference feature of the fifth-level left and right eye images. That is, the estimated phase difference of the corresponding fifth-level left and right eye images is determined based on the preset estimated phase difference corresponding to the difference feature of the fifth-level left and right eye images. If the preset estimated phase difference corresponding to the difference feature of the fifth-level left and right eye images is larger, the estimated phase difference of the fifth-level left and right eye images obtained correspondingly is also larger. If the preset estimated phase difference corresponding to the difference feature of the fifth-level left and right eye images is smaller, the estimated phase difference of the fifth-level left and right eye images obtained correspondingly is also smaller. The preset estimated phase difference may be acquired through model training of positive and negative samples.
In step S409, since there is no corrected phase difference of the fifth-level left and right eye images, the binocular image quick processing apparatus directly takes the estimated phase difference of the fifth-level left and right eye images as a total estimated phase difference of the fifth-level left and right eye images.
In step S410, the binocular image quick processing apparatus performs tiling dimensionality raising operation on the difference feature of the fifth-level left and right eye images acquired in step S407 to obtain the corrected difference feature of the fourth-level left and right eye images; and the binocular image quick processing apparatus performs the tiling dimensionality raising operation on the total estimated phase difference of the fifth-level left and right eye images acquired in step S409 to obtain the corrected phase difference of the fourth-level left and right eye images.
In step S411, the binocular image quick processing apparatus sets the count value m minus one, that is, executes m=m−1, and then returns to step S404.
Specifically, in step S404, the binocular image quick processing apparatus acquires a fourth-level left eye image feature and a fourth-level right eye image feature. In step S405, the binocular image quick processing apparatus performs correction on a fourth-level right eye image by using a corrected phase difference of fourth-level left and right eye images, and performs phase difference distribution estimation respectively on the fourth-level left eye image feature and a corrected fourth-level right eye image feature to obtain a fourth-level image phase difference distribution estimation feature. In step S406, the binocular image quick processing apparatus fuses the fourth-level image phase difference distribution estimation feature, the fourth-level left eye image feature and a corrected difference feature of the fourth-level left and right eye images to obtain a fourth-level fusion feature. In step S407, the binocular image quick processing apparatus obtains a difference feature of the fourth-level left and right eye images. In step S408, the binocular image quick processing apparatus obtains a current-level estimated phase difference of the fourth-level left and right eye images. In step S409, the binocular image quick processing apparatus obtains a total estimated phase difference of the fourth-level left and right eye images based on the current-level estimated phase difference of the fourth-level left and right eye images and the corrected phase difference of the fourth-level left and right eye images. In step S410, the binocular image quick processing apparatus performs tiling dimensionality raising operation on the difference feature of the fourth-level left and right eye images to obtain a corrected difference feature of third-level left and right eye images; and the binocular image quick processing apparatus performs the tiling dimensionality raising operation on the total estimated phase difference of the fourth-level left and right eye images to obtain a corrected phase difference of the third-level left and right eye images.
The binocular image quick processing apparatus sets the count value m minus one again, returns to step S404, and repeats until m=1. At this time, the binocular image quick processing apparatus acquires a corrected difference feature of second-level left and right eye images and a corrected phase difference of the second-level left and right eye images.
Specifically, the specific step that the binocular image quick processing apparatus obtains the total estimated phase difference of the fourth-level left and right eye images based on the current-level estimated phase difference of the fourth-level left and right eye images and the corrected phase difference of the fourth-level left and right eye images please refer to
The preset activation function here can be a segmented reversible TrLU type activation function, that is:
When γ=0, the function is a standard ReLU function, and when γ>0, the function is a LeakyReLU function. The preset activation function can effectively improve significant precision, reduce the model lift and shorten running time in a model training stage and a model deployment stage.
Further, the preset activation function here may further be a convolved TrLU type activation function for phase difference prediction, that is:
Where α and β are an upper boundary and lower boundary of the difference effective interval respectively. The preset activation function is applied to the convolution of a predicted phase difference to be able to improve the precision of an output result.
In step S412, the binocular image quick processing apparatus fuses the first-level left eye image, the first-level right eye image, the corrected difference feature of the second-level left and right eye images and the corrected phase difference of the second-level left and right eye images are fused to obtain a first-level fusion feature. The fusion operation here may be feature superposition of above image features, the corrected difference feature and a corrected difference feature.
In step S413, the binocular image quick processing apparatus performs the phase difference distribution estimation on the first-level fusion feature obtained in step S412 to obtain the estimated phase difference of the first-level left and right eye images. The corresponding relationship between the first-level fusion feature and the estimated phase difference of the first-level left and right eye images may be acquired through model training of positive and negative samples.
In step s414, the binocular image quick processing apparatus performs the image processing operation on the corresponding images by using the estimated phase difference of the first left and right eye images acquired in step s413, such as synthesizing binocular images into a corresponding three-dimensional scenario image, or performing three-dimensional image change operation on a monocular image.
In this way, a binocular image quick processing process of the binocular image quick processing method of the present embodiment is completed.
On the basis of the first embodiment, the binocular image quick processing method of the present embodiment uses the next-level left and right eye images to correct the estimated phase difference of the previous-level left and right eye images of adjacent levels, so that the difference feature of each level and the corresponding estimated phase difference can be accurately fused into the difference feature and estimated phase difference of the previous level, and then finally fused into the first-level left eye image and the first-level right eye image, and thus the estimated phase difference of the first-level left and right eye images can be accurately obtained, which further improves the corresponding image processing efficiency.
The present invention further provides a binocular image quick processing apparatus. Please refer to
The image acquiring module 61 is configured to acquire a first-level left eye image and a corresponding first-level right eye image; the folding dimensionality reduction module 62 is configured to perform folding dimensionality reduction operation on the first-level left eye image to acquire at least one next-level left eye image corresponding to the first-level left eye image, and perform the folding dimensionality reduction operation on the first-level right eye image to acquire at least one next-level right eye image corresponding to the first-level right eye image; the first feature extraction module 63 is configured to perform feature extraction on the next-level left eye image by using a first preset residual convolutional network to obtain a next-level left eye image feature, and perform feature extraction on the next-level right eye image by using the first preset residual convolutional network to obtain a next-level right eye image feature; the phase difference distribution estimation module 64 is configured to perform phase difference distribution estimation on the next-level left eye image feature and the next-level right eye image feature to obtain a corresponding next-level image phase difference distribution estimation feature; the fusing module 65 is configured to fuse the next-level image phase difference distribution estimation feature with the next-level left eye image feature to obtain a next-level fusion feature; the second feature extraction module 66 is configured to perform feature extraction on the next-level fusion feature by using a second preset residual convolutional network to obtain a difference feature of next-level left and right eye images; the next-level estimated phase difference acquiring module 67 is configured to obtain an estimated phase difference of the next-level left and right eye images based on the difference feature of the next-level left and right eye images; the tiling dimensionality raising module 68 is configured to perform tiling dimensionality raising operation on the difference feature to obtain a corrected difference feature of first-level left and right eye images, and perform the tiling dimensionality raising operation on the estimated phase difference to obtain a corrected phase difference of the first-level left and right eye images; the previous-level estimated phase difference acquiring module 69 is configured to obtain an estimated phase difference of the first-level left and right eye images according to first-level left and right eye feature data, the corrected difference feature of the first-level left and right eye images, and the corrected phase difference of the first-level left and right eye images; and the image processing module 6A is configured to perform image processing operation on the corresponding images by using the estimated phase difference of the first-level left and right eye images.
When the binocular image quick processing apparatus 60 of the present embodiment is used, first the image acquiring module 61 may acquire the first-level left eye image captured by a binocular camera and the corresponding first-level right eye image. The first-level left eye image and the corresponding first-level right eye image may be combined into a 3d scenario of the corresponding image.
Since sizes of scenario objects contained in the first-level left eye image and the first-level right eye image are different, in order to better perform feature recognition on the scenario objects with the different sizes, a left-eye image folding dimensionality reduction unit of the folding dimensionality reduction module 62 performs the folding dimensionality reduction operation on the first-level left eye image to acquire the plurality of next-level left eye images corresponding to the first-level left eye image, such as four second-level left eye images; and four third-level left eye images may be acquired if the folding dimensionality reduction operation is continued being performed on the second-level left eye image. An image resolution of the second-level left eye image is 1/4 of an image resolution of the first-level left eye image, and an image resolution of the third-level left eye image is 1/4 of an image resolution of the second-level left eye image.
In a similar way, a right-eye image folding dimensionality reduction unit of the folding dimensionality reduction module 62 performs the folding dimensionality reduction operation on the first-level right eye image to acquire the plurality of next-level right eye images corresponding to the first-level right eye image, such as four second-level right eye images; and four third-level right eye images may be acquired if the folding dimensionality reduction operation is continued being performed on the second-level right eye image. An image resolution of the second-level right eye image is 1/4 of an image resolution of the first-level right eye image, and an image resolution of the third-level right eye image is 1/4 of an image resolution of the second-level right eye image.
The setting of left and right eye images with different levels or resolutions may better meet needs of object receptive fields in different scenarios.
Then the first feature extraction module 63 performs feature extraction on the plurality of acquired next-level left eye images (such as the second-level left eye images and the third-level left eye images) by using the first preset residual convolutional network to obtain the plurality of next-level left eye image features at different levels.
Meanwhile, the first feature extraction module 63 performs feature extraction on the plurality of acquired next-level right eye images by using the first preset residual convolutional network to obtain the plurality of next-level right eye image features at different levels.
Then the phase difference distribution estimation module 64 performs phase difference analysis estimation on the next-level left eye image feature and the next-level right eye image feature of each level. That is, the possible phase difference on each point in the next-level left eye image feature and the next-level right eye image feature is evaluated to obtain the possibility of a certain phase difference value appearing at this point, namely, feasible distribution of an effective phase difference interval on a certain feature point. Later, the most possible phase difference value of this feature point is obtained through the analysis of this distribution.
When a probability corresponding to the most possible phase difference value of each point in the next-level left eye image feature and the next-level right eye image feature is maximum, the image phase difference distribution estimation feature under this level is obtained.
Subsequently the fusing module 65 fuses the acquired next-level image phase difference distribution estimation feature with the acquired next-level left eye image feature of the corresponding level to obtain a next-level fusion feature. The fusion here may be feature superposition of the next-level image phase difference distribution estimation feature and the next-level left eye image feature of the corresponding level. The fusion operation of the next-level left eye image feature may reduce an impact of an initial difference of the next-level left eye image, improve the accuracy of the subsequent feature extraction operation, and thus improve the accuracy of the subsequent difference feature.
Then, the second feature extraction module 66 performs the feature extraction on the acquired next-level fusion feature by using a second preset residual convolutional network to acquire the difference feature of next-level left and right eye images of the corresponding level.
Subsequently, the next-level estimated phase difference acquiring module 67 obtains the estimated phase difference of the next-level left and right eye images based on the acquired difference feature of the next-level left and right eye images.
That is, the estimated phase difference of the corresponding next-level left and right eye images is determined based on the preset estimated phase difference corresponding to the difference feature of the next-level left and right eye images. If the preset estimated phase difference corresponding to the difference feature of the next-level left and right eye images is larger, the estimated phase difference of the next-level left and right eye images obtained correspondingly is also larger. If the preset estimated phase difference corresponding to the difference feature of the next-level left and right eye images is smaller, the estimated phase difference of the next-level left and right eye images obtained correspondingly is also smaller. The preset estimated phase difference may be acquired through model training of positive and negative samples.
Then tiling dimensionality raising module 68 performs the tiling dimensionality raising operation on the acquired difference feature of the next-level left and right eye images to obtain the corrected difference feature of the first-level left and right eye images. The tiling dimensionality raising module 68 performs the tiling dimensionality raising operation on the acquired estimated phase difference of the next-level left and right eye images to obtain the corrected phase difference of the first-level left and right eye images.
For example, the tiling dimensionality raising module 68 may perform the tiling dimensionality raising operation on a difference feature of third-level left and right eye images to obtain a corrected difference feature of second-level left and right eye images, and the corrected difference feature of the second-level left and right eye images may be used to calculate a difference feature of the second-level left and right eye images; and then the tiling dimensionality raising module 68 performs the tiling dimensionality raising operation on the difference feature of the second-level left and right eye images to obtain the corrected difference feature of the first-level left and right eye images.
In a similar way, the tiling dimensionality raising module 68 may perform the tiling dimensionality raising operation on an estimated phase difference of the third-level left and right eye images to obtain a corrected phase difference of the second-level left and right eye images, and the corrected phase difference of the second-level left and right eye images may be used to calculate an estimated phase difference of the second-level left and right eye images; and then the tiling dimensionality raising module 68 performs the tiling dimensionality raising operation on the estimated phase difference of the second-level left and right eye images to obtain the corrected phase difference of the first-level left and right eye images.
Subsequently, the previous-level estimated phase difference acquiring module 69 performs feature fusion according to the acquired first-level left and right eye feature data such as the first-level left eye image and the corresponding first-level right eye image, the corrected difference feature of the first-level left and right eye images, and the corrected phase difference of the first-level left and right eye images, and obtains the estimated phase difference of the first-level left and right eye images based on the fusion feature. The corresponding relationship between the fusion feature and the estimated phase difference of the first-level left and right eye images may be acquired through model training of the positive and negative samples.
Finally, the image processing module 6A performs the image processing operation on the corresponding images by using the estimated phase difference of the first left and right eye images, such as synthesizing the binocular images into a corresponding three-dimensional scenario image, or performing three-dimensional image change operation on a monocular image.
In this way, a binocular image quick processing process of the binocular image quick processing apparatus 60 of the present embodiment is completed.
According to the binocular image quick processing apparatus of the present embodiment, the difference features of the first-level left eye image and the first-level right eye image in different dimensions and the corresponding estimated phase differences can be acquired through the next-level left eye images and next-level right eye images of the plurality of different dimensions, so that the estimated phase difference of the first-level left and right eye images can be quickly and accurately acquired, thereby improving a corresponding image processing efficiency.
Please refer to
When the binocular image quick processing apparatus of the present embodiment is used, first an image acquiring module 71 may acquire a first-level left eye image captured by a binocular camera and a corresponding first-level right eye image. The first-level left eye image and the corresponding first-level right eye image may be combined into a 3d scenario of the corresponding image.
Subsequently, a folding dimensionality reduction module 72 performs folding dimensionality reduction operation on the first-level left eye image to acquire a plurality of next-level left eye images corresponding to the first-level left eye image, such as four second-level left eye images; four third-level left eye images may be acquired if the folding dimensionality reduction operation is continued being performed on the second-level left eye image; and in turn, fourth-level left eye images and fifth-level left eye images are acquired.
In a similar way, the folding dimensionality reduction module 72 may also perform the folding dimensionality reduction operation on the first-level right eye image to acquire a plurality of next-level right eye images corresponding to the first-level right eye image, such as four second-level right eye images; four third-level right eye images may be acquired if the folding dimensionality reduction operation is continued being performed on the second-level right eye image; and in turn, fourth-level right eye images and fifth-level right eye images are acquired.
Then, the counting module 7B sets a count value m, and the current count value m is the level number of the next-level image with a lowest resolution, that is, m=5.
Subsequently, a first feature extraction module 73 performs feature extraction on the fifth-level left eye image by using a first preset residual convolutional network to obtain a fifth-level left eye image feature. Meanwhile, the first feature extraction module 73 performs the feature extraction on the fifth-level right eye image by using the first preset residual convolutional network to obtain a fifth-level right eye image feature.
Since there is no corrected phase difference of fifth-level left and right eye images, a phase difference distribution estimation module 74 performs phase difference distribution estimation on the fifth-level left eye image feature and the fifth-level right eye image feature respectively. That is, a possible phase difference on each point in the fifth-level left eye image feature and the fifth-level right eye image feature is evaluated to obtain a possibility of a certain phase difference value appearing at this point, namely, feasible distribution of an effective phase difference interval on a certain feature point. Later, a most possible phase difference value of this feature point is obtained through the analysis of this distribution.
When a probability corresponding to the most possible phase difference value of each point in the fifth-level left eye image feature and the fifth-level right eye image feature is maximum, a fifth-level image phase difference distribution estimation feature is obtained.
Since there is no corrected difference feature of the fifth-level left and right eye images, a fusing module 75 fuses the fifth-level image phase difference distribution estimation feature with the fifth-level left eye image feature to obtain a fifth-level fusion feature. The fusion here may be feature superposition of the fifth-level image phase difference distribution estimation feature and the fifth-level left eye image feature. The fusion operation of the fifth-level left eye image feature may reduce an impact of an initial difference of the fifth-level left eye image, improve the accuracy of the subsequent feature extraction operation, and thus improve the accuracy of the subsequent difference feature.
Then, a second feature extraction module 76 performs the feature extraction on the fifth-level fusion feature by using a second preset residual convolutional network to acquire a difference feature of the fifth-level left and right eye images.
Subsequently, a next-level estimated phase difference acquiring module 77 obtains an estimated phase difference of the fifth-level left and right eye images based on the difference feature of the fifth-level left and right eye images. That is, the estimated phase difference of the corresponding fifth-level left and right eye images is determined based on a preset estimated phase difference corresponding to the difference feature of the fifth-level left and right eye images. If the preset estimated phase difference corresponding to the difference feature of the fifth-level left and right eye images is larger, the estimated phase difference of the fifth-level left and right eye images obtained correspondingly is also larger. If the preset estimated phase difference corresponding to the difference feature of the fifth-level left and right eye images is small, the estimated phase difference of the fifth-level left and right eye images obtained correspondingly is also small. The preset estimated phase difference may be acquired through model training of positive and negative samples.
Since there is no corrected phase difference of the fifth-level left and right eye images, the next-level estimated phase difference acquiring module 77 directly takes the estimated phase difference of the fifth-level left and right eye images as a total estimated phase difference of the fifth-level left and right eye images.
Then, a tiling dimensionality raising module 78 performs tiling dimensionality raising operation on the difference feature of the fifth-level left and right eye images to obtain a corrected difference feature of fourth-level left and right eye images. The tiling dimensionality raising module 78 performs the tiling dimensionality raising operation on the total estimated phase difference of the fifth-level left and right eye images to obtain a corrected phase difference of the fourth-level left and right eye images.
Subsequently, the counting module 7B sets the count value m minus one, that is, executes m=m−1, and then returns to the first feature extraction module 73 to perform feature extraction on a fourth-level left eye image feature and a fourth-level right eye image feature.
Specifically, please refer to the relevant description in the second embodiment of the above binocular image quick processing method and the subsequent implementation flow in
Repetition is made until m=1. At this time, the binocular image quick processing apparatus 70 acquires a corrected difference feature of second-level left and right eye images and a corrected phase difference of the second-level left and right eye images.
A previous-level estimated phase difference acquiring module 79 fuses the first-level left eye image, the first-level right eye image, the corrected difference feature of the second-level left and right eye images and the corrected phase difference of the second-level left and right eye images to obtain a first-level fusion feature. The fusion operation here may be feature superposition of the above image features, corrected difference features and corrected difference features.
Subsequently, the previous-level estimated phase difference acquiring module 79 performs the phase difference distribution estimation on the first-level fusion feature to obtain an estimated phase difference of the first-level left and right eye images. The corresponding relationship between the first-level fusion feature and the estimated phase difference of the first-level left and right eye images may be acquired through model training of the positive and negative samples.
Finally, an image processing module 7A performs the image processing operation on the corresponding images by using the estimated phase difference of the first left and right eye images, such as synthesizing binocular images into a corresponding three-dimensional scenario image, or performing three-dimensional image change operation on a monocular image.
In this way, a binocular image quick processing process of the binocular image quick processing apparatus 70 of the present embodiment is completed.
On the basis of the first embodiment, the binocular image quick processing apparatus of the present embodiment uses the next-level left and right eye images to correct the estimated phase difference of the previous-level left and right eye images of adjacent levels, so that the difference feature of each level and the corresponding estimated phase difference may be accurately fused into the difference feature and estimated phase difference of the previous level, and then finally fused into the first-level left eye image and the first-level right eye image, and thus the estimated phase difference of the first-level left and right eye images may be accurately acquired, which further improves the corresponding image processing efficiency.
Please refer to
The binocular image quick processing method and apparatus of the present invention may acquire the difference features of the first-level left eye image and the first-level right eye image in different dimensions and the corresponding estimated phase differences through the next-level left eye images and next-level right eye images of the plurality of different dimensions, so that the estimated phase difference of the first-level left and right eye images may be quickly and accurately acquired, thereby improving a corresponding image processing efficiency; and the technical problems of a larger difference and lower accuracy of a binocular visual image phase difference acquired by an existing binocular image quick processing method and apparatus are effectively solved.
To sum up, although the present invention has been disclosed as above based on the embodiments, sequence numbers before the embodiments are used only for the convenience of description, and do not limit the order of the embodiments of the present invention. Moreover, the above embodiments are not intended to limit the present invention. Those ordinarily skilled in the art can make various changes and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention is subject to the scope defined in the claims.
Number | Date | Country | Kind |
---|---|---|---|
202010471421.8 | May 2020 | CN | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/CN2021/088003 | 4/19/2021 | WO |