This application claims priority to and the benefit of Korean Patent Application No. 10-2020-0016764 filed in the Korean Intellectual Property Office on Feb. 12, 2020, the entire content of which is incorporated herein by reference.
The present invention relates to a method and apparatus for enhancing a disparity map, and more particularly, to a method and apparatus for enhancing a disparity map for improving disparity errors of a disparity map according to matching of stereo images.
The stereo matching result is output as a disparity map. The disparity map is an image in which a position difference value between each pixel of a reference image and a corresponding pixel of a reference image is displayed in coordinates of the reference image.
There are various stereo matching methods, but the matching result is not perfect, so the disparity map correction is required as a post-processing operation. The disparity errors occur due to the limitation of the stereo matching algorithm, or also occur due to the image acquired from the camera itself. For example, the disparity error due to the former case may occur when the object is visible only in one image due to the difference between the baseline and the camera rotation in the left and right images, and the disparity error due to the latter case may occur when the object is in both the left and right images, but the pixel contrast or color value between the left and right images are different. Due to these disparity errors, an invalid disparity value is generated, so the reliability of the stereo matching result is decreased.
A method of enhancing the errors by using the peripheral disparity values of the disparity map is used as a conventional method to solve this problem.
The present invention has been made in an effort to provide a disparity map capable of improving the reliability of a disparity map stereo matching result.
According to an embodiment of the present invention, a method for enhancing a disparity map generated by performing stereo matching on a stereo image in an apparatus for enhancing the disparity map is provided. The method for enhancing a disparity map includes: generating a disparity map of a reference image by performing stereo matching for each stereo image; dividing the reference image into at least one homogeneous area based on a homogeneity criterion; extracting valid disparity values from the disparity map of the reference image; calculating a linear rate of change for the valid disparity values in each homogeneous area divided based on the extracted valid disparity values; and enhancing the disparity map by updating invalid disparity values in a corresponding homogeneous area using the linear rate of change calculated for each homogeneous area.
The calculating includes: a first step of setting a linear rate of change in a first homogeneous area by using valid disparity values in the first homogeneous area, when the number of valid disparity values in the first homogeneous area is greater than a minimum value; and a second step of setting a linear rate of change in the first homogeneous area by using the valid disparity values of at least one second homogeneous area adjacent to the first homogeneous area, when the number of valid disparity values in the first homogeneous area is less than or equal to the minimum value.
The first step includes, when there are two or more pixels having a valid disparity value on the same y-axis in the first homogeneous area, calculating the linear rate of change using the two valid disparity values, and the enhancing includes updating an invalid disparity value existing on a line connecting the coordinates of the two pixels having the valid disparity values using the linear rate of change and the valid disparity values.
The first step includes, when there is one pixel having a valid disparity value on the same y-axis in the first homogeneous area, calculating the linear rate of change using the valid disparity value of the one pixel and a valid disparity value of a pixel closest to the one pixel, and the enhancing includes updating an invalid disparity value existing on a line connecting coordinates of two pixels having the valid disparity values using the linear rate of change and the valid disparity value of the one pixel.
The second step includes, when two pixels having the same y-axis coordinate value in two second homogeneous areas have a valid pixel value, respectively, and there is one pixel having a valid disparity value in the first homogeneous area: estimating disparity values of two pixels in the first homogeneous area having the same y-axis coordinate value as the two pixels of the two second homogeneous areas using a linear rate of change with respect to the valid pixel values of the two pixels of the two second homogeneous areas; and calculating a linear rate of change in the first homogeneous area by using the distance between each pixel with the estimated disparity value in the first homogeneous area and the pixel having the valid disparity value in the first homogeneous area.
The second step includes, when one pixel has a valid disparity value on the y-axis in one second homogeneous area and one pixel has a valid disparity value in the first homogeneous area: estimating valid disparity values of two pixels of the first homogeneous area having the same y-axis coordinate value as the one pixel in the one second homogeneous area using by using a linear rate of change of the valid disparity value in the one second homogeneous area; and calculating a linear rate of change in the first homogeneous area by using the distance between each pixel with the estimated disparity value in the first homogeneous area and the pixel having the valid disparity value in the first homogeneous area.
The enhancing includes updating the invalid disparity values in the first homogeneous area by using a linear rate of change in the first homogeneous area and a valid disparity value in the first homogeneous area.
The second step includes, when two pixels having the same y-axis coordinate value in two second homogeneous areas have a valid pixel value, respectively, and there is no pixel having a valid disparity value in the first homogeneous area: estimating disparity values of two pixels in the first homogeneous area having the same y-axis coordinate value with the two pixels of the two second homogeneous areas using a linear rate of change with respect to the valid pixel values of the two pixels of the two second homogeneous areas; and calculating a linear rate of change in the first homogeneous area by using the estimated disparity values of two pixels in the first homogeneous area.
The second step includes, when one pixel has a valid disparity value on the y-axis in one second homogeneous area, and there is no pixel having a valid disparity value in the first homogeneous area: estimating valid disparity values of two pixels of the first homogeneous area having the same y-axis coordinate value as the one pixel in the one second homogeneous area using by using a linear rate of change of the valid disparity value in the one second homogeneous area; and calculating a linear rate of change in the first homogeneous area by using the estimated disparity values of two pixels in the first homogeneous area.
The enhancing includes updating an invalid disparity value in the first homogeneous region by using a linear rate of change in the first homogeneous region and a disparity value of two pixels estimated in the first homogeneous region.
The dividing includes grouping pixels having homogeneity in at least one of brightness, texture, and color of the pixel into the same homogeneous area.
The enhancing includes, after updating the invalid disparity values for all homogeneous areas in the reference image, smoothing an updated disparity map.
According to another embodiment of the present invention, an apparatus for enhancing a disparity map generated from a stereo image is provided. The apparatus for enhancing a disparity map includes: a stereo matching unit that performs stereo matching for each input stereo image to generate a disparity map of a reference image; and a disparity enhancing unit that divides the reference image into at least one homogeneous area based on a homogeneity criterion, extracts valid disparity values from the disparity map of the reference image, and updates invalid disparity values in a corresponding homogeneous area by calculating a linear rate of change for the valid disparity values in each divided homogeneous area.
The disparity enhancing unit includes a disparity allocator for setting a linear rate of change for valid disparity values of the first homogeneous area by using at least one valid disparity value in the first homogeneous area, or by using the valid disparity values of at least one second homogeneous area adjacent to the first homogeneous area, based on the number of valid disparity values in the first homogeneous area.
The disparity allocator updates invalid disparity values existing in the first homogeneous area by using the linear rate of change for valid disparity values of the first homogeneous area and the at least one valid disparity value in the first homogeneous area.
The disparity allocator updates estimating disparity values of two pixels in the first homogeneous area having the same y-axis coordinate value as at least one pixel having a valid disparity value of the second homogeneous areas using a linear rate of change with respect to the at least one valid pixel value of the second homogeneous areas, and calculates a linear rate of change in the first homogeneous area by using the estimated disparity values of two pixels in the first homogeneous area.
When there is one pixel having a valid disparity value in the first homogeneous area, the disparity allocator calculates the linear rate of change in the first homogeneous area by using the estimated disparity values of two pixels in the first homogeneous area and the valid disparity value of the one pixel of the first homogeneous area.
The disparity allocator updates the invalid disparity values in the first homogeneous area by using the linear rate of change in the first homogeneous area and the estimated disparity values of two pixels in the first homogeneous area.
The disparity enhancing unit smooths an updated disparity map after updating the invalid disparity values for all homogeneous areas in the reference image.
The disparity enhancing unit includes an image divider for grouping pixels having homogeneity in at least one of brightness, texture, and color of the pixel into the same homogeneous area.
Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that a person of ordinary skill in the art may easily implement the present invention. The present invention may be modified in various ways, and is not limited thereto. In the drawings, elements that are irrelevant to the description of the present invention are omitted for clarity of explanation, and like reference numerals designate like elements throughout the specification.
Throughout the specification and claims, when a part is referred to “include” a certain element, it means that it may further include other elements rather than exclude other elements, unless specifically indicated otherwise.
Hereinafter, a method and apparatus for enhancing a disparity map according to embodiments of the present invention will be described in detail with reference to the accompanying drawings.
Referring to
The stereo matching unit 110 performs stereo matching on the stereo image to obtain the disparity of the reference image and the comparative image, and displays the obtained disparity on the coordinates of the corresponding image, respectively, to generate a disparity map of the reference image and a disparity map of the comparative image. Stereo matching is the process of finding corresponding points in the comparative image (e.g., the right image) for each pixel in the reference image (e.g., the left image) in the stereo image. The stereo matching disparity process is to calculate disparity values indicating how far the corresponding pixels are from each other in the reference image and the comparative image. For example, if the disparity value of the pixel at the (x, y) position calculated through stereo matching on the left image, which is the reference image, is DL(x, y), a pixel at a position (x-(DL(x, y),y) separated by a pixel distance of DL(x, y) based on the (x, y) position in the right image, which is the comparative image, becomes a corresponding point with a pixel at the (x, y) position of the left image.
Meanwhile, image rectification may be performed through image pre-processing before stereo matching. The image rectification is to align the y-axis of the image using external and internal parameters of the camera. The stereo matching is to find a point corresponding to a point of reference image in the comparative image. If the image rectification is performed, the entire image is not searched for finding the point corresponding to a point of reference image, but only on the same y-axis, is searched.
The disparity enhancing unit 120 enhances disparity errors of the disparity map generated by the stereo matching unit 110.
The disparity enhancing unit 120 includes an image divider 122, a valid disparity extractor 124, and a disparity allocator 126. Hereinafter, for convenience of explanation, an embodiment of enhancing the disparity map of the left image will be described, and the method for enhancing the disparity map of the left image described below may be applied for enhancing the disparity map of the right image.
The image divider 122 divides the reference image, which is an input image of the disparity map, into at least one homogeneous area based on homogeneity. The homogeneous area refers to a set of pixels having homogeneity such as brightness, texture, and color of a pixel. The image divider 122 may divide the image into at least one homogeneous region using, for example, a superpixel technique. The superpixel technique is a method of collecting pixels with similar characteristics and dividing the image into small areas. At this time, the number of divisions may vary depending on the size of the image and the complexity of the captured scene, and one homogeneous region is represented as one divided area.
The valid disparity extractor 124 extracts valid disparity values from the disparity map and displays them on the disparity map. As a method of extracting valid disparity values, a method of comparing disparity value differences between corresponding points between the left image and the right image and a set threshold value may be used.
As shown in
As shown in Equation 1, when the disparity difference value of the corresponding points between the left image and the right image with respect to the disparity value of one pixel at the (x, y) position in the left image is less than the threshold value, the valid disparity extractor 124 determines the disparity value of one pixel at the (x, y) position in the left image as a valid disparity, otherwise determines the disparity value of one pixel at the (x, y) position in the left image as an invalid disparity.
|dST−dRF|<thd (Equation 1)
In Equation 1, the DST represents the disparity value for the pixel at the (x, y) position in the left image, and the DRF represents the disparity value for the pixel at the (x−DST, y) position in the right image. The thd represents the threshold value.
Again referring to
Referring to
If the number of valid disparity values in the ROI is greater than a predetermined minimum value (S320), the disparity allocator 126 sets an internal linear relationship for the valid disparity value by using the valid disparity values inside the ROI (S330), and updates invalid disparity values in the ROI using the internal linear relationship (S340).
If the number of valid disparity values in the ROI is less than or equal to the predetermined minimum value (S320), the disparity allocator 126 sets an external linear relationship for the valid disparity value in the ROI by using valid disparity values of the area adjacent to the ROI (S350), and updates invalid disparity values in the ROI using the external linear relationship (S360).
The disparity allocator 126 checks whether there is an invalid disparity in the ROI (S370), and repeats steps S310 to S360 until there is no invalid disparity in the ROI.
If there is no invalid disparity in the ROI, the disparity allocator 126 changes the ROI to another divided area (380), and repeats steps S310 to S370.
In this way, invalid disparity values are updated for all divided areas in the left image.
After the disparity allocator 126 updates invalid disparity values in all divided areas in the left image, the disparity map is smoothed by applying a smoothing filter to the updated disparity map. In this case, the size of the filter and the number of times the filter is applied may be determined according to the utilization direction of the disparity map.
Next, a method of setting an internal linear relationship and updating invalid disparity values in the ROI using the internal linear relationship will be described in detail with reference to
The disparity allocator 126 calculates a linear rate of change (in the x-axis direction and the y-axis direction) of the valid disparity value by using two valid disparity values within the ROI of the left image, and updates invalid disparity values by applying the linear rate of change to the invalid disparity values. To explain more specifically, when the number of valid disparity values in the t-th ROI (Rt) is nt, it is expressed as (Rt:nt), and the coordinate set of the valid disparity values in the t-th ROI (Rt) is expressed as (Rt: nt)=((x1t, y1t), (x2t, y2t), . . . , (xnt, ynt)}. “pixel (xnt, ynt)” refers to a pixel at a position of the (xnt, ynt) coordinates.
As shown in
When a pixel (xot, yot) having an invalid disparity value exists on this line, the disparity allocator 126 updates the disparity value of the pixel (xot, yot) by using the linear rate of change (Δdt) as in Equation 3.
d
x
t=(xmt−xot)*Δdt+dx
On the other hand, as shown in
Here, the dX
If there is a pixel (xot, yot) having an invalid disparity on the line to which the two coordinates [(xmt, ymt), (xnt, ynt)] are connected, the disparity allocator 126 updates the disparity value of the pixel (xot, yot) using the linear rate of change (Δdt) of Equation 4, as shown in Equation 5.
d
X
t=(√{square root over ((xmt−xot)2+(ymt−yot)2)})*Δdt+dX
Next, a method of setting an external linear relationship and updating an invalid disparity in the ROI by using the external linear relationship will be described in detail with reference to
When the number of valid disparity values in the ROI is less than the predetermined minimum value, the disparity allocator 126 updates the invalid disparity value of the ROI by using a linear rate of change of the valid disparity values of the area adjacent to the ROI. Here, the linear rate of change of the adjacent area of the ROI is referred to as the external linear rate of change of the ROI.
As shown in
The disparity allocator 126 estimates disparity values (dX
d
X
t=(xnt−xns)*ΔdS+dX
Here, Δds represents the linear rate of change of the valid disparity values of the adjacent area (Rs), that is, the external linear rate of change.
d
X
t=(xmu−xmt)*Δdu+dX
Here, Δdu represents the linear rate of change of the valid disparity values of the adjacent area (Ru).
The internal linear rate of change (Δdt) of the ROI (Rt) can be calculated as in Equation 8.
Here, Δdlnt represents the rate of change of the disparity values between xn and xl in the ROI (Rt), and may be calculated as in Equation 9. Δdlmt represents the rate of change of the disparity values between xm and xl in the ROI (Rt), and can be calculated as in Equation 10.
The disparity allocator 126 updates a disparity value of the pixel (xot, yot) having an invalid disparity value in the ROI (Rt) using the calculated internal linear rate of change (Δdt) of the ROI (Rt) and the valid disparity value (dx
d
X
t=(√{square root over ((xlt−xot)2+(ylt−yot)2)})*Δdt+dX
Next, as shown in
The disparity allocator 126 estimates disparity values (dX
In this case, the disparity allocator 126 updates a disparity value of the pixel (xot, yot) having an invalid disparity value in the ROI (Rt) using the calculated internal linear rate of change (Δdt) of the ROI (Rt) and the estimated disparity values (dX
d
X
t=(xnt−xot)*Δdt+dX
On the other hand, as shown in
d
X
t=(xnt−xns)*ΔdS+dX
d
X
t=(xmt−xns)*Δds+dX
The disparity allocator 126 updates a disparity value of the pixel (xot, yot) having an invalid disparity value in the ROI (Rt) using the calculated internal linear rate of change (Δdt) of the ROI (Rt) and the valid disparity value (dX
In addition, as shown in
In this case, the disparity allocator 126 updates a disparity value of the pixel (xot, yot) having an invalid disparity value in the ROI (Rt) using the calculated internal linear rate of change (Δdt) of the ROI (Rt) and the estimated disparity values (dX
After the disparity allocator 126 updates invalid disparity values in all divided areas in the left image, the disparity map is smoothed by applying a smoothing filter to the updated disparity map.
The method for enhancing the disparity map according to an embodiment of the present invention described above may correct the disparity map generated from the first input image by using the peripheral valid disparity values.
At least some functions of the method and apparatus for enhancing the disparity map may be implemented by hardware or by software coupled to hardware. Hereinafter, an embodiment in which a method and apparatus for enhancing a disparity map are combined with a computer system will be described in detail with reference to
Referring to
The processor 1100 may be implemented as a central processing unit (CPU), another chipset, or a microprocessor.
The memory 1200 may be implemented as a medium such as random access memory (RAM), a dynamic random access memory (DRAM), a rambus DRAM (RDRAM), a synchronous DRAM (SDRAM), a static RAM (SRAM), etc.
The storage device 1300 may be implemented as a hard disk, an optical disk such as a compact disk read only memory (CD-ROM), a CD rewritable (CD-RW), a digital video disk ROM (DVD-ROM), a DVD-RAM, a DVD-RW disk, Blu-ray disks, etc., a flash memory, or permanent or volatile storage devices such as various types of RAM.
The memory 1200 or the storage device 1300 may store the above-described left image and right image, disparity values of the left image, and disparity values of the right image.
The I/O interface 1400 allows the processor 1100 and/or the memory 1200 to access the storage device 1300. In addition, the I/O interface 1400 may provide an interface with a user. For example, a stereo image may be input through the I/O interface 1400, and an enhanced disparity map may be output through the I/O interface 1400.
The processor 1100 may perform a function of enhancing the disparity map described in
According to an embodiment of the present invention, a disparity map may be corrected from the first input image by using a valid disparity value of the near periphery.
In addition, it is possible to enhance the disparity value even when an occluded area occurs due to a limitation of stereo matching and when a corresponding point cannot be found due to a problem in an acquired image.
The components described in the example embodiments may be implemented by hardware components including, for example, at least one digital signal processor (DSP), a processor, a controller, an application-specific integrated circuit (ASIC), a programmable logic element, such as an FPGA, other electronic devices, or combinations thereof. At least some of the functions or the processes described in the example embodiments may be implemented by software, and the software may be recorded on a recording medium. The components, the functions, and the processes described in the example embodiments may be implemented by a combination of hardware and software.
The method according to example embodiments may be embodied as a program that is executable by a computer, and may be implemented as various recording media such as a magnetic storage medium, an optical reading medium, and a digital storage medium.
Various techniques described herein may be implemented as digital electronic circuitry, or as computer hardware, firmware, software, or combinations thereof. The techniques may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device (for example, a computer-readable medium) or in a propagated signal for processing by, or to control an operation of a data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program(s) may be written in any form of a programming language, including compiled or interpreted languages and may be deployed in any form including a stand-alone program or a module, a component, a subroutine, or other units suitable for use in a computing environment. A computer program may be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
Processors suitable for execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer may include at least one processor to execute instructions and one or more memory devices to store instructions and data. Generally, a computer will also include or be coupled to receive data from, transfer data to, or perform both on one or more mass storage devices to store data, e.g., magnetic, magneto-optical disks, or optical disks. Examples of information carriers suitable for embodying computer program instructions and data include semiconductor memory devices, for example, magnetic media such as a hard disk, a floppy disk, and a magnetic tape, optical media such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), etc. and magneto-optical media such as a floptical disk, and a read only memory (ROM), a random access memory (RAM), a flash memory, an erasable programmable ROM (EPROM), and an electrically erasable programmable ROM (EEPROM) and any other known computer readable medium. A processor and a memory may be supplemented by, or integrated into, a special purpose logic circuit.
The processor may run an operating system (08) and one or more software applications that run on the OS. The processor device also may access, store, manipulate, process, and create data in response to execution of the software. For purpose of simplicity, the description of a processor device is used as singular; however, one skilled in the art will be appreciated that a processor device may include multiple processing elements and/or multiple types of processing elements. For example, a processor device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such as parallel processors.
Also, non-transitory computer-readable media may be any available media that may be accessed by a computer, and may include both computer storage media and transmission media.
The present specification includes details of a number of specific implements, but it should be understood that the details do not limit any invention or what is claimable in the specification but rather describe features of the specific example embodiment. Features described in the specification in the context of individual example embodiments may be implemented as a combination in a single example embodiment. In contrast, various features described in the specification in the context of a single example embodiment may be implemented in multiple example embodiments individually or in an appropriate sub-combination. Furthermore, the features may operate in a specific combination and may be initially described as claimed in the combination, but one or more features may be excluded from the claimed combination in some cases, and the claimed combination may be changed into a sub-combination or a modification of a sub-combination.
Similarly, even though operations are described in a specific order on the drawings, it should not be understood as the operations needing to be performed in the specific order or in sequence to obtain desired results or as all the operations needing to be performed. In a specific case, multitasking and parallel processing may be advantageous. In addition, it should not be understood as requiring a separation of various apparatus components in the above described example embodiments in all example embodiments, and it should be understood that the above described program components and apparatuses may be incorporated into a single software product or may be packaged in multiple software products.
It should be understood that the example embodiments disclosed herein are merely illustrative and are not intended to limit the scope of the invention. It will be apparent to one of ordinary skill in the art that various modifications of the example embodiments may be made without departing from the spirit and scope of the claims and their equivalents.
Number | Date | Country | Kind |
---|---|---|---|
10-2020-0016764 | Feb 2020 | KR | national |