Patent Grant
12131493
This application claims priority to Korean Patent Application No. 10-2021-0054824 filed on Apr. 28, 2021, which is hereby incorporated by reference in its entirety.
This research was supported by the Ministry of Science and ICT (MSIT), Republic of Korea, under the Grand Information Technology Research Center support program (IITP-2021-2016-0-00318) supervised by an Institute for Information & communications Technology Planning & Evaluation (IITP).
The present invention relates to an apparatus and method for generating a depth map using a monocular image, and more specifically, to an apparatus and method for generating a depth map using a monocular image, which can quickly estimate a depth from the monocular image by reducing the amount of computation while having an accuracy higher than that of the prior art by using a deep convolution neural network (DCNN) optimized based on an encoder-decoder architecture.
With the advancement in the field of small robots and drones, various techniques for unmanned autonomy are developed. The techniques may be categorized into a recognition area for recognizing surrounding environments, a determination area for planning a driving route based on the recognized environments, and a control area for driving along the planned route. It is very important in this unmanned autonomy technique to quickly and accurately visualize two-dimensionally captured images in three dimensions of real world using a depth map, and this is since that it is helpful for the small robots and drones to grasp a location or avoid obstacles.
A three-dimensional (3D) depth estimation technique using a depth map may be largely divided into an active type and a passive type. Although a laser scanner, which is a representative example of the active type, provides high accuracy and 3D depth information, there is a disadvantage in that it is difficult to apply to a real environment until now due to the low resolution and high price. In addition, an active lighting method based on structured light is difficult to use in an outdoor environment where strong lighting exists due to the limitation of the structured light, and the range of depth that can be estimated to the maximum is limited to around 10 meters.
As the passive method, there is stereo matching that estimates 3D information from multi-view images acquired using two or more cameras, in which two cameras are required, and since binocular images captured by the two cameras should be analyzed, a large amount of computation and storage space are required.
That is, although the conventional 3D depth estimation technique using a depth map may recognize three dimensions using a depth sensor such as a LiDAR, a structured light sensor or the like, since the sensors are voluminous and heavy and consume much power, there is a limit in applying the technique to small robots or drones. Accordingly, a technique capable of three-dimensionally recognizing a photographed object with a compact size, low cost, and low power is required in the corresponding field.
Therefore, the present invention has been made in view of the above problems, and it is an object of the present invention to provide an apparatus and method for generating a depth map using a monocular image, the apparatus comprising: an encoder for extracting one or more features from the monocular image according to the number of feature layers, and a decoder for calculating displacements of mismatched pixels from the features extracted from different feature layers and generating a depth map for the monocular image to apply the apparatus to a drone or a small robot that should mount lightweight software and hardware by achieving a low delay time together with high accuracy and high resolution compared to the prior art, although the depth map is generated with only a small number of parameters.
To accomplish the above object, according to one aspect of the present invention, there is provided an apparatus for generating a depth map using a monocular image, the apparatus comprising: a DCNN optimized based on an encoder and decoder architecture, wherein the encoder extracts one or more features from the monocular image according to the number of provided feature layers, and the decoder calculates displacements of mismatched pixels from the features extracted from different feature layers, and generates the depth map for the monocular image.
To accomplish the above object, there is provided a method of generating a depth map using a monocular image, the method comprising the steps of: extracting one or more features from the monocular image according to the number of provided feature layers, by an encoder; and calculating displacements of mismatched pixels from the features extracted from different feature layers and generating the depth map for the monocular image, by a decoder.
Although general terms widely used at present are selected as terms used in this specification as much as possible considering the functions of the present invention, this may vary according to the intention of those skilled in the art, precedents, or emergence of new techniques. In addition, in specific cases, there are terms arbitrarily selected by an applicant, and in this case, the meaning of the terms will be described in detail in the corresponding description of the present invention. Therefore, the terms used in the present invention should be defined based on the meaning of the terms and the overall contents of the present invention, not by the simple names of the terms.
Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those skilled in the art. The terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of related techniques, and should not be interpreted as an ideal or excessively formal meaning unless clearly defined in this application.
Apparatus for Generating Depth Map Using Monocular Image
Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.
First, referring to
In addition, since the DCNN 100 is based on pixels when it estimates a depth from a monocular image, it should acquire semantic features and spatial information of an object to estimate the boundary of the object. Therefore, most preferably, the DCNN 100 may further include a pose estimation network (PoseNet) and a depth estimation network (DepthNet) to learn data sets on the basis of unsupervised learning.
First, since the DCNN 100 learns data sets on the basis of unsupervised learning, a data set including a Ground Truth Depth, i.e., a separate correct answer value, is not required, and accordingly, there is an effect of reducing the cost for providing a data set having a correct answer value.
In addition, the pose estimation network (PoseNet) may regress conversion between adjacent frames used for reconstructing a monocular image in the data set. For example, 6 Degrees of Freedom (DoF) may be predicted based on the monocular image, and the first three dimension represents a translation vector, and next three dimension may represent the Euler angle. The depth estimation network (DepthNet) may calculate a loss that occurs during the unsupervised learning based on the output of the pose estimation network (PoseNet), and then estimate a depth map for each monocular image in the data set.
Next, in the architecture of the optimized DCNN 100, the encoder 110 extracts one or more features Xi (i=1, 2, 3, 4 . . . n) from the monocular image according to the number of provided feature layers. The number of features Xi may be the same as the number of provided feature layers.
Most preferably, the encoder 110 may be based on MobileNetV2 to be mounted on a drone or a small robot for fast computation.
Conventionally, there are various CNN architectures such as SqueezeNet, MobileNet, MobiletNetV2, and MobileNetV3 suitable for the encoder 110. All of these neural networks may classify objects from an image without need of complex calculations and may be easily distributed to embedded systems in real-time. However, SqueezeNet and MobileNet classify images with a small piece information of input images and thus have a disadvantage of low accuracy. In addition, although MobileNetV3 is faster than MobileNetV2 in classifying images, it has a disadvantage of low accuracy in a work of image segmentation or object detection that requires more pixel-based information. Therefore, most preferably, the encoder 110 of the present invention may be based on MobileNetV2 trained in advance using data set ImageNet.
In addition, most preferably, the encoder 110 may have a first feature layer FL1 to a fifth feature layer FL5 provided with 16, 24, 32, 96, and 160 channels at the scales of ½, ¼, ⅛, 1/16, and 1/32, respectively. In addition, a first feature Xi to a fifth feature X5 may be extracted from the feature layers, respectively.
Next, the decoder 120 calculates displacements of mismatched pixels from the features Xi and Xi+1 extracted from different feature layers, and generates a depth map for the monocular image.
The decoder 120 may include an encoder SE block 121, a high-density block 122, an up-sampling block 123, a decoder SE block 124, and a disparity convolution block 125 to generate a depth map for the monocular image.
More specifically, the encoder SE block 121 may generate first channel information C1 using the features Xi and Xi+1 extracted from different feature layers to enable channel attention, and outputs first major channel information CA1 from the first channel information C1.
Referring to
In addition, the encoder SE block 121 may determine the first major channel information CA1 from the first channel information C1 using a fully-connected (FC) function, and activate the first major channel information CA1 with a higher weighting value using a ReLu function. This series of processes may be referred to as a Squeeze process.
In addition, the encoder SE block 121 may perform 1×1 convolution after expanding the compressed first major channel information CA1 using a fully-connected (FC) function and a Sigmoid function and then scaling the size. This series of processes may be referred to as an excitation process. Here, the 1×1 convolution may reduce the number of parameters for the entire operation by reducing the channels using a filter having a size of 1×1.
Accordingly, since the encoder SE block 121 may extract one or more features Xi only from a monocular image captured by one camera, not a stereo image captured by two or more cameras, there is an effect of remarkably reducing the number of stored or processed images. In addition, since the first major channel information CA1 may be output using the features Xi and Xi+1 extracted from two different feature layers FLi and FLi+1 in the encoder 110, the operation parameters of the decoder 120 may be reduced remarkably, and therefore, there is a remarkable effect of reducing the operation delay time.
Next, referring to
Most preferably, the high-density block 122 may include a first density layer DL1 to a fourth density layer DL4 and may include a plurality of channels between the density layers, and the growth rate may be 32. Here, the channels may be classified into one input channel and a plurality of output channels. That is, for each of the density layers DLi (i=1, 2, 3, 4), the dimension of the output channel of the high-density block 122 may increase by 32 times according to the growth rate. Accordingly, the high-density block 122 may finally output a feature set XCi in the form of high-density collective knowledge.
Meanwhile, the high-density block 122 may further include 1×1 convolution to fuse the input channel and reduce the number of parameters for calculation. Accordingly, the high-density block 122 has an effect of alleviating a gradient loss problem, enhancing feature propagation, and enabling feature reuse.
Next, the up-sampling block 123 may perform a Nearest Neighbor Interpolation operation using double scaling for the feature set XCi to improve the resolution of the depth map.
Meanwhile, the up-sampling block 123 may perform up-sampling on the first major channel information CA1 output from the encoder SE block 121, as well as on the feature set XCi. In addition, the up-sampling block 123 may include 3×3 convolution and perform operation by expanding the feature set XCi on which the 1×1 convolution is performed by the high-density block 122.
Next, the decoder SE block 124 may generate second channel information C2 from the feature set XCi up-sampled by the up-sampling block 123 to enable channel attention, and output second major channel information CA2 from the second channel information C2.
For example, first, the decoder SE block 124 may receive the up-sampled feature set XCi from the up-sampling block 123. Then, the decoder SE block 124 may perform a global pooling process of generating the second channel information C2 by averaging and compressing the features aggregated in the feature set XCi at a high density.
In addition, the decoder SE block 124 may determine the second major channel information CA2 from the second channel information C2 using a fully-connected (FC) function, and activate the second major channel information CA2 with a higher weighting value using a ReLu function. This series of processes may be referred to as a Squeeze process.
In addition, the decoder SE block 124 may perform 1×1 convolution after expanding the compressed second major channel information CA2 using a fully-connected (FC) function and a Sigmoid function and then scaling the size. This series of processes may be referred to as an excitation process. Here, the 1×1 convolution may reduce the number of parameters for the entire operation by reducing the channels using a filter having a size of 1×1.
Therefore, since the decoder SE block 124 may output the second major channel information CA2 using the features aggregated in the feature set XCi at a high density, the operation parameters of the decoder 120 may be reduced remarkably, and therefore, there is a remarkable effect of reducing the operation delay time.
Next, referring to
Therefore, most preferably, the decoder 120 may perform decoding using all of the feature Xi extracted from an arbitrary feature layer FL1 provided in the encoder 110, the first major channel information CA1, and the second major channel information CA2 to generate the depth map.
Next, the decoder SE block 124 may be skip-connected to the encoder SE block 121. Here, the meaning of being skip-connected is to obtain more semantic information from the monocular image. That is, the present invention may finally generate a depth map having a more improved resolution by combining strong features of low resolution and weak features of high resolution using skip-connection between corresponding objects.
Therefore, as the present invention has a DCNN 100 optimized based on an encoder and decoder architecture, although the depth map is generated using as few parameters as only 4.1 million or so, there is a remarkable effect of applying the apparatus to a drone or a small robot by achieving a low delay time together with high accuracy and high resolution compared to the prior art.
Method of generating depth map using monocular image Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Referring to
First, at the encoding step (S100), one or more features are extracted from a monocular image according to the number of provided feature layers, by the encoder 110.
According to the embodiment of [Table 1] and
The first feature layer FL1 may include 16 channels, and scale an input dimension of H×W by ½ to output an output dimension of (H/2)×(W/2). That is, like the first feature layer FL1, the second feature layer FL2 to the fifth feature layer FL 5 may include 24, 32, 96, and 160 channels, respectively, and scale input dimensions of H×W by ¼, ⅛, 1/16, and 1/32 to finally output an output dimension of (H/32)×(W/32).
Next, at the decoding step (S200), displacements of mismatched pixels are calculated from the features extracted from different feature layers, and a depth map is generated for the monocular image, by the decoder 120.
The decoding step (S200) may include an encoder SE step (S210), a high-density step (S220), an up-sampling step (S230), a decoder SE step (S240), and a disparity convolution step (S250) to generate a depth map for the monocular image.
First, at the encoder SE step (S210), first channel information C1 may be generated using the features Xi and Xi+1 extracted from different feature layers to enable channel attention, and first major channel information CA1 may be output from the first channel information C1, by the encoder SE block 121 in the decoder 120.
In other words, the encoder SE step (S210) may receive a feature Xi extracted from the i-th feature layer FL1 of the encoder 110 and a feature Xi+1 extracted from the i+1-th feature layer FLi+1. Then, the encoder SE step (S210) may perform a global pooling process of generating the first channel information C1 by averaging and compressing the two features Xi and Xi+1.
Most preferably, the encoder SE step (S210) may be performed starting from the last feature layer. Referring to [Table 1], for example, at the encoder SE step (S210), the fifth feature X5 extracted from the fifth feature layer FL5 and the fourth feature X4 extracted from the fourth feature layer FL4 may be input into the encoder SE block 121, and the fourth feature X4 extracted from the fourth feature layer FL4 and the third feature X3 extracted from the third feature layer FL3 may be input into the encoder SE block 121, and this process may be performed by each encoder SE block 121. Finally, the second feature X2 extracted from the second feature layer FL2 and the first feature Xi extracted from the first feature layer FL1 may be input into the encoder SE block 121.
In addition, at the encoder SE step (S210), a global pooling process of generating the first channel information C1 may be performed by averaging and compressing the two features Xi and Xi+1.
In addition, at the encoder SE step (S210), the first major channel information CA1 may be determined from the first channel information C1 using a fully-connected (FC) function, and the first major channel information CA1 may be activated with a higher weighting value using a ReLu function. This series of processes may be referred to as a Squeeze process.
In addition, the encoder SE step (S210) may perform 1×1 convolution after expanding the compressed major channel information CA1 using a fully-connected (FC) function and a Sigmoid function and then scaling the size. This series of processes may be referred to as an excitation process. Here, the 1×1 convolution may reduce the number of parameters for the entire operation by reducing the channels using a filter having a size of 1×1.
Accordingly, since one or more features Xi may be extracted only from a monocular image captured by one camera, not a stereo image captured by two or more cameras, at the encoder SE step (S210), there is an effect of remarkably reducing the number of stored or processed images. In addition, since the first major channel information CA1 may be output using the features Xi and Xi+1 extracted from two different feature layers FLi and FLi+1 in the encoder 110, the operation parameters at the decoding step (S200) may be remarkably reduced in the future, and therefore, there is a remarkable effect of reducing the operation delay time.
Next, at the high-density step (S220), a feature set XCi may be output after learning the features according to the number of density layers DL and a growth rate, by the high-density block 122 in the decoder 120. That is, at the high-density step (S220), an arbitrary density layer DLi may output a feature set XCi by adding the learned features to the feature set XCi−1 obtained from the previous density layer DLi−1.
Most preferably, at the high-density step (S220), the dimension of the output channel of the high-density block 122 may be increased by 32 times according to the growth rate for each density layer DLi (i=1, 2, 3, 4), by the high-density block 122 including a first density layer DL1 to a fourth density layer DL4. Accordingly, at the high-density step (S220), a feature set XCi may be finally output in the form of high-density collective knowledge.
Meanwhile, at the high-density step (S220), 1×1 convolution may be performed on the feature set XCi to reduce the number of parameters for calculation. Accordingly, there is an effect of alleviating a gradient loss problem, enhancing feature propagation, and enabling feature reuse.
Next, at the up-sampling step (S230), a Nearest Neighbor Interpolation operation may be performed for the feature set XCi using double scaling by the up-sampling block 123 to improve the resolution of the depth map.
Meanwhile, at the up-sampling step (S230), 3×3 convolution may be performed on the feature set XCi so that the feature set XCi on which the 1×1 convolution is performed at the high-density step (S220) may be expanded and calculated.
Next, at the decoder SE step (S240), second channel information C2 may be generated from the feature set XCi up-sampled at the up-sampling step (S230) to enable channel attention, and second major channel information CA2 may be output from the second channel information C2, by the decoder SE block 124 in the decoder 120.
For example, at the decoder SE step (S240), first, the up-sampled feature set XCi may be input from the up-sampling block 123. Then, at the decoder SE step (S240), a global pooling process of generating the second channel information C2 by averaging and compressing the features aggregated in the feature set XCi at a high density may be performed.
In addition, at the decoder SE step (S240), the second major channel information CA2 may be determined from the second channel information C2 using a fully-connected (FC) function, and the second major channel information CA2 may be activated with a higher weighting value using a ReLu function. This series of processes may be referred to as a Squeeze process.
In addition, the decoder SE step (S240) may perform 1×1 convolution after expanding the compressed second major channel information CA2 using a fully-connected (FC) function and a Sigmoid function and then scaling the size. This series of processes may be referred to as an excitation process. Here, the 1×1 convolution may reduce the number of parameters for the entire operation by reducing the channels using a filter having a size of 1×1.
Therefore, since the second major channel information CA2 may be output at the decoder SE step (S240) using the features aggregated in the feature set XCi at a high density, the operation parameters of the decoder 120 may be reduced remarkably, and therefore, there is a remarkable effect of also reducing the operation delay time.
Next, referring to
Accordingly, most preferably, at the decoding step (S200), decoding may be performed using all of the feature Xi extracted from an arbitrary feature layer FL1 provided in the encoder 110, the first major channel information CA1, and the second major channel information CA2 to generate the depth map.
Therefore, as the method of generating a depth map using a monocular image of the present invention is provided with an encoding step (S100) and a decoding step (S200), there is a remarkable effect of applying the method to a drone or a small robot that should mount lightweight software and hardware as high accuracy and high resolution may be output and low delay time is also achieved compared to the prior art although the depth map is generated using as few parameters as only 4.1 million or so.
As described above, although the embodiments have been described through the limited embodiments and drawings, those skilled in the art may make various changes and modifications from the above descriptions. For example, although the described techniques are performed in an order different from that of the described method, and/or components such as the systems, structures, devices, circuits, and the like described above are coupled or combined in a form different from those of the described method, or replaced or substituted by other components or equivalents, an appropriate result can be achieved.
Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims described below.
According to the present invention as described above, as an encoder for extracting one or more features from a monocular image according to the number of feature layers, and a decoder for calculating displacements of mismatched pixels from the features extracted from different feature layers and generating a depth map for the monocular image are provided, there is an effect of applying the apparatus to a drone or a small robot that should mount lightweight software and hardware by achieving a low delay time together with high accuracy and high resolution compared to the prior art, although the depth map is generated with only a small number of parameters.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2021-0054824 | Apr 2021 | KR | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20210090280 | Guizilini | Mar 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 10-2103944 | Apr 2020 | KR |
| Entry |
|---|
| Howard et al., “Searching for MobileNetV3,” 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1314-1324. (Year: 2019). |
| Y. Wang, “MobileDepth: Efficient Monocular Depth Prediction on Mobile Devices,” 2020, in Computing Research Repository (CoRR), abs/2011.10189. (Year: 2020). |
| Z. Wang et al., “Deep Learning for Image Super-resolution: A Survey,” 2019, in Computing Research Repository (CoRR), abs/1902.06068. (Year: 2019). |
| Hu et al., “Squeeze-and-Excitation Networks,” 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132-7141 (Year: 2018). |
| Lyu et al., “HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation,” 2020, in Computing Research Repository (CoRR), abs/2011.10189. (Year: 2020). |
| Takagi et al., “Boosting Monocular Depth Estimation with Channel Attention and Mutual Learning,” 2019, 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR) (Year: 2019). |
| Li et al., “Deep attention-based classification network for robust depth prediction,” 2018, in Computing Research Repository (CoRR) , abs/1807.03959 (Year: 2018). |
| Rundo et al., “USE-Net: Incorporating Squeeze-and-Excitation blocks into U-Net for prostate zonal segmentation of multi-institutional MRI datasets,” Neurocomputing, vol. 365, pp. 31-43, 2019, https://doi.org/10.1016/j.neucom.2019.07.006 (Year: 2019). |
| Chen et al., “Unsupervised Monocular Depth Estimation with Encoder-decoder Network,” 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), Dali, China, 2019, pp. 2601-2606, doi: 10.1109/ROBIO49542.2019.8961751. (Year: 2019). |
| Durasov et al., “Double Refinement Network for Efficient Monocular Depth Estimation,” 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China, 2019, pp. 5889-5894, doi: 10.1109/IROS40897.2019.8968227. (Year: 2019). |
| Korean Office Action for related KR Application No. 10-2021-0054824 mailed Jul. 25, 2023 from Korean Intellectual Property Office. |
| Clement Godard et al., “Digging Into Self-Supervised Monocular Depth Estimation”, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 3828-3838. |
| Clement Godard et al., “Unsupervised Monocular Depth Estimation with Left-Right Consistency”, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 270-279. |
| Number | Date | Country | |
|---|---|---|---|
| 20220351399 A1 | Nov 2022 | US |
