Patent Application
20190370989
The disclosure of this patent document contains material which is subject to copyright protection. The copyright is owned by the copyright owner. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent records or archives.
The present application relates to the field of point cloud data processing and, more particularly, to a method and an apparatus for a 3-dimensional point cloud reconstruction.
In existing technologies, a three-dimensional (3D) point cloud reconstruction is usually performed based on a simultaneous localization and mapping (SLAM) algorithm. In a conventional SLAM algorithm, usually, feature points in image data are extracted first, and then a 3D point cloud is reconstructed based on the feature points in the image data (or points with notable gradients). In the image data, the number of the feature points (or the points with notable gradients) is relatively small. Thus, a 3D point cloud reconstructed based on the feature points (or the points with notable gradients) is a sparse (or semi-dense) 3D point cloud.
A sparse 3D point cloud may miss important information in a 3D scene, and is not suitable for a task that needs a relatively high accuracy on the 3D scene. For example, to reconstruct a 3D point cloud for machine navigation, a sparse 3D point cloud may likely miss important features of navigation. For example, a sparse 3D point cloud may miss a traffic light, or a sparse 3D point cloud may not provide accurate road information, such as information about whether a road in a 3D scene is allowed to travel on.
The present application provides a method and an apparatus for a 3-dimensional point cloud reconstruction to improve an accuracy of reconstructed 3-dimensional point cloud.
In one aspect, there is provided a method for reconstructing a three-dimensional point cloud. The method includes obtaining image data in a current view angle; generating voxels of a target object in the current view angle according to the image data, where the voxels of the target object include a first voxel, and the first voxel contains depth information; discarding the first voxel if a value of the depth information of the first voxel is not within a preset range; and fusing the first voxel with stored voxels if the value of the depth information of the first voxel is within the preset range.
In another aspect, there is provided an apparatus for reconstructing a three-dimensional point cloud. The apparatus includes an obtaining module configured to obtain image data in a current view angle; a generating module configured to generate, according to the image data, voxels of a target object in the current view angle, where a voxel of the target object includes a first voxel, and the first voxel contains depth information; a discarding module configured to discard the first voxel if a value of the depth information of the first voxel is not within a preset range; and a fusing module configured to fuse the first voxel with stored voxels if the value of the depth information of the first voxel is within the preset range.
In another aspect, there is provided an apparatus for reconstructing a three-dimensional point cloud. The apparatus includes a memory and a processor. The memory is used for storing instructions, and the processor is used for executing instructions stored in the memory to perform the methods described in the above aspects.
In another aspect, there is provided a computer-readable storage medium having instructions stored therein. The instructions cause a computer to perform the methods described in the above aspects when the instructions are executed on the computer.
In another aspect, there is provided a computer program product including instructions that cause a computer to perform the methods described in the above aspects when the instructions are executed on the computer.
The technical solution provided by the present application uses voxels as points in the three-dimensional point cloud, and can reconstruct a high accuracy dense three-dimensional point cloud. Further, in the technical solution provided in the present application, depth information is stored in voxels of the target object, and the voxels of the target object are screened based on the depth information of the voxels. Voxels having depth information that does not meet the requirement are discarded, which reduces the amount of point cloud data that needs to be fused and stored, and improves a real-time performance of the dense 3-dimensional cloud point reconstruction process.
The three-dimensional point cloud reconstructed by the SLAM algorithm may be mainly formed of feature points in the image data (or points with notable gradients). In the image data, the number of the feature points (or the points with notable gradients) may be relatively small. Thus, the 3D point cloud reconstructed by using the SLAM algorithm may be generally a sparse point cloud, and is not suitable for a task that needs a relatively high accuracy on the 3D scene.
A voxel is a small volume unit in a 3D space. An object in a 3D space may be usually made up of tens of thousands of piled-up voxels. Thus, the voxels of the object may have the characteristics of large number and dense distribution, and may be suitable for generating a dense 3D point cloud. Based on that, the embodiments of the present disclosure provide a 3D point cloud reconstruction method based on voxels to generate a dense 3D point cloud. Compared with a sparse point cloud, a dense 3D point cloud may have a relatively high accuracy, and can retain important information of the 3D scene.
In the existing technologies, generally, object voxels are usually stored in a 3D array, and each storage unit can be used to store a voxel. When it is needed to search for a target voxel of the object in the storage space, addressing is directly performed through subscripts of the 3D array. The voxel storage method based on the 3D array can reduce voxel search complexity. However, such a storage method needs to ensure a continuity of storage. That is, each voxel of the object needs to be stored. In order not to destroy the continuity of storage, even if a voxel is an empty voxel (a voxel that does not contain any useful information, such as a voxel inside the target object), a storage space needs to be allocated for the voxel, and voxels cannot be arbitrarily discarded. In fact, among object voxels, generally more than 90% of the voxels may be empty voxels. If these voxels are retained, a lot of storage and computing resources may be wasted, resulting in a relatively poor real-time performance in the voxel-based 3D point cloud reconstruction process and a limitation that only small-area and small-scale scenes, such as indoor scenes, can be reconstructed.
With reference to
At 210, image data in a current view angle is obtained.
At 220, voxels of a target object in the current view angle are generated according to the image data. A voxel of the target object includes a first voxel, also referred to as a “target voxel.” The first voxel contains depth information.
It should be understood that the above-described target object may be any one of objects detected in the current view angle. In addition, the first voxel may be any voxel in the voxels of the target object.
At 230, if a value of the depth information of the first voxel is not within a preset range, the first voxel is discarded.
At 240, if the value of the depth information of the first voxel is within the preset range, the first voxel is fused with stored voxels.
In the embodiments of the present disclosure, voxels may be used as points in a 3D point cloud, and a high accuracy dense 3D point cloud may be reconstructed. Further, in the embodiments of the present disclosure, depth information may be stored in voxels of the target object, and the voxels of the target object may be screened based on the depth information of the voxels. Voxels having depth information that does not meet the requirement may be discarded. As such, the amount of the point cloud data that needs to be fused and stored may be reduced, and a real-time performance of the dense 3D cloud point reconstruction process may be improved.
In the embodiments of the present disclosure, the selection manner of the preset range is not restricted, and may be chosen according to actual needs. For example, the above-described preset range may be set using a surface of the target object as a base, such that voxels in the vicinity of the surface of the object (or referred to as visible voxels) may be retained, and other voxels may be discarded. That is, the embodiments of the present disclosure focus on those among the voxels of the target object that store useful information, thereby making more efficient use of storage and computing resources to improve the real-time performance of the voxel-based 3D point cloud reconstruction process.
Step 220 describes a voxelization process of the target object. In the embodiments of the present disclosure, voxelization manner of the target object is not restricted. For example, step 220 may include generating a depth image in the current view angle according to the image data, and performing voxelization on the target object according to the depth image to obtain the voxels of the target object.
The above-described generating the depth image in the current view angle according to the image data may include estimating a parallax corresponding to each pixel in the image data, and converting the parallax corresponding to each pixel to a depth value corresponding to each pixel.
For example, formula zi=bf/di may be used to determine a depth value corresponding to each pixel, where zi and di represent a depth value and a parallax of an i-th pixel in the image data, respectively, b represents a baseline, and f represents a focal length.
It should be noted that, each pixel in the image data may correspond to a depth value, and solving processes of depth values corresponding to the pixels may be performed in parallel to improve a processing efficiency of the image data.
After the depth image is calculated, a 3D model of the target object is obtained. Then, a certain algorithm may be used to perform voxelization on the 3D model of the target object. For example, first, a bounding box of the 3D model is created according to the depth values in the depth image, and then the bounding box is decomposed into a plurality of small volume units based on octree. Each volume unit may correspond to a voxel of the target object.
If a target storage space includes a storage space of a first voxel, color information and depth information of the first voxel can be fused with color information and depth information in the storage space of the first voxel, respectively. The fusing method is described below in detail. There are various fusing methods for color information and depth information. The methods are described in detail below in conjunction with detail embodiments.
Optionally, in some embodiments, the first voxel may include color information and a weight of the color information of the first voxel, and the above-described fusing the first voxel with stored voxels may include: if the target storage space includes a storage space of the first voxel, obtaining a weighted sum of the color information of the first voxel and color information of voxel in the storage space of the first voxel according to the weight of the color information of the first voxel. Of course, the embodiments of the present disclosure are not limited thereto. Further, the color information of the first voxel may also be directly used to replace the color information of voxel in the storage space of the first voxel.
Optionally, in some embodiments, the first voxel may further include a weight of depth information of the first voxel. The above-described fusing the first voxel with the stored voxels may include: if the target storage space includes a storage space of the first voxel, obtaining a weighted sum of the depth information of the first voxel and the depth information of voxel in the storage space of the first voxel according to the weight of the depth information of the first voxel. Of course, the embodiments of the present disclosure are not limited thereto. Further, the depth information of the first voxel may also be directly used to replace the depth information of voxel in the storage space of the first voxel.
In the embodiments of the present disclosure, the manner of selecting a value for the depth information of the first voxel is not restricted. In some embodiments, the value of the depth information of the first voxel may be z value of the first voxel in a camera coordinate system. In some other embodiments, the value of the depth information of the first voxel may be a truncated signed distance function (tsdf) value. The tsdf value can be used to indicate a distance between the first voxel and a surface of the target object (a surface of the target object that is closest to the first voxel).
For example, the first voxel can use the following data structure:
where tsdf is the depth information of the first voxel and is used to represent the distance between the first voxel and the surface of the target object, color[3] is used to represent color information of the first voxel, and weight is used to represent the weight of the color information and/or the depth information of the first voxel.
Assuming that the preset range is (Dmin, Dmax), if tsdf value of the first voxel satisfies Dmin<tsdf<Dmax, it is considered that the first voxel contains useful information (or the first voxel is a visible voxel), and the first voxel is retained. If tsdf value of the first voxel does not satisfy Dmin<tsdf<Dmax, it is considered that the first voxel does not contain useful information (or the first voxel is an invisible voxel), and the first voxel is discarded.
It should be noted that the 3D point cloud reconstruction may be a continuous process. In the 3D point cloud reconstruction process, the mobile apparatus may first select an initial view angle (hereinafter referred to as a first view angle) to capture images, then perform voxelization on an object of the first view angle and store voxels having depth information that meets needs. Then, the mobile apparatus may adjust the camera's pose, select another view angle (hereinafter referred to as a second view angle) to capture images, and obtain new voxels. Because a difference between two adjacent view angles is usually small, the objects captured in the two adjacent view angles may overlap with each other. Thus, new voxels captured in the second view angle may need to be fused with the stored voxels. The foregoing current view angle may refer to any view angle in the camera's capturing process. A stored voxel may be a voxel that has been obtained and retained before capturing in the current view angle.
The implementation of step 240 (i.e., the manner of fusing voxels) can have various ways. In some embodiments, as shown in
At 242, it is determined whether the target storage space includes the storage space of the first voxel, where the target storage space is storage spaces allocated for the stored voxels.
At 244, if the target storage space includes the storage space of the first voxel, voxel information in the storage space of the first voxel is updated according to information of the first voxel.
At 246, if the target storage space does not include the storage space of the first voxel, a new storage space is allocated for the first voxel, and the first voxel is stored in the new storage space.
Various methods may be available for achieving step 244. For example, old voxel information stored in the storage space of the first voxel may be replaced directly with the information of the first voxel. As another example, the information of the first voxel and the old voxel information stored in the storage space of the first voxel may be weightedly summed.
For example, the voxel may use the following data structure:
The depth information of voxel stored in the storage space of the first voxel can be updated (or fused) using the following formula:
D
i+1=(Wi*Di+wi+1*di+1)/(Wi+wi+1), and
W
i+1
=W
i
+w
i+1,
where Di represents the tsdf value of voxel stored in the storage space of the first voxel, di+1 represents the tsdf value of the first voxel, Di+1 represents the updated tsdf value, Wi represents a weight of voxel stored in the storage space of the first voxel, wi+1 represents a weight of the first voxel, and Wi+1 represents the updated weight.
An example of how the depth information of the first voxel is updated is described above. Correspondingly, the color information of the first voxel may also be updated in a similar manner, which is not further described here in detail.
In some embodiments, step 242 may include searching for a storage space of the first voxel in the target storage space according to the first voxel and pre-stored mapping relationship information, where the mapping relationship information can be used for indicating correspondences between stored voxels and storage spaces of the stored voxels; if the storage space of the first voxel is found, determining that the target storage space includes the storage space of the first voxel; if the storage space of the first voxel is not found, determining that the target storage space does not include the storage space of the first voxel.
As described above, in the existing technologies, voxels are continuously stored in a 3D array, which may have the advantage of simple addressing, but needs to ensure a continuity of voxel storage. Even if a voxel is an empty voxel (a voxel that does not contain any useful information, such as a voxel inside the target object), a storage space needs to be allocated for the voxel, and voxels cannot be arbitrarily discarded. In the embodiments of the present disclosure, voxels in the target object that have depth information not meeting the requirement may be discarded. Although the continuity of the voxel storage is destroyed by doing so, a large number of empty voxels can be discarded, and the storage resources can be saved to a large extent to improve computation efficiency. Further, in the case that the voxel storage is discontinuous, in order to enable rapid addressing of the voxels, in the embodiments of the present disclosure, mapping relationship information may be introduced to record a correspondence between the voxels and the storage locations of the stored voxels.
How the mapping relationship information indicates a correspondence between the stored voxels and the storage spaces of the stored voxels may have a plurality of types of manners. In some embodiments, a correspondence between the positions of the stored voxels in the 3D space and the storage spaces of the stored voxels (such as a starting address and an offset of the storage address) can be established.
In some embodiments, the voxel positions in the 3D space may be represented by coordinates of voxels in the world coordinate system. In order to obtain coordinates of a voxel in the world coordinate system, a camera pose during capturing in the current view angle may be determined first, and then based on the camera pose, the voxel may be converted into the world coordinate system. Various methods may be used to determine the camera pose. For example, a fast odometry from vision (FOVIS) may be used to determine the camera pose. Specifically, first, input image data can be filtered using a Gaussian smoothing filter to construct a three-layer image pyramid. Then a features-from-accelerated-test (FAST) corner detector may be used to detect enough local features in the image. Then a random sample consensus (RANSAC) algorithm may be used to robustly estimate a camera attitude using a key frame in the image data.
Various methods may be used to store and use the mapping relationship information. In some embodiments, the mapping relationship information may be stored in a hash table. The above-described searching for the storage space of the first voxel in the target storage space according to the first voxel and the pre-stored mapping relationship information may include, according to the first voxel and a hash table, through a hash algorithm, searching for the storage space of the first voxel in the target storage space. The hash table may have advantages of fast addressing, and can improve an efficiency of the 3D point cloud reconstruction process.
Optionally, in some embodiments, the hash table may record a correspondence between position information of stored voxel blocks and storage spaces of the stored voxel blocks. A voxel block may contain a plurality of spatially adjacent voxels. The position information of the stored voxel block may be used to indicate a spatial position of the stored voxel block in a 3D scene (e.g., 3D coordinates of a center point of the voxel block in the world coordinate system). The above-described searching for the storage space of the first voxel in the target storage space through the hash algorithm, according to the first voxel and the hash table, may include determining position information of a target voxel block that the first voxel belongs to; searching for a storage space of the target voxel block through the hash algorithm, according to the position information of the target voxel block and the hash table; and according to a position of the first voxel in the target voxel block, searching for the storage space of the first voxel in the storage space of the target voxel block.
In the embodiments of the present disclosure, each entry of a hash table may not correspond to one voxel, but may correspond to a voxel block. Recording the mapping relationship information in a unit of voxel block can reduce the number of the hash table entries, thereby reducing a probability of hash conflicts.
The voxel block may be a collection of voxels having adjacent relationships in certain space. For example, a voxel block may include spatially adjacent 8*8*8 voxels.
In some embodiments, voxel blocks may all be stored in an array. The hash table in the embodiments of the present disclosure may be designed in order to quickly and efficiently find a voxel block in an array. A hash table entry can contain a position of the voxel block, a pointer pointing to an array that stores the voxel block, an offset, or another member variable.
For example, in some embodiments, the hash table may record the above-described mapping relationship information using the following data structure:
where pos[3] can represent world coordinates (x, y, z) of the voxel block in the 3D space, pointer can point to a starting address of an array storing the voxel block, offset can be used to indicate an offset between a starting address of a storage position of the voxel block and the starting address of the array storing the voxel block.
In a process of actual use, first, a target voxel block that the first voxel belongs to can be determined according to a spatial position relationship between the voxel and the voxel block. Then, according to the world coordinates (x, y, z) of the target voxel block in the 3D space and by searching in the hash table, it is determine whether a storage space has been allocated for the target voxel block. If the storage space has not been allocated for the target voxel block, then the storage space is allocated for the target voxel block and table entries of the hash table are updated. Then, information of the first voxel is stored in corresponding position in the storage space of the target voxel block. If the storage space has been allocated for the target voxel block, the storage space of the first voxel can be located in the storage space of the target voxel block, and then the information of the first voxel can be used to update the voxel information in the storage space of the first voxel. For detailed update methods for the voxel information, reference can be made to the foregoing descriptions, which are not further described here.
It should be noted that, in the embodiments of the present disclosure, locations of the target storage spaces are not restricted. The target storage spaces may be located in a memory of a graphics processing unit (GPU), or may be located in a memory of a central processing unit (CPU), or may even be located in an external storage device.
Optionally, in some embodiments, the target storage space may be located in a memory of a CPU and/or an external memory. Step 242 may include that a GPU determines whether the target storage space includes the storage space of the first voxel. Step 244 may include that the GPU reads the voxel information in the storage space of the first voxel from the target storage space through the CPU; that the GPU updates the voxel information in the storage space of the first voxel according to the information of the first voxel to obtain updated voxel information; and that the GPU stores the updated voxel information in the storage space of the first voxel through the CPU.
A GPU may have a strong image data processing capability, but a memory of the GPU may usually be relatively small, generally including 2-4 GB, which is not enough to store point cloud data of a large-scale scene. In order to avoid limitations of a GPU memory on a scale of a scene to be reconstructed, in the embodiments of the present disclosure, a CPU memory and/or an external storage device may be used to store voxels. Compared with the GPU memory (or referred to as a video memory), the CPU memory and/or the external storage device can provide a larger storage space.
Specifically, the GPU can first generate new voxels according to the image data. Then, before the new voxels are fused with the stored voxels, the GPU can read the voxels that need to be fused with the new voxels from the CPU memory and/or the external storage device into the GPU memory. Then, after voxel fusion is complete, the GPU can re-store the fused voxel data into the CPU memory and/or the external storage device. That is, in the embodiments of the present disclosure, data exchange may be based on a GPU, a CPU and/or an external storage device, such that the GPU memory may only need to store local data that is to be fused, and global data may be always stored in the CPU memory and/or the external storage device. Accordingly, the limitations of the GPU memory space may be overcome, allowing the embodiments of the present disclosure to be used for reconstructing a three-dimensional space of a larger dimension.
For example, in a 3D point cloud map reconstruction, the embodiments of the present disclosure may include using the CPU memory to store a reconstructed global map, using the GPU memory to store a local map, and constantly updating the local map stored in the GPU memory according to a scene change or a change of view angle, such that a limitation caused by the relatively small GPU memory on the scale of the scene to be reconstructed may be overcome.
It should be understood that the target object in the current view angle may be a stationary object, or may be a moving object. In the 3D point cloud reconstruction, what needs to be reconstructed may be one or more relatively stationary objects in a 3D scene, such as a building, a traffic light, etc. A moving object that appears temporarily in the 3D scene can be considered as an interference. Thus, a method for processing a moving object is provided below. The method can identify and effectively handle a moving object to speed up the 3D point cloud reconstruction process.
Step 220 may include determining whether the target object is a moving object according to the image data; if the target object is a moving object, generating voxels of the target object, such that a value of depth information of each voxel in the target object is not within the preset range.
In the embodiments of the present disclosure, the value of the depth information of the moving object may be controlled to be outside the preset range, such that each voxel in the target object is discarded. That is, in the embodiments of the present disclosure, processing the moving object can be avoided through controlling the value of the depth information, and the efficiency of the 3D point cloud reconstruction process can be improved.
Assume each voxel in the target object uses the following data structure:
where tsdf is depth information of the voxel and is used to represent a distance between the first voxel and a surface of the target object. Assuming that the preset range can be (Dmin, Dmax), if the tsdf value of the voxel satisfies Dmin<tsdf<Dmax, the voxel may be retained; and if the tsdf value of the voxel does not satisfy D n<tsdf<Dmax, the voxel may be discarded.
When the target object is identified as a moving object, a tsdf value of each voxel in the target object can be controlled to not satisfy Dmin<tsdf<Dmax and hence the voxels in the target object can be discarded. As such, handling of the moving object can be avoided and the three-dimensional point cloud reconstruction process can be sped up.
It should be noted that, in the embodiments of the present disclosure, the manner of determining whether the target object is a moving object according to the image data is not restricted. An existing algorithm for detecting and tracking a moving object may be used.
The apparatus embodiments of the present disclosure are described below. Because the apparatus embodiments can perform the above-described methods, foregoing method embodiments can be referenced to for those not described in detail.
In the embodiments of the present disclosure, voxels may be used as points in a 3D point cloud, and a high accuracy dense 3D point cloud may be reconstructed. Further, in the embodiments of the present disclosure, depth information may be stored in voxels of the target object, and the voxels of the target object may be screened based on the depth information of the voxels. Voxels having depth information that does not meet the requirement may be discarded. As such, the amount of point cloud data that needs to be fused and stored may be reduced, and a real-time performance of the dense 3D cloud point reconstruction process may be improved.
Optionally, in some embodiments, the fusing module 440 may be configured to determine whether the target storage space includes a storage space of the first voxel, where the target storage space is a storage space allocated for the stored voxels; if the target storage space includes the storage space of the first voxel, update voxel information in the storage space of the first voxel according to information of the first voxel; and, if the target storage space does not include the storage space of the first voxel, allocate a new storage space for the first voxel and store the first voxel in the new storage space.
Optionally, in some embodiments, the fusing module 440 may be configured to search for the storage space of the first voxel in the target storage space according to the first voxel and pre-stored mapping relationship information, where the mapping relationship information can be used for indicating correspondences between stored voxels and storage spaces of the stored voxels; if the storage space of the first voxel is found, determine that the target storage space includes the storage space of the first voxel; and, if the storage space of the first voxel is not found, determine that the target storage space does not include the storage space of the first voxel.
Optionally, in some embodiments, the mapping relationship information may be stored in a hash table, and the fusing module 440 may be configured to search for, according to the first voxel and the hash table, the storage space of the first voxel in the target storage space through a hash algorithm.
Optionally, in some embodiments, the hash table may record the correspondence between the position information of the stored voxel block and the storage space of the stored voxel block. A voxel block may include a plurality of spatially adjacent voxels, and position information of the stored voxel block may be used to indicate a spatial position of the stored voxel block in a 3D scene. The fusing module 440 may be configured to determine position information of target voxel block that the first voxel belongs to; search for a storage space of the target voxel block through a hash algorithm, according to the position information of the target voxel block and the hash table; and according to a position of the first voxel in the target voxel block, search for the storage space of the first voxel in the storage space of the target voxel block.
Optionally, in some embodiments, the target storage space may be located in a memory of a CPU and/or an external storage device. The fusing module 440 may be configured to call a GPU to perform following operations, including: determining whether the target storage space includes the storage space of the first voxel; reading voxel information in the storage space of the first voxel from the target storage space through the CPU; updating voxel information in the storage space of the first voxel according to information of the first voxel to obtain updated voxel information; and storing the updated voxel information in the storage space of the first voxel through the CPU.
Optionally, in some embodiments, the generating module 420 may be configured to determine whether the target object is a moving object according to the image data; and if the target object is a moving object, generate voxels of the target object, such that a value of depth information of each voxel in the target object is not within the preset range.
Optionally, in some embodiments, the generating module 420 may be configured to generate a depth image in the current view angle according to the image data; and perform voxelization on the target object according to the depth image to obtain voxels of the target object.
Optionally, in some embodiments, the first voxel may include color information and a weight of the color information. The fusing module 440 may be configured to, if the target storage space includes a storage space of the first voxel, obtain a weighted sum of the color information of the first voxel and the color information of voxel in the storage space of the first voxel according to the weight of the color information of the first voxel.
Optionally, in some embodiments, the first voxel may also include a weight of the depth information. The fusing module 440 may be configured to, if the target storage space includes the storage space of the first voxel, obtain a weighted sum of the depth information of the first voxel and the depth information of the voxel in the storage space of the first voxel according to the weight of the depth information of the first voxel.
Optionally, in some embodiments, a value of the depth information of the first voxel may be a truncated signed distance function value. The truncated signed distance function value may be used to indicate a distance between the first voxel and a surface of the target object.
In the embodiments of the present disclosure, voxels may be used as points in a 3D point cloud, and a high accuracy dense 3D point cloud may be reconstructed. Further, in the embodiments of the present disclosure, depth information may be stored in voxels of the target object, and the voxels of the target object may be screened based on the depth information of the voxels. Voxels having depth information that does not meet the requirement may be discarded. As such, the amount of point cloud data that needs to be fused and stored may be reduced, and a real-time performance of the dense 3D cloud point reconstruction process may be improved.
Optionally, in some embodiments, fusing the first voxel with the stored voxels may include determining whether the target storage space includes a storage space of the first voxel, where the target storage space is a storage space allocated for the stored voxels; if the target storage space includes the storage space of the first voxel, updating voxel information in the storage space of the first voxel according to information of the first voxel; and, if the target storage space does not include the storage space of the first voxel, allocating a new storage space for the first voxel and storing the first voxel in the new storage space.
Optionally, in some embodiments, determining whether the target storage space includes the storage space of the first voxel may include searching for the storage space of the first voxel in the target storage space according to the first voxel and pre-stored mapping relationship information, where the mapping relationship information can be used for indicating correspondences between stored voxels and storage spaces of the stored voxels; if the storage space of the first voxel is found, determining that the target storage space includes the storage space of the first voxel; and, if the storage space of the first voxel is not found, determining that the target storage space does not include the storage space of the first voxel.
Optionally, in some embodiments, the mapping relationship information may be stored in a hash table. Searching for the storage space of the first voxel in the target storage space according to the first voxel and the pre-stored mapping relationship information may include searching for, according to the first voxel and the hash table, the storage space of the first voxel in the target storage space through a hash algorithm.
Optionally, in some embodiments, the hash table may record a correspondence between position information of a stored voxel block and a storage space of the stored voxel block. A voxel block may include a plurality of spatially adjacent voxels, and position information of the stored voxel block may be used to indicate a spatial position of the stored voxel block in a 3D scene. Searching for the storage space of the first voxel in the target storage space through the hash algorithm according to the first voxel and the hash table may include determining position information of the target voxel block that the first voxel belongs to; searching for a storage space of the target voxel block through the hash algorithm, according to the position information of target voxel block and the hash table; and according to a position of the first voxel in the target voxel block, searching for the storage space of the first voxel in the storage space of the target voxel block.
Optionally, in some embodiments, the target storage space may be located in a memory of a CPU and/or an external memory. A GPU can perform both the operation of determining whether the target storage space includes a storage space of the first voxel and the operation of updating voxel information in the storage space of the first voxel according to information of the first voxel. Further, determining whether the target storage space includes the storage space of the first voxel may include operations including determining whether the target storage space includes the storage space of the first voxel. Updating the voxel information in the storage space of the first voxel according to the information of the first voxel may include reading the voxel information in the storage space of the first voxel from the target storage space through the CPU; updating the voxel information in the storage space of the first voxel according to the information of the first voxel to obtain updated voxel information; and storing the updated voxel information in the storage space of the first voxel through the CPU.
Optionally, in some embodiments, generating voxels of the target object in the current view angle according to the image data may include determining whether the target object is a moving object according to the image data; and, if the target object is a moving object, generating voxels of the target object, such that a value of depth information of each voxel in the target object is not within a preset range.
Optionally, in some embodiments, generating voxels of the target object in the current view angle according to the image data may include generating a depth image in the current view angle according to the image data; and performing voxelization on the target object according to the depth image to obtain voxels of the target object.
Optionally, in some embodiments, the first voxel may include color information and a weight of the color information. Fusing the first voxel with the stored voxels may include, if the target storage space includes a storage space of the first voxel, obtaining a weighted sum of the color information of the first voxel and the color information of voxel in the storage space of the first voxel according to the weight of the color information of the first voxel.
Optionally, in some embodiments, the first voxel may further include a weight of the depth information. Fusing the first voxel with the stored voxels may include, if the target storage space contains the storage space of the first voxel, obtaining a weighted sum of the depth information of the first voxel and the depth information of voxel in the storage space of the first voxel according to the weight of the depth information of the first voxel.
Optionally, in some embodiments, a value of the depth information of the first voxel may be a truncated signed distance function value. The truncated signed distance function value may be used to indicate a distance between the first voxel and a surface of the target object.
The above-described embodiments may be implemented in whole or in part by software, hardware, firmware, or any other combination. When the above-described embodiments are implemented using software, the implementation may be performed in whole or in part in the form of a computer program product. The computer program product may include one or more computer instructions. The process or function described in accordance with the embodiments of the present disclosure may be generated in whole or in part when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a dedicated computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or may be transferred from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transferred from a website, a computer, a server, or a data center to another website, computer, server or data center, through a wired connection (such as coaxial cable, optical fiber, or digital subscriber line (DSL)) or a wireless connection (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that the computer can access or a data storage device such as a server, a data center, or the like that contains one or more integrated available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid state disks (SSDs)), etc.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of each of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for different application scenarios, but such implementations should not be considered as beyond the scope of the present application.
It should be understood that, in the embodiments provided in the present application, the disclosed system, apparatus, and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely for illustrative purposes. For example, the division of units may only be a logical function division, and there may be other ways of dividing the units in actual implementation. For example, multiple units or components can be combined or may be integrated into another system, or some features can be ignored, or not executed. Further, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or a communication connection through some interfaces, devices, or units, which may be electrical, mechanical, or in other form.
The units described as separate components may or may not be physically separate, and a component shown as a unit may or may not be a physical unit. That is, the units may be located in one place or may be distributed over a plurality of network elements. Some or all of the units may be selected according to the actual needs to achieve the object of the solution of the embodiments.
In addition, the functional units in the various embodiments of the present application may be integrated in one processing unit, or each unit may exist physically as an individual unit, or two or more units may be integrated in one unit.
The foregoing descriptions are merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art will be able to easily make variations or substitutions within the technical scope disclosed in the present application, all of which are within the protection scope of the present application. Thus, the protection scope of the present application should be based on the protection scope of the claims.
This application is a continuation of International Application No. PCT/CN2017/073874, filed Feb. 17, 2017, the entire content of which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2017/073874 | Feb 2017 | US |
| Child | 16541886 | US |
