Patent Application
20230410254
The present principles generally relate to the domain of point cloud processing. The present document is also understood in the context of the analysis, the interpolation, the representation and the understanding of point cloud signals.
The present section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present principles that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present principles. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
Point cloud is a data format used across several business domains including autonomous driving, robotics, AR/VR, civil engineering, computer graphics, and the animation/movie industry. 3D LIDAR sensors have been deployed in self-driving cars, and affordable LIDAR sensors are included with, for example, Apple iPad Pro 2020 and Intel Real Sense LIDAR camera L515. With advances in sensing technologies, three-dimensional (3D) point cloud data has become more practical and is expected to be a valuable enabler in the applications mentioned.
At the same time, point cloud data may consume a large portion of network traffic, e.g., among connected cars over a 5G network, and immersive communications (virtual or augmented reality (VR/AR)). Point cloud understanding and communication would essentially lead to efficient representation formats. In particular, raw point cloud data need to be properly organized and processed for the purposes of world modeling and sensing.
Furthermore, point clouds may represent a sequential scan of the same scene, which contains multiple moving objects. These are called dynamic point clouds as compared to static point clouds captured from a static scene or static objects. Dynamic point clouds are typically organized into frames, with different frames being captured at different time.
3D point cloud data are essentially discrete samples of the surfaces of objects or scenes. To fully represent the real world with point samples, in practice, a large number of points is required. For instance, a typical VR immersive scene contains millions of points, while point cloud maps typically contain hundreds of millions of points. Therefore, the processing of such large-scale point clouds is computationally expensive, especially for consumer devices that have limited computational power, e.g., smartphones, tablets, and automotive navigation systems.
Point cloud data are key for various applications, such as autonomous driving, VR/AR, topography and cartography, etc. However, consuming a large point cloud directly incurs significant computational costs. Consequently, it is important to adaptively down-sample the input point cloud to facilitate subsequent tasks. Such a down-sampling process is useful for scene-flow estimation, point cloud compression, and other general computer vision tasks.
The following presents a simplified summary of the present principles to provide a basic understanding of some aspects of the present principles. This summary is not an extensive overview of the present principles. It is not intended to identify key or critical elements of the present principles. The following summary merely presents some aspects of the present principles in a simplified form as a prelude to the more detailed description provided below.
The present principles relate to a method that generates, using a neural network, a point-level feature vector for each point of a point cloud and a set-level feature vector for the point cloud. A representative position based on the point-level feature vectors and on the set-level feature vector is generated. The representative position and the set-level feature vector are output as a set descriptor.
In another embodiment, a method for retrieving a point cloud from a data stream obtains, from the data stream, a down-sampled point cloud and a residual point cloud. The down-sampled point cloud is fed to a predictor construction module to obtain a predicted point cloud. The point cloud is retrieved by adding the predicted point cloud to the residual point cloud.
The present principles also relate to device comprising at least one processor associated with at least one memory configured to implement embodiments corresponding to the methods above.
The present disclosure will be better understood, and other specific features and advantages will emerge upon reading the following description, the description making reference to the annexed drawings wherein:
The present principles will be described more fully hereinafter with reference to the accompanying figures, in which examples of the present principles are shown. The present principles may, however, be embodied in many alternate forms and should not be construed as limited to the examples set forth herein. Accordingly, while the present principles are susceptible to various modifications and alternative forms, specific examples thereof are shown by way of examples in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the present principles to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present principles as defined by the claims.
The terminology used herein is for the purpose of describing particular examples only and is not intended to be limiting of the present principles. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising,” “includes” and/or “including” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Moreover, when an element is referred to as being “responsive” or “connected” to another element, it can be directly responsive or connected to the other element, or intervening elements may be present. In contrast, when an element is referred to as being “directly responsive” or “directly connected” to other element, there are no intervening elements present. As used herein the term “and/or” includes any and all combinations of one or more of the associated listed items and may be abbreviated as “/”.
It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the teachings of the present principles.
Although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.
Some examples are described with regard to block diagrams and operational flowcharts in which each block represents a circuit element, module, or portion of code which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in other implementations, the function(s) noted in the blocks may occur out of the order noted. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.
Reference herein to “in accordance with an example” or “in an example” means that a particular feature, structure, or characteristic described in connection with the example can be included in at least one implementation of the present principles. The appearances of the phrase in accordance with an example” or “in an example” in various places in the specification are not necessarily all referring to the same example, nor are separate or alternative examples necessarily mutually exclusive of other examples.
Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims. While not explicitly described, the present examples and variants may be employed in any combination or sub-combination.
The automotive industry and autonomous cars are domains in which point clouds may be used. Autonomous cars should be able to “probe” their environment to make good driving decisions based on the reality of their immediate surroundings. Typical, sensors like LIDARs produce (dynamic) point clouds that are used by a decision engine. These point clouds are not intended to be viewed by human eyes and they are typically sparse, not necessarily colored, and dynamic with a high frequency of capture. They may have other attributes like the reflectance ratio provided by the LIDAR as this attribute is indicative of the material of the sensed object and may help in making a decision.
Virtual Reality (VR) and immersive worlds have become widely discussed, foreseen by many as the future of 2D flat video. The basic idea is to immerse the viewer in an environment all around him as opposed to standard TV where he only views the virtual world in front of him. There are several gradations in the immersivity depending on the freedom of the viewer in the environment. Point cloud is a good format candidate to distribute VR worlds. They may be static or dynamic and are typically of average size, for example, no more than millions of points at a time.
Point clouds may be also used for various purposes such as cultural heritage/buildings in which objects like statues or buildings are scanned in 3D in order to share the spatial configuration of the object without sending or physically visiting it. This also provides a way to preserve the information and data about the object in case it may be destroyed; for instance, a temple by an earthquake. Such point clouds are typically static, colored, and relatively large.
Another use case is in topography and cartography in which using 3D representations and maps are not limited to a plane and may include relief features. Google Maps is one example of 3D maps that uses meshes instead of point clouds. Nevertheless, point clouds may be a suitable data format for 3D maps and such point clouds are typically static, colored, and relatively large.
World modeling & sensing via point clouds could be a technology to allow machines to gain knowledge about the 3D world around them, helpful for the applications discussed above.
3D point cloud data are essentially discrete samples of the surfaces of objects or scenes. To fully represent the real world with point samples, in practice, a large number of points is required. Therefore, the processing of such large-scale point clouds is computationally expensive, especially for consumer devices that have limited computational power, e.g., smartphones, tablets, and automotive navigation systems.
To process the input point cloud at an affordable computational cost, one solution is to down-sample it first, where the down-sampled point cloud summarizes the geometry of the input point cloud while having significantly fewer points. The down-sampled point cloud is then fed to the subsequent machine task for further consumption. However, point cloud data can be exploited for various tasks, such as scene flow estimation, classification, detection, segmentation, and compression, etc. Different tasks focus on different aspects of a point cloud. For instance, classification relies on the saliency points of the geometry, while object segmentation needs to distinguish the points on one object from the others, and scene flow estimation counts the dynamics of a point cloud. Hence, an adaptive point cloud down-sampling algorithm that is task-aware is helpful. Therefore, when faced with different tasks, the same point cloud can be down-sampled to different ones to facilitate the subsequent tasks.
According to the present principles, given a point set (and other auxiliary information if available), the SD function first computes a point-level feature vector for each point in the point set, and a set-level feature vector describing the overall point set. This step is accomplished, for example, using a neural network module (herein called P-Net) structured according to the present principles. By taking as inputs the point-level feature vectors of each point and the set-level feature vector, a representative position is computed. This step is achieved through either a deterministic approach or another neural network module. After that, the SD function outputs the representative position, as well as the set-level feature to represent the geometry of the point set. By using the SD function, the obtained representative position is not limited to the points within the point set.
At step 14, the m representative points are aggregated as the updated down-sampled point cloud, which is fed to the subsequent task for further processing. The m set-level features are also optionally output and fed to the subsequent task.
Down-sampling method 10 is integrated with the subsequent task and trained in an end-to-end manner, allowing down-sampling method 10 be task-aware, i.e., adaptive to the machine task. On the other hand, through end-to-end training, the down-sampled point clouds obtained by method 10 are able to capture the underlying geometry for a particular machine task, regardless of the how original input point cloud is sampled from the scene. Specifically, given two different point clouds which sample the same surface, i.e., one point cloud is the resampled version of the other, for the same subsequent machine task, method 10 results in two down-sampled point clouds that are closely resemble each other.
According to the present principles, a set of weights 26 is computed for the points in the whole point set. To do so, an affinity value (e.g., weight estimate) between each point-level feature vector 22 and the set-level feature vector 24, is provided by a module 25 that computes the inner-product between them. This affinity value describes the of the degree to which its associated point is representative of the whole point set. A module 27 performs a weighted average 28 of the points with the computed weights 26, to generate a representative position for the point set. The affinity values are converted to a set of weights using the Softmax(·) function, so that all the weight values are greater than 0 and summed up to 1. A weighted averaging of the x coordinates of all points in the point set with the obtained weights is performed, leading to the x coordinate of the generated representative point. Similarly, the y and z coordinates of the representative point are computed with the weights. The generated x, y and z coordinates form the position of the representative point. The SD function outputs the representative point, as well as the set-level feature generated by PointNet.
In a first embodiment of the present principles, the down-sampling of a given point cloud X containing n points is performed using the presented SD function. At a first step, an initial down-sampled point cloud with m points is generated using the Farthest point sampling (FPS) method, where the obtained points are called the “anchor points”. Farthest point sampling is a known point cloud down-sampling approach and is described for instance in “The Farthest point strategy for progressive image sampling,” IEEE Trans. on Image Processing, vol. 6, no. 9, pp. 1306-1315, 1997. FPS is based on repeatedly choosing the next sample point in the least-explored area. Given a point cloud X and its sampled subset, the FPS algorithm chooses the farthest point to the subset from the rest of the points in X with some distance measure. This farthest point is then added to the subset. Here the subset is initialized by randomly picking a point from X. The FPS algorithm repeats this point selection process until a certain condition is met, e.g., the number of points in the subset reaches a predefined threshold. This classic sampling approach is deterministic and does not consider the downstream task.
At a second step of the first embodiment, for each anchor point, its nearby points are collected through a ball query procedure, i.e., all points in X lying within a predefined distance r to the anchor point are identified and collected, forming a local point set for that anchor point. At a third step, every local point set (m in total) is fed to the SD function individually, leading to the updated down-sampled point cloud (with m points), accompanied with m set-level features. At a fourth step, the m down-sampled points (and optionally, the set level features) are fed to the subsequent task. This down-sampling method is trained end-to-end with the subsequent machine task, to make the neural network layers in the SD function task-aware, i.e., be adaptive to the subsequent task.
In a second embodiment, the computation of the point-wise weights of the SD function differs. Specifically, in the SD function of this embodiment a distance is computed for each point in the point set, which is the Euclidean distance between its point-level feature vector, and the set-level feature vector. Herein, this distance value is denoted by di for a point i. di value is plugged into a Gaussian kernel to compute a weight, i.e., wi=exp(−di2/σ2) with a constant σ. The weight values of the point set are further normalized so that they summed up to one. Then, a weighted averaging of the points in the point set is performed using the obtained weights, as presented in relation to the first embodiment, leading to the representative point position. The SD function returns the representative point as well as the set-level feature obtained by the PointNet.
In a third embodiment, each down-sampled point is obtained by selecting a critical point in a local point set. A difference between this embodiment and the first embodiment lies in the SD function, where the SD function here chooses a representative point from the input point set. Similar to the first embodiment, given a point set, the SD function computes a set of weights for each point in the whole set. Then the SD function directly returns the point with the maximum weight as the representative point, as well as returns the set-level feature generated by PointNet.
In a fourth embodiment, the SD function takes as inputs not only a local point set from the point cloud X but also a one-hot vector indicating which point is the anchor point of the point set. Consequently, the SD function in this embodiment can utilize the knowledge of the anchor point position to generate the representative point. Specifically, in the SD function, before the computation of the PointNet, the position vector of each point in the point set is augmented, by appending the position vector of the anchor point (and the feature vector of the anchor, which is another input to the SD function, if available). With the information of the anchor position appended, the augmented point set is then processed by the PointNet, leading to the point-level feature vectors of each point and the set-level feature vector.
In this sixth embodiment, the down-sampling methods presented in the previous embodiments are applied multiple times. The overall neural network architecture of this embodiment takes an hour-glass structure with skip connections. The method of the present sixth embodiment comprises a first stage generating a first and a second down-sampled point clouds, from the first and the second point cloud frames, respectively. This is achieved using two task-aware down-sampling modules 54a and 54b (based on any one of the previous embodiments) for both inputs. Two consecutive point clouds of a point cloud sequence may be considered as one point cloud in which points carry a temporal information indicating whether they belong to the first or to the second point cloud. Indeed, two point clouds of a sequence of point clouds share the same frame of reference and their points can be merged in one point cloud. At a second stage, a point set for each point in the first down-sampled point cloud is aggregated by searching for its nearest-neighboring points from the second down-sampled point cloud. The method computes a first inter-frame feature fusing the information from both point cloud frames, for each point in the first down-sampled point cloud, using the information of the point (its position and the point-level feature) as well as its associated nearest-neighboring point set. This second stage is accomplished using a neural network module 55, herein called “F1-Net”. At a third stage, the first down-sampled point cloud is further down-sampled with a task-aware down-sampling module 54c according to the present principles, taking the points and the associated inter-frame features as inputs. At a fourth stage, a second inter-frame feature is computed for each point in the first point cloud frame, using an up-sampled neural network module 56, herein called “F2-Net”. This F2-Net module corresponds to stacks of Set Up-Cony layers, for example as presented in “FlowNet3D: Learning scene flow in 3D point clouds,” in proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 529-537, 2020. Such a neural network module interpolates the point-wise features hierarchically. At a fifth stage, a scene flow vector is computed for each point in the first point cloud frame, using a feature-to-flow transformation neural network module 57, herein called “F3-Net”. F3-Net is implemented with pointwise MLP layers. According to the present principles, skip-connections between the task-aware down-sampling module and the F2-Net are added to merge information from the early layers.
The entire neural network architecture of
In
With the first down-sampled point cloud and all the updated nearest-neighboring point sets, a second point-wise scene flow for the first point cloud frame, by executing F1-Net 55, the second task-aware down-sampling module 54c, F2-Net 56, and F3-Net 57 again. An alternative of this stage is that, based on the shifted down-sampled point cloud, and the updated nearest-neighboring point sets, executing F1-Net, the second task-aware down-sampling module, F2-Net, and F3-Net again, leading to a residual point-wise scene flow. By adding the residual point-wise scene flow to the initial point-wise scene flow, a second point-wise scene flow for the first point cloud frame can be obtained. In the end, the second point-wise scene flow is output as the result. This recurrent scene flow estimation scheme can be executed iteratively for more than two iterations until a certain condition is satisfied, e.g., the number of iterations reaches a predefined threshold.
For this seventh embodiment, integrate the task-aware point cloud down-sampling method is described in relation to a scene flow estimation. The same iterative principles may apply without loss of generality to any other task dealing with point clouds as described above.
This decoder embodiment can be used for either inter-frame predictive coding or intra-frame predictive coding. It differs from conventional scalable coding in two aspects. On one hand, the decoder does not limit the down-sampled point cloud to be a subset of the input point cloud, On the other hand, aside from the down-sampled point cloud, the feature vectors produced the task-aware down-sampling module of the encoder in relation to
Device 30 comprises the following elements that are linked together by a data and address bus 31:
In accordance with an example, the power supply is external to the device. In each of the mentioned memory, the word «register» used in the specification may correspond to an area of small capacity (some bits) or to very large area (e.g. a whole program or large amount of received or decoded data). The ROM 33 comprises at least a program and parameters. The ROM 33 may store algorithms and instructions to perform techniques in accordance with present principles. When switched on, the CPU 32 uploads the program in the RAM and executes the corresponding instructions.
The RAM 34 comprises, in a register, the program executed by the CPU 32 and uploaded after switch-on of the device 30, input data in a register, intermediate data in different states of the method in a register, and other variables used for the execution of the method in a register.
The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.
In accordance with examples of the present disclosure, the device 30 belongs to a set comprising:
Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and/or depth information. Examples of such equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices. As should be clear, the equipment may be mobile and even installed in a mobile vehicle.
Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions (and/or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD”), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory (“RAM”), or a read-only memory (“ROM”). The instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of the two. A processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.
As will be evident to one of skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax-values written by a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.
A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill will understand that other structures and processes may be substituted for those disclosed and the resulting implementations will perform at least substantially the same function(s), in at least substantially the same way(s), to achieve at least substantially the same result(s) as the implementations disclosed. Accordingly, these and other implementations are contemplated by this application.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2021/059074 | 11/12/2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63119129 | Nov 2020 | US |
