Patent Grant
12223745
The present invention relates to the technical field of data annotation, in particular to a pavement element annotation method for point cloud data with fusion of height and a device for constructing and pre-annotation point cloud data with fusion of height.
The development of modern high-tech technologies such as autonomous driving and unmanned aerial vehicles depends on high-precision point cloud data, especially the information of pavement elements. Therefore, it is very important to construct an accurate method of pavement element identification and annotation. However, the conventional annotation method of pavement elements needs professional manual work, which is time-consuming and labor-intensive. At the same time, there are also problems of poor annotation error and consistency, especially in complex environment, which may cause misjudgment of navigation system and bring security risks.
An existing solution is to annotate pavement elements by using high-precision point cloud data collected by lidar, such as the method and device for annotation point cloud data disclosed in U.S. patent US20180108146A1, a method and device for annotating point cloud data. A specific implementation of the method includes: collecting data in a given scenario by using a laser radar and a non-laser radar sensor to respectively obtain point cloud data and sensor data; segmenting and tracking the point cloud data to obtain point cloud segmentation and tracking results; recognizing and tracking feature objects in the sensor data to obtain feature object recognition and tracking results; correcting the point cloud segmentation and tracking results by using the feature object recognition and tracking results, to obtain confidence levels of the point cloud recognition and tracking results; and determining a point cloud segmentation and tracking result whose confidence level is greater than a confidence level threshold as a point cloud annotation result. However, the processing complexity of point cloud data is large, which requires high hardware equipment for annotators. In addition, due to the characteristics of point cloud data obtained by lidar, the annotator needs to consider X, Y and Z dimensions at the same time in the annotation process, which leads to a lot of time to label and affects the annotation efficiency. In addition, single-frame point cloud data often have occlusion and holes in some areas, which leads to incomplete annotation.
The U.S. invention patent US20180075666A1 discloses a method and a device for processing point cloud data, wherein an obstacle recognition algorithm is used to recognize an object in a point cloud frame to be annotated, and a recognition result is obtained; the recognition result is taken as the initial annotation result of the point cloud frame; the annotation result is updated according to the user's correction operation on the annotation result. However, the above solution still fails to avoid the problem of incomplete annotation.
The terms “invention,” “the invention,” “this invention” and “the present invention” used in this patent are intended to refer broadly to all of the subject matter of this patent and the patent claims below. Statements containing these terms should be understood not to limit the subject matter described herein or to limit the meaning or scope of the patent claims below. Embodiments of the invention covered by this patent are defined by the claims below, not this summary. This summary is a high-level overview of various embodiments of the invention and introduces some of the concepts that are further described in the Detailed Description section below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings and each claim.
In order to solve the problems of high hardware requirements, low annotation efficiency and incomplete annotation when using traditional point cloud data to annotate pavement elements in the prior art, the present invention provides a pavement element annotation method for point cloud data with fusion of height, a device for constructing and pre-annotation point cloud data with fusion of height, an electronic device and a computer-readable storage medium.
In order to achieve the above object, the technical solutions of the present invention are as follows:
On the other hand, the present invention further provides:
In another aspect, the present invention further provides:
Compared with the prior art, the present invention has the advantages that the advantages of automatic pre-annotation and manual adjustment are comprehensively utilized, the annotation time is greatly saved, the annotation error is reduced, and the annotation quality is also improved; accurate annotation of pavement elements is helpful to improve the positioning, navigation and obstacle avoidance performance in the fields of automatic driving, robot navigation and so on, thus enhancing the safety and reliability of these systems.
In order to explain the technical solutions of this application more clearly, the drawings needed in the implementation will be briefly introduced below. Obviously, the drawings described below are only some implementations of this application. For those skilled in the art, other drawings can be obtained according to these drawings without creative work.
In describing the preferred embodiments, specific terminology will be resorted to for the sake of clarity. It is to be understood that each specific term includes all technical equivalents which operate in a similar manner to accomplish a similar purpose.
While various aspects and features of certain embodiments have been summarized above, the following detailed description illustrates a few exemplary embodiments in further detail to enable one skilled in the art to practice such embodiments. Reference will now be made in detail to embodiments of the inventive concept, examples of which are illustrated in the accompanying drawings. The accompanying drawings are not necessarily drawn to scale. The described examples are provided for illustrative purposes and are not intended to limit the scope of the invention. It should be understood, however, that persons having ordinary skill in the art may practice the inventive concept without these specific details.
It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first attachment could be termed a second attachment, and, similarly, a second attachment could be termed a first attachment, without departing from the scope of the inventive concept.
It will be understood that when an element or layer is referred to as being “on,” “coupled to,” or “connected to” another element or layer, it can be directly on, directly coupled to or directly connected to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being “directly on,” “directly coupled to,” or “directly connected to” another element or layer, there are no intervening elements or layers present. Like numbers refer to like elements throughout. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.
As used in the description of the inventive concept and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates other.
For each frame of single-frame point cloud, each point is rotated by the rotation matrix R, and the translation vector t is added to get the new coordinates of each point in the global coordinate system.
Finally, the single-frame point cloud in the global coordinate system of each frame is spliced together to obtain the joint point cloud in the global coordinate system.
Specifically, the motion model uses a Kalman filtering algorithm, which may include the following sub-steps:
A1: prediction step:
Firstly, a dynamic model of the system is used to predict the state of the next time step, that is, prior estimation. This predicted state will be used as the initial estimate of the next time step. At the same time, the uncertainty of the prior state, namely a covariance matrix, will be predicted.
The specific calculation formula is:
Predicted state estimated value: Xprior=F×Xposterior
Predicted covariance estimated value: Pprior=F×Pposterior×FT+Q
where X is the state estimation, P is the covariance matrix, F is a state transition matrix, and Q is the process noise covariance matrix.
A2: updating step: the prior estimation is modified, and the estimation is updated by using the observation data, so as to obtain the posterior estimation. In this step, it is necessary to calculate the Kalman gain, which is a weight parameter used to adjust the prediction state.
The specific calculation formula is:
Kalman gain: K=Pprior×HT×(H×Ppror×HT+R)−1
Posterior covariance estimated value: Pposterior=(1−K×H)×Pprior
where H is the observation model matrix, Z is the observation value, R is the observation noise covariance matrix, and I is the identity matrix.
These two steps are carried out alternately, that is, the prediction step is followed by the update step, and then the prediction step, and so on, the final effect is the optimal estimation of the real state of the dynamic target.
The object association algorithm may include the following sub-steps:
A visibility method can also be used to remove the dynamic target in the joint point cloud, that is:
The distance from the coordinates of each point in the point cloud to the origin (0,0,0) is calculated and normalized the distance to the range of 0-255 as the depth value of the image.
The joint point cloud and all single-frame point clouds are respectively processed in the above way to obtain the depth map corresponding to the joint point cloud and the sub-depth map corresponding to all single-frame point clouds.
The dynamic targets in the point cloud are determined by the depth map and sub-depth map and removed:
specifically, the depth map has the space occupations of dynamic objects at all moments, and all sub-depth maps corresponding to single-frame point clouds have the space occupation of dynamic objects at every moment. By comparing the depth map with the sub-depth maps corresponding to all time series point clouds, if a position has no space occupation at t0 and has space occupation at t1, it is considered that the object occupying space at this position is a dynamic object, and the joint point cloud with the dynamic objects removed is obtained after removing these dynamic objects.
The ground points that are mistakenly removed in the process of removing dynamic targets are recovered.
Specifically, the ground points of the joint point cloud are obtained by the ground point algorithm, and the ground points are combined with the joint point cloud from which the dynamic targets are removed to obtain the joint point cloud after the ground points are restored.
In an embodiment, the ground point algorithm may include the following sub-steps:
Specifically, a matrix with the shape of (W, L, 3) is created according to the length and width of the overhead image, where W is the width of the overhead image, L is the length of the overhead image, and 3 indicates that the final overhead image is 3 channels.
The indexes of the static joint point cloud to the X axis and Y axis of the overhead image are calculated. Because in the coordinate system of the point cloud, the X-axis is upward and the Y-axis is leftward, while in the overhead image coordinate system, the X-axis is rightward and the Y-axis is downward and the origin of the overhead image is in the upper left corner of the overhead image, it is necessary to reverse the coordinate system to correspond to the transformation from the point cloud to the overhead image. The specific calculation method is as follows:
yi=ymax−yp
xi=xp−xmin
where y is the coordinate on the Y axis of the point that needs to be converted from the static joint point cloud to the overhead image, and ymax is the largest y of the static joint point cloud; x is the coordinate on the X axis of the point that needs to be converted from the static joint point cloud to the overhead image, and xmin is the smallest x of the static joint point cloud.
After determining the indexes of the static joint point cloud to the X-axis and Y-axis of the overhead image, the RGB values are assigned to the corresponding positions. The RGB values are determined according to the intensity information. First, a color domain with a value of 0-1 is created. The higher the value, the closer the color is to red (255, 0, 0), and the lower the value, the closer the color is to blue (0, 0, 255). Then the intensity value is expanded by the intensity enhancement function to expand the intensity range of the static joint point cloud, and the specific calculation method is as follows:
le=gamma×I2
Finally, all intensity values are normalized to 0-1 to be acted on the color domain to confirm the color RGB value of this point. The specific normalization method is as follows:
where, In is the normalized intensity, Imin is the smallest intensity value in the static joint point cloud and Imax is the largest intensity value in the static joint point cloud.
After the color assignment of each position is completed, the matrix is saved as an image to obtain the overhead image of the static joint point cloud. Because the intensity value of each point is enlarged by the intensity enhancement function, the processed static joint point cloud can assign values for colors with lager different, and finally the differences of various regions can be distinguished more clearly the saved image.
Specifically, first, the query center and query radius are set, the point (xp, yp) is selected as the query center, and a radius of r=5 is set as the query range. All points in the query range are traversed, the distance from each point to the central point is calculated, and a distance list consisting of the distance from each point to the central point is obtained:
Corresponding to the embodiment of the pavement element annotation method for point cloud data with fusion of height, the application also provides an embodiment of a device for constructing and pre-annotating point cloud data wit fusion of height.
The present invention also provides another technical solution:
As shown in
Finally, the present invention also provides another technical solution:
The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above embodiments, but also include the technical scheme composed of any combination of the above technical features. It should be pointed out that for those skilled in the art, several improvements and embellishments can be made without departing from the principle of the present invention, and these improvements and embellishments are also regarded as the protection scope of the present invention.
The invention has now been described in detail for the purposes of clarity and understanding. However, those skilled in the art will appreciate that certain changes and modifications may be practiced within the scope of the appended claims.
Conditional language used herein, such as, among others, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain examples include, while other examples do not include, certain features, elements, and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more examples or that one or more examples necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular example.
The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list. The use of “adapted to” or “configured to” herein is meant as open and inclusive language that does not foreclose devices adapted to or configured to perform additional tasks or steps. Additionally, the use of “based on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or values beyond those recited. Similarly, the use of “based at least in part on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based at least in part on” one or more recited conditions or values may, in practice, be based on additional conditions or values beyond those recited. Headings, lists, and numbering included herein are for ease of explanation only and are not meant to be limiting.
The various features and processes described above may be used independently of one another, or may be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states relating thereto can be performed in other sequences that are appropriate. For example, described blocks or states may be performed in an order other than that specifically disclosed, or multiple blocks or states may be combined in a single block or state. The example blocks or states may be performed in serial, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed examples. Similarly, the example systems and components described herein may be configured differently than described. For example, elements may be added to, removed from, or rearranged compared to the disclosed examples.
| Number | Name | Date | Kind |
|---|---|---|---|
| 11069133 | Yan | Jul 2021 | B2 |
| 11580328 | Ho | Feb 2023 | B1 |
| 20200257901 | Walls | Aug 2020 | A1 |
| 20230271607 | Kobashi | Aug 2023 | A1 |
