Patent Application
20250209839
The present application claims the benefit of the earlier filing date of Korean provisional patent application No. 10-2023-0189016, filed on Dec. 21, 2023, the entire contents of which are incorporated herein by reference.
The present disclosure relates to a method and a computing device for annotation based on a LiDAR map.
Annotation is a process of labelling data to generate training data such that an artificial intelligence may be trained by using the training data.
Generally, the annotation should be done per frame. Therefore, repetitive annotating processes for a same target may decrease a speed thereof and a consistency of the annotation during the repetitive annotating processes may be difficult to be achieved, which result in a problem of a difficulty in achieving an efficiency of the annotation.
Further, there is another problem in that an annotation result may be reset on a next frame depending on an annotation tool. An example of the another problem is provided in the link below:
Such annotation lacking the consistency may affect training processes, and therefore, a method of achieving the consistency and fast speed of the repetitive annotating processes is required to increase an amount of the training data.
It is an object of the present disclosure to solve all the aforementioned problems.
It is still another object of the present disclosure to selectively choose key frames among all frames having been used to generate a LiDAR map and to perform annotation on the key frames, without annotating all the frames.
It is still another object of the present disclosure to simultaneously display both each of partial LiDAR maps and each of images corresponding to each of the key frames, thereby allowing annotating processes to be performed while referring to both the partial LiDAR maps and their corresponding images.
It is still yet another object of the present disclosure to cumulatively record each of annotation data for each of objects, thereby preventing repetitive annotating processes for each same object.
It is still yet another object of the present disclosure to generate an annotated LiDAR map where each of the objects included in the LiDAR map is annotated, thereby managing all the annotation data in the annotated LiDAR map.
In accordance with one aspect of the present disclosure, there is provided a method for annotation based on a LiDAR map, including steps of: (a) generating, by a computing device, a LiDAR map and a-key frame trajectory using LiDAR point cloud data and a LiDAR SLAM algorithm; (b) allowing, by the computing device, each of key frames included in the key frame trajectory to be annotated, thereby allowing annotation result data to be generated; and (c) cumulatively recording, by the computing device, the annotation result data in the LiDAR map.
As one example, the step of (b) includes steps of: (b1) the computing device selecting a t-th key frame among the plurality of key frames that are part of the key frame trajectory; (b2) the computing device allowing a t-th partial LiDAR map and one or more t-th image data corresponding to location information of the t-th key frame to be displayed; and (b3) the computing device allowing a t-th annotating process to be performed by referring to the t-th partial LiDAR map and the t-th image data; and wherein the steps of (b1) to (b3) are repeated while increasing t from 1 to n, wherein t is an integer, and n is a cardinal number of the key frames.
As one example, at the step of (c), the computing device converts coordinates of t-th annotation result data into coordinates of the t-th partial LiDAR map, to thereby generate and record converted t-th annotation result data.
In accordance with another aspect of the present disclosure, there is provided a computing device for annotation based on a LiDAR map, including: at least one memory that stores instructions; and at least one processor configured to execute the instructions to perform processes of (I) generating a LiDAR map and a key frame trajectory using LiDAR point cloud data and a LiDAR SLAM algorithm; (II) allowing each of key frames included in the key frame trajectory to be annotated, thereby allowing annotation result data to be generated; and (III) cumulatively recording the annotation result data in the LiDAR map.
As one example, the process of (II) includes processes of: (II-1) selecting a t-th key frame among the plurality of key frames that are part of the key frame trajectory; (II-2) allowing a t-th partial LiDAR map and one or more t-th image data corresponding to location information of the t-th key frame to be displayed; and (II-3) allowing a t-th annotating process to be performed by referring to the t-th partial LiDAR map and the t-th image data; and wherein the processes of (II-1) to (II-3) are repeated while increasing t from 1 to n, wherein t is an integer, and n is a cardinal number of the key frames.
As one example, at the process of (III), the processor converts coordinates of t-th annotation result data into coordinates of the t-th partial LiDAR map, to thereby generate and record converted t-th annotation result data.
In addition, recordable media that are readable by a computer for storing a computer program to execute the method of the present disclosure is further provided.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
The above and other objects and features of the present disclosure will become apparent from the following description of preferred embodiments given in conjunction with the accompanying drawings.
The following drawings to be used to explain example embodiments of the present disclosure are only part of example embodiments of the present disclosure and other drawings can be obtained based on the drawings by those skilled in the art of the present disclosure without inventive work.
Detailed explanation on the present disclosure to be made below refer to attached drawings and diagrams illustrated as specific embodiment examples under which the present disclosure may be implemented to make clear of purposes, technical solutions, and advantages of the present disclosure. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure.
Besides, in the detailed description and claims of the present disclosure, a term “include” and its variations are not intended to exclude other technical features, additions, components or steps. Other objects, benefits, and features of the present disclosure will be revealed to one skilled in the art, partially from the specification and partially from the implementation of the present disclosure. The following examples and drawings will be provided as examples but they are not intended to limit the present disclosure.
Moreover, the present disclosure covers all possible combinations of example embodiments indicated in this specification. It is to be understood that the various embodiments of the present disclosure, although different, are not necessarily mutually exclusive. For example, a particular feature, structure, or characteristic described herein in connection with one embodiment may be implemented within other embodiments without departing from the spirit and scope of the present disclosure. In addition, it is to be understood that the position or arrangement of individual elements within each disclosed embodiment may be modified without departing from the spirit and scope of the present disclosure. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined only by the appended claims, appropriately interpreted, along with the full range of equivalents to which the claims are entitled. In the drawings, like numerals refer to the same or similar functionality throughout the several views.
To allow those skilled in the art to the present disclosure to be carried out easily, the example embodiments of the present disclosure by referring to attached drawings will be explained in detail as shown below.
By referring to
Specifically, the computing device 100 may typically achieve a desired system performance by using combinations of at least one computing device and at least one computer software, e.g., a computer processor, a memory, a storage, an input device, an output device, or any other conventional computing components, an electronic communication device such as a router or a switch, an electronic information storage system such as a network-attached storage (NAS) device and a storage area network (SAN) as the computing device and any instructions that allow the computing device to function in a specific way as the computer software.
Also, the processors of such devices may include hardware configuration of MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, data bus, etc. Additionally, the computing device may further include operating system (OS) and software configuration of applications that achieve specific purposes.
Such description of the computing device does not exclude an integrated device including any combination of a processor, a memory, a medium, or any other computing components for implementing the present disclosure.
Moreover, the computing device 100 may communicate with a database 900 which stores information required for performing processes of the annotation based on the LiDAR map. Herein, the database 900 may include at least part of a flash memory type, a hard disk type, a multimedia card micro type, a card memory (such as SD or XD memory), RAM (Random Access Memory), SRAM (Statis Random Access Memory), ROM (ReadOnly Memory), EEPROM (Electrically Erasable Programmable ReadOnly Memory), PROM (Programmable ReadOnly Memory), magnetic memory, magnetic disk memory, and optical disk. Further, depending on an operating condition of the present disclosure, the database 900 may be installed separately from the computing device 100, or may be installed within the computing device 100 to transmit and record data, and may also be implemented separately into two or more DBs, contrary to the illustration.
A method using the computing device 100 for the annotation based on the LiDAR map in accordance with one example embodiment of the present disclosure is as follows.
First, the computing device 100 may refer to dataset recorded in the database to thereby acquire LiDAR point cloud data.
Herein, the dataset may include LiDAR point cloud data collected every predetermined period by a LiDAR mounted on a vehicle driven along a predetermined path, and image data collected every predetermined period by one or more cameras installed on the vehicle.
Since the LiDAR point cloud data are 3D (3-Dimensional) data, and the image data are 2D (2-Dimensional) data, the LiDAR and the cameras may be calibrated with each other in order to match each other's coordinates.
Further, the computing device 100 may generate the LiDAR map and the key frame trajectory by using the LiDAR point cloud data through LiDAR SLAM (Simultaneous Localization and Mapping) algorithm.
Herein, the LiDAR SLAM algorithm may include ICP (Interactive Closet Point) algorithm, and HDL GRAPH SLAM algorithm, but it is not limited thereto.
Herein, the computing device 100 may select some part of frames as key frames from all LiDAR frames such that the key frame trajectory is generated by using the selected key frames.
Herein, the key frame trajectory may be a trajectory including each of sequential locations corresponding to each of the key frames.
It is to be appreciated that each of the LiDAR frames may correspond to each location information on the LiDAR map, which will be explained later.
Herein, the location information may include information on a point and an orientation of the vehicle on the LiDAR map, or may be a point and an orientation of the LiDAR on the LiDAR map, but it is not limited thereto.
Further, the location information may include information on 6-DOF (i.e., Six degrees of freedom) which are information on X, Y, Z, Roll, Pitch and Yaw.
Furthermore, the computing device 100 may select the key frames by using at least part of traveled distances and rotation angles, but it is not limited thereto. Herein, each of the traveled distances and each of the rotation angles may be determined by referring to each location information of each of LiDAR frames.
Meanwhile, the computing device 100 may generate the LiDAR map by additionally using at least part of GPS sensor data and IMU sensor data acquired from a GPS sensor and an IMU sensor installed on the vehicle, to thereby increase a quality of the LiDAR map.
Also, if the LiDAR map is generated by using the GPS sensor data, even in case at least one of dates/times of acquiring the LiDAR data and the image data and surroundings of the LiDAR data and the image data is changed, the LiDAR data and the image data having already been acquired can be re-used.
By referring to
Herein, t is an integer, and n is the number of the key frames.
Further, the computing device 100 may allow a t-th partial LiDAR map and its corresponding one or more t-th image data corresponding to the t-th key frame to be displayed.
Next, the computing device 100 may allow a t-th annotating process to be performed on at least one object included in the t-th partial LiDAR map by referring to the displayed t-th partial LiDAR map and the t-th image data.
Herein, if the object is a vehicle with a 3D volume, then a 3D bounding box with a shape of a cuboid capable of including the vehicle therein may be generated to annotate the vehicle, and if the object is a parking slot with a 2D region, then a 2D bounding box capable of including the parking slot therein may be generated to annotate the parking slot, but it is not limited thereto. It is to be appreciated that a shape of the bounding box of the present disclosure may include any quadrangle shape such as variations of a rectangle.
Further, the computer device 100 may allow intermediate data and/or result data of the t-th annotating process performed on the object(s) included in the t-th partial LiDAR map to be projected onto its corresponding t-th image data.
Herein, “projected” may mean that the intermediate data and/or the result data of the t-th annotating process are allowed to be displayed at specific coordinates of the t-th image data corresponding to specific coordinates of the t-th partial LiDAR map. The specific coordinates of the t-th partial LiDAR map represents a location of the object on the LiDAR coordinate system.
Next, the computing device 100 may convert coordinates of the t-th annotation result data into coordinates of the t-th partial LiDAR map, to thereby generate and record converted t-th annotation result data.
After that, the computing device 100 may allow a (t+1)-th key frame to be selected among all the key frames included in the key frame trajectory.
Further, the computing device 100 may allow a (t+1)-th partial LiDAR map and its corresponding one or more (t+1)-th image data corresponding to location information of a (t+1)-th key frame to be displayed (through a labelling worker's terminal).
Herein, the computing device 100 may convert the t-th annotation result data to be correctly located on the location information of the (t+1)-th key frame, and may allow the converted t-th annotation result data to be displayed on the (t+1)-th partial LiDAR map and the (t+1)-th image data.
Next, the computing device 100 may allow a (t+1)-th annotating process to be performed on at least one object that did not exist on the t-th partial LiDAR map among all objects included in the (t+1)-th partial LiDAR map by referring to at least part of the (t+1)-th partial LiDAR map and the (t+1)-th image data.
As such, the computing device 100 may allow a first annotating process to an n-th annotating process to be performed on a first key frame to an n-th key frame, and may cumulatively record a first annotation result to the n-th annotation result on the LiDAR map, thereby allowing management of all the annotated objects on the single LiDAR map.
For example, by referring to
As another example, by referring to
The present disclosure has an effect of selectively choosing key frames among all the frames having been used to generate the LiDAR map and performing annotation on the key frames, without annotating all the frames.
The present disclosure has another effect of simultaneously displaying both each of the partial LiDAR maps and each of the image data corresponding to each of the key frames, thereby allowing each of annotating processes to be performed while referring to both each of the partial LiDAR maps and its corresponding image data.
The present disclosure has still another effect of cumulatively recording each of annotation data for each of the objects, thereby preventing repetitive annotating processes for each same object.
The present disclosure has still yet another effect of generating an annotated LiDAR map where each of the objects included in the LiDAR map is annotated, thereby managing all the annotation data in the annotated LiDAR map.
The embodiments of the present disclosure as explained above can be implemented in a form of executable program command through a variety of computer means recordable to computer readable media. The computer readable media may include solely or in combination, program commands, data files, and data structures. The program commands recorded to the media may be components specially designed for the present disclosure or may be usable to a skilled human in a field of computer software. Computer readable media include magnetic media such as hard disk, floppy disk, and magnetic tape, optical media such as CD-ROM and DVD, magneto-optical media such as floptical disk and hardware devices such as ROM, RAM, and flash memory specially designed to store and carry out program commands. Program commands include not only a machine language code made by a compiler but also a high level language code that can be executed by a computer using an interpreter, etc. The hardware device can work as more than a software module to perform the process in accordance with the present disclosure and they can do the same in the opposite case.
As seen above, the present disclosure has been explained by specific matters such as detailed components, limited embodiments, and drawings. They have been provided only to help more general understanding of the present disclosure. It, however, will be understood by those skilled in the art that various changes and modification may be made from the description without departing from the spirit and scope of the disclosure as defined in the following claims.
Accordingly, the thought of the present disclosure must not be confined to the explained embodiments, and the following patent claims as well as everything including variations equal or equivalent to the patent claims pertain to the category of the thought of the present disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2023-0189016 | Dec 2023 | KR | national |
