Patent Application
20240377535
This application is based on, and claims priority from, Korean Patent Application Number 10-2023-0059938, filed May 9, 2023, the disclosure of which is incorporated by reference herein in its entirety.
The present disclosure relates to a method and device for object tracking of a mobile robot using a 2D LiDAR-based marker. More specifically, the present disclosure relates to a technology for a mobile robot to identify a target object and estimate a pose of the target object using a 2D LiDAR-based marker.
The content described below simply provides background information related to the present embodiment and does not constitute the related art.
Mobility as a service (MaaS) is expected to grow in market size in the future, and based on this expectation, a mobile robot will mainly track a person and perform a role of helping or collaborating with the person.
A mobile robot may be used to carry and move objects at various places such as a factory, a distribution center, a museum, a restaurant, a supermarket, an airport, or a smart farm, or to provide information or services to people while moving. In particular, when a person loads an object on the mobile robot and moves it at an airport, supermarket, factory, or the like, the mobile robot may move along with the person. Since the mobile robot automatically recognizes and tracks the person, there is an advantage that the person does not need to directly control a transport device such as a cart to load objects into the transport device and move the transport device.
In order for the mobile robot to perform this role, the mobile robot should be able to recognize and track the person and secure a safe distance. Further, the mobile robot should be able to reliably track the person even in a crowded environment.
Deep learning-based methods (for example, Mask R-CNN, YOLO, or SSD) can be generally used to recognize and/or track an object. However, the deep learning-based methods require high-performance hardware, which increases price of the mobile robot.
In the case of image sensors such as cameras, the accuracy of object recognition in captured images may decrease depending on factors such as changes in lighting, low resolution, and occlusion. A LIDAR (Light Detection And Ranging) sensor is a sensor that measures a distance to a reflection body by shooting a laser pulse and measuring a return time, has high precision, but is relatively expensive.
Therefore, there is a need for technology that can construct a cost-effective and robust system for a mobile robot to recognize and track a target object.
The purpose of the present disclosure is to provide a technology for allowing a mobile robot to recognize and track an object using a marker attached to the object and a 2D LiDAR sensor mounted on the mobile robot.
The purpose of the present disclosure is to provide a technology for detecting an object and estimating a pose of the object using a marker including a high-brightness reflective material in a mobile robot.
The purposes of the present disclosure are not limited to those mentioned above, and other purposes not mentioned herein will be clearly understood by those skilled in the art from the following description.
According to one embodiment, the present disclosure provides a method for estimating a pose of a target object using a 2D LiDAR-based marker in a mobile robot, including acquiring first point cloud data including distance and reflection intensity information using a 2D LiDAR sensor of a mobile robot, extracting second point cloud data corresponding to a high-brightness reflective material included in a marker attached to an object from the first point cloud data, clustering the second point cloud data into at least one cluster, identifying a target cluster corresponding to the target object among the at least one cluster, and estimating a pose of the target object based on change in the target cluster.
According to another embodiment, the present disclosure provides a device for estimating a pose of a target object using a 2D LiDAR-based marker in a mobile robot, including at least one memory storing computer-executable instructions; and at least one processor, wherein the at least one processor is configured to execute the computer-executable instructions to: acquire first point cloud data including distance and reflection intensity information using a 2D LiDAR sensor of a mobile robot; extract second point cloud data corresponding to a high-brightness reflective material included in a marker attached to an object from the first point cloud data; cluster the second point cloud data into at least one cluster; identify a target cluster corresponding to the target object among the at least one cluster; and estimate a pose of the target object based on change in the target cluster.
According to an embodiment of the present disclosure, there is an effect that it is possible to construct a cost-effective system for object recognition and tracking of a mobile robot.
According to an embodiment of the present disclosure, there is an effect that it is possible to improve the accuracy of object recognition and tracking of the mobile robot and reduce computational costs.
The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.
Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, like reference numerals preferably designate like elements, although the elements are shown in different drawings. Further, the following description of some embodiments will omit for the purpose of clarity and for brevity, a detailed description of related known components and functions when considered obscuring the subject of the present disclosure.
Various ordinal numbers or alpha codes such as first, second, i), ii), a), b), etc., are prefixed solely to differentiate one component from the other but not to imply or suggest the substances, order, or sequence of the components. Throughout this specification, when a part “includes” or “comprises” a component, the part is meant to further include other components, to not exclude thereof unless specifically stated to the contrary. The terms such as “unit,” “module,” and the like refer to units in which at least one function or operation is processed and they may be implemented by hardware, software, or a combination thereof.
The description of the present disclosure to be presented below in conjunction with the accompanying drawings is intended to describe exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the technical idea of the present disclosure may be practiced.
The present disclosure provides a cost-effective and robust object tracking technology for estimating a pose of a tracking target object and predicting a behavior of the object based on sensors, that is, using a 2D LiDAR sensor and a marker attached to the tracking target object. Here, the tracking target object may include not only a person but also another mobile robot, but is not limited thereto.
A marker referred to herein is attached to an object and includes a plurality of high-brightness reflective materials having different sizes. The marker may include a plurality of high-brightness reflective materials aligned in a certain direction. The marker may be attached to a specific position (for example, a back or elbow) of the object within a detection range of the 2D LiDAR sensor. To increase the accuracy of marker recognition, it is desirable to attach the marker at a position with the same or similar height as the 2D LiDAR sensor of the mobile robot.
Further, different types of markers may be used for each object in order to identify the object. For example, each marker may have a barcode-like form, but is not limited thereto.
Referring to
The target object pose estimation device 10 shown in
The acquisition unit 110 acquires the point cloud data using the 2D LiDAR sensor of the mobile robot. The point cloud data includes information on distance and reflection intensity. The reflection intensity may refer to the intensity of a reflected signal that is received by the 2D LiDAR sensor. The point cloud data includes 2D coordinate information on detected points.
The preprocessing unit 120 extracts point cloud data corresponding to a high-brightness reflective material included in a marker attached to an object from the point cloud data acquired by the acquisition unit 110. This is intended to use only the point cloud data reflected from the high-brightness reflective material included in the marker attached to the object for identification and pose estimation of the target object.
The preprocessing unit 120 extracts the point cloud data having a value of the reflection intensity equal or greater than a threshold among the acquired point cloud data as the point cloud data corresponding to the high-brightness reflective material included in the marker attached to the object. This is because the point cloud data having the value of the reflection intensity equal or greater than the threshold may be determined to have been reflected from the high-brightness reflective material included in the marker attached to the object.
Referring to
The preprocessing unit 120 may further perform noise removal, downsampling, or the like on the point cloud data.
The clustering unit 130 clusters the preprocessed the point cloud data into at least one cluster. That is, the preprocessed the point cloud data is divided into groups of points corresponding to the marker for each object.
The clustering unit 130 may perform distance-based clustering on the preprocessed the point cloud data. For example, Euclidean distance, Manhattan distance, Minkowski distance, Chevyshev distance, density, or the like may be used as similarity measures, and a K-mean algorithm or the like may be used as a clustering algorithm, but the present disclosure is not limited to thereto.
The identification unit 140 identifies a target cluster corresponding to the target object among at least one cluster. The target object refers to an object that the mobile robot follows, and the target object may be identified by an attached marker. The target cluster refers to a cluster of points that detect the marker attached to the target object.
The identification unit 140 may compare a shape of at least one cluster with a preset shape to identify the target cluster corresponding to the target object. Even when there are a plurality of clusters, markers having different shapes can be used for each object, and thus, a cluster that is most similar to a shape set in the mobile robot in advance among the plurality of clusters can be identified as the target cluster.
The pose estimation unit 150 estimates the pose of the target object based on the target cluster.
A normal vector for a straight line passing through the points of the point cloud data included in the target cluster may be calculated, and the pose of the target object may be estimated based on the calculated normal vector. However, the pose estimation using the normal vector may predict a direction in which a person wants to move, but there is a difficulty in pose estimation.
The pose estimation unit 150 divides the points of the point cloud data included in the target cluster into groups corresponding to a plurality of high-brightness reflective materials having different sizes, and estimates the pose of the target object using change in a ratio between the numbers of points included in the respective groups.
First, change in the points of the point cloud data corresponding to the high-brightness reflective materials included in the marker attached to the object depending on a position and/or direction of the object will be described.
When the object rotates clockwise or counterclockwise and its pose changes, a distance between the 2D LiDAR sensor and the high-brightness reflective material changes, and thus, the number of points of the point cloud data corresponding to the high-brightness reflective materials included in the marker may change. Accordingly, change in the pose of the object can be estimated based on the change in the number of points of the point cloud data corresponding to the high-brightness reflective material included in the marker.
It can be seen from
Referring to
Here, nl is the number of points corresponding to the left high-brightness reflective material, nr is the number of points corresponding to the right high-brightness reflective material, dinit(l) is a distance between an average position of points corresponding to the left and right high-brightness reflective materials and the left high-brightness reflective material, and dinit (r) is a distance between the average positions of the points corresponding to the left and right high-brightness reflective materials and the right high-brightness reflective material.
As long as the 2D LiDAR sensor and the object are located perpendicularly, the ratio nr:nl remains constant regardless of the distance between the object and the 2D LiDAR sensor. That is, as shown in
When the object rotates clockwise or counterclockwise and its pose changes, the ratio nr:nl changes.
Referring to
Referring to
Therefore, when the ratio nr:nl changes, it can be estimated that the pose of the object has changed. A rotation direction and rotation angle of the object can be calculated as shown in Equation 2 and Equation 3.
Here, rot is the rotation direction of the object with a pose in which the object is located perpendicular to the 2D LiDAR sensor as a reference, and dinit (l) and dinit (r) are distances calculated according to Equation 1 when the object is located perpendicular to the 2D LiDAR sensor, and dl and dr are distances calculated according to Equation 1 when the object rotates and is not located perpendicular to the 2D LiDAR sensor.
Here, θ is a rotation angle of the object with a pose in which the object is located perpendicular to the 2D LiDAR sensor as a reference.
As described above, even when the distance between the object and the 2D LiDAR sensor is not constant, the ratio nr:nl is constant, and thus, it is possible to robustly estimate the pose of the object regardless of the distance between the object and the 2D LiDAR sensor.
The pose estimation unit 150 divides the target cluster into a first group and a second group based on position information of the points of the point cloud data included in the target cluster. For example, the points corresponding to the left high-brightness reflective material 21a may be divided into the first group, and the points corresponding to the right high-brightness reflective material 21b may be divided into the second group.
The pose estimation unit 150 obtains a ratio between the number of points of the point cloud data included in the first group and the number of points of the point cloud data included in the second group.
The pose estimation unit 150 estimates the pose of the target object based on the change in the ratio over time. The pose estimation unit 150 may calculate the rotation direction and rotation angle of the target object using Equations 1 to 3 described above to estimate the pose.
The pose of the target object estimated by the pose estimation unit 150 may be used to predict and estimate a next state of the target object using a Kalman filter.
A control unit (not shown) controls the behavior of the mobile robot based on the estimated pose of the target object. The control unit may predict a behavior of the target object based on the estimated pose and control the behavior of the mobile robot to track the target object.
Referring to
The target object pose estimation device 10 extracts second point cloud data corresponding to the high-brightness reflective material included in the marker attached to the object from the first point cloud data (S720). The target object pose estimation device 10 may extract, as the second point cloud data, the point cloud data having a value of the reflection intensity equal or greater than the threshold from the first point cloud data.
The target object pose estimation device 10 clusters the second point cloud data into at least one cluster (S730). Distance-based clustering may be performed on the second point cloud data.
The target object pose estimation device 10 identifies the target cluster corresponding to the target object among at least one cluster (S740). The target cluster corresponding to the target object may be identified by comparing a shape of at least one cluster with a preset shape.
The target object pose estimation device 10 estimates the pose of the target object based on the change in the target cluster (S750). The points of the point cloud data included in the target cluster are divided into groups corresponding to the plurality of high-brightness reflective materials having different sizes, and the pose of the target object may be estimated using the change in the ratio between the number of points included in the respective groups.
The target object pose estimation device 10 can control the behavior of the mobile robot based on the estimated pose of the target object (S760).
At least some of the components described in the exemplary embodiments of the present disclosure may be implemented as a hardware element including at least one or a combination of a digital signal processor (DSP), a processor, a controller, an application-specific IC (ASIC), a programmable logic device (FPGA, or the like), and other electronic devices. Further, at least some of functions or processes described in the exemplary embodiments may be implemented as software, and the software may be stored in a recording medium. At least some of the components, functions, and processes described in the exemplary embodiments of the present disclosure may be implemented through a combination of hardware and software.
The method according to exemplary embodiments of the present disclosure can be written as a program that can be executed on a computer, and can also be implemented in various recording media such as magnetic storage media, optically readable media, and digital storage media.
Implementations of the various techniques described herein may be realized by digital electronic circuitry, or by computer hardware, firmware, software, or combinations thereof. Implementations may be made as a computer program tangibly embodied in a computer program product, i.e., an information carrier, e.g., machine-readable storage device (computer-readable medium) or a radio signal, for processing by, or controlling the operation of a data processing device, e.g., a programmable processor, a computer, or multiple computers. Computer programs, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form as a stand-alone program or as a module, component, subroutine, or other units suitable for use in a computing environment. The computer program may be processed on one computer or multiple computers at one site or distributed across multiple sites and developed to be interconnected through a communications network.
Processors suitable for processing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. Elements of the computer may include at least one processor that executes instructions and one or more memory devices that store instructions and data. In general, the computer may include one or more mass storage devices that store data, such as magnetic disks, magneto-optical disks, or optical disks, or may be coupled to the mass storage devices to receive data therefrom and/or transmit data thereto. Information carriers suitable for embodying computer program instructions and data include, for example, semiconductor memory devices, magnetic mediums such as hard disks, floppy disks, and magnetic tapes, optical mediums such as CD-ROM (Compact Disk Read Only Memory), DVD (Digital Video Disk), magneto-optical mediums such as floptical disk, ROM (Read Only Memory), RAM (Random Access Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically Erasable Programmable ROM). The processor and memory may be supplemented by or included in special purpose logic circuitry.
The processor may execute an operating system and software applications executed on the operating system. In addition, the processor device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the processor device may be described as being used as a single processor device, but those skilled in the art will understand that the processor device may include a plurality of processing elements and/or a plurality of types of processing elements. For example, a processor device may include a plurality of processors or one processor, and one controller. Further, other processing configurations, such as parallel processors, are also possible.
In addition, a non-transitory computer-readable medium may be any available medium that can be accessed by a computer and may include both a computer storage medium and a transmission medium.
The present specification includes details of a number of specific implements, but it should be understood that the details do not limit any invention or what is claimable in the specification but rather describe features of the specific example embodiment. Features described in the specification in the context of individual example embodiments may be implemented as a combination in a single example embodiment. In contrast, various features described in the specification in the context of a single example embodiment may be implemented in multiple example embodiments individually or in an appropriate sub-combination. Furthermore, the features may operate in a specific combination and may be initially described as claimed in the combination, but one or more features may be excluded from the claimed combination in some cases, and the claimed combination may be changed into a sub-combination or a modification of a sub-combination.
Similarly, even though operations are described in a specific order on the drawings, it should not be understood as the operations needing to be performed in the specific order or in sequence to obtain desired results or as all the operations needing to be performed. In a specific case, multitasking and parallel processing may be advantageous. In addition, it should not be understood as requiring a separation of various apparatus components in the above described example embodiments in all example embodiments, and it should be understood that the above-described program components and apparatuses may be incorporated into a single software product or may be packaged in multiple software products.
It should be understood that the example embodiments disclosed herein are merely illustrative and are not intended to limit the scope of the invention. It will be apparent to one of ordinary skill in the art that various modifications of the example embodiments may be made without departing from the spirit and scope of the claims and their equivalents.
Accordingly, one of ordinary skill would understand that the scope of the claimed invention is not to be limited by the above explicitly described embodiments but by the claims and equivalents thereof.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2023-0059938 | May 2023 | KR | national |
