The present disclosure relates to a robot that generates a map and configures a correlation of nodes, based on multi sensors and artificial intelligence, and that moves based on the map, and a method of generating a map.
A large-scale retail store, a department store, an airport, a golf course, and the like are places where exchange of goods and services takes place between people. Robots may be useful in the places to offer information or convenience to people.
Robots may be classified as guide robots, security robots, cleaning robots and the like. The robots move in a space, confirming their positions.
The robots are required for holding information on a space, on their current positions, on a path previously moved by the robots and the like such that the robots move confirming their positions and avoiding obstacles.
The robots may store maps to confirm a space and to move in the space. To generate a map, the robots may draw up a map using a variety of sensors, and may match and store various pieces of information in the map.
However, errors may be produced due to features of each sensor and may be produced during the process in which a robot moves. Accordingly, technologies for generating and correcting maps are required considering the errors.
As a means to solve the above-described problems, the present disclosure is to allow a robot to generate a map for a space using various sensors, and to store information based on the sensors in the map.
Additionally, the present disclosure is to implement fusion SLAM in which the position of a robot is identified using various sensors in a space.
Further, the present disclosure is to allow a robot to set a correlation between nodes in a map and to enhance accuracy of fusion SLAM.
Objectives of the present disclosure are not limited to what has been described. Additionally, other objectives and advantages that have not been mentioned may be clearly understood from the following description and may be more clearly understood from embodiments. Further, it will be understood that the objectives and advantages of the present disclosure may be realized via means and a combination thereof that are described in the appended claims.
A robot generating a map on the basis of multiple sensors and artificial intelligence and configuring a correlation of nodes according to an embodiment includes a LiDAR sensor sensing a distance between an object outside the robot and the robot and generating a LiDAR frame, and a camera sensor photographing an object outside the robot and generating a visual frame.
A robot generating a map on the basis of multiple sensors and artificial intelligence and configuring a correlation of nodes according to an embodiment includes a controller that generates a pose graph comprised of LiDAR branch, visual branch, and backbone, and the LiDAR branch includes one or more of the LiDAR frames, and the visual branch includes one or more of the visual frames, and the backbone includes two or more frame nodes registered with any one or more of the LiDAR frames or the visual frames, and the controller generates a correlation between nodes in the pose graph.
A robot generating a map on the basis of multiple sensors and artificial intelligence and configuring a correlation of nodes according to an embodiment includes a map storage that stores the LiDAR branch, the visual branch, the backbone and the correlation.
A robot generating a map on the basis of multiple sensors and artificial intelligence and configuring a correlation of nodes according to an embodiment selects two adjacent nodes among nodes, sets wheel odometry information, which is generated by a wheel encoder when the robot moves between the two nodes, as a correlation between the two nodes, and stores the correlation in the map storage.
A robot generating a map on the basis of multiple sensors and artificial intelligence and configuring a correlation of nodes according to an embodiment generates LiDAR odometry information using two LiDAR frames registered between two nodes, or generates visual odometry information using two visual frames registered between two nodes.
A robot generating a map on the basis of multiple sensors and artificial intelligence and configuring a correlation of nodes according to an embodiment selects nodes, which are not adjacent and in which a visual frame is registered, among nodes, generates an essential graph, and generates a correlation between nodes in which a LiDAR frame is registered, in the generated essential graph.
A robot that moves using a map in which a correlation of nodes is stored on the basis of multiple sensors and artificial intelligence according to an embodiment includes a wheel encoder generating wheel odometry information while the robot is moving, a LiDAR sensor sensing a distance between an object outside the robot and the robot and generating a first LiDAR frame, and a camera sensor photographing an object outside the robot and generating a first visual frame, and a controller calculating the current position of the robot by comparing the wheel odometry information and the first LiDAR frame and the first visual frame and the correlation of nodes stored in the map storage.
The present disclosure according to embodiments allows a robot to generate a map for a space using various sensors and to configure a correlation between each of the nodes in the map.
The present disclosure according to embodiments allows a robot to perform fusion SLAM in a space using various sensors.
The present disclosure according to embodiments allows a robot to use a correlation while the robot moves on the basis of a map comprised of various sensors and to accurately calculate the current position of the robot.
Effects of the present disclosure are not limited to the above-described ones, and one having ordinary skill in the art to which the disclosure pertains may easily draw various effects from the configuration of the disclosure.
Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art to which the present disclosure pertains can easily implement the present disclosure. The present disclosure may be implemented in many different manners and is not limited to the embodiments described herein.
In order to clearly illustrate the present disclosure, technical explanation that is not directly related to the present disclosure may be omitted, and same or similar components are denoted by a same reference numeral throughout the specification. Further, some embodiments of the present disclosure will be described in detail with reference to the drawings. In adding reference numerals to components of each drawing, the same components may have the same reference numeral as possible even if they are displayed on different drawings. Further, in describing the present disclosure, a detailed description of related known configurations and functions will be omitted when it is determined that it may obscure the gist of the present disclosure.
In describing components of the present disclosure, it is possible to use the terms such as first, second, A, B, (a), and (b), etc. These terms are only intended to distinguish a component from another component, and a nature, an order, a sequence, or the number of the corresponding components is not limited by that term. When a component is described as being “connected,” “coupled” or “connected” to another component, the component may be directly connected or able to be connected to the other component; however, it is also to be understood that an additional component may be “interposed” between the two components, or the two components may be “connected,” “coupled” or “connected” through an additional component.
Further, with respect to embodiments of the present disclosure, for convenience of explanation, the present disclosure may be described by subdividing an individual component, but the components of the present disclosure may be implemented within a device or a module, or a component of the present disclosure may be implemented by being divided into a plurality of devices or modules.
In this specification, a robot includes devices that are used for specific purposes (cleaning, ensuring security, monitoring, guiding and the like) or that moves offering functions according to features of a space in which the robot is moving, hereunder. Accordingly, in this specification, devices that have transportation means capable of moving using predetermined information and sensors, and that offer predetermined functions are generally referred to as a robot.
In this specification, a robot may move with storing a map. The map denotes information on fixed objects such as fixed walls, fixed stairs and the like that do not move in a space. Additionally, information on movable obstacles that are disposed periodically, i.e., information on dynamic objects may be stored on the map.
As an example, information on obstacles disposed within a certain range with respect to a direction in which the robot moves forward may also be stored in the map. In this case, unlike the map in which the above-described fixed objects are stored, the map includes information on obstacles, which is registered temporarily, and then removes the information after the robot moves.
Further, in this specification, the robot may confirm an external dynamic object using various sensors. When the robot moves to a destination in an environment that is crowded with a large number of pedestrians after confirming the external dynamic object, the robot may confirm a state in which waypoints to the destination are occupied by obstacles.
Furthermore, the robot may determine the robot arrives at a waypoint on the basis of a degree in a change of directions of the waypoint, and the robot moves to the destination for next waypoint successfully.
A main body 10 may be configured to be long in the up-down direction, and may have the shape of a roly poly toy that gradually becomes slimmer from the lower portion toward the upper portion, as a whole.
The main body 10 may include a case 30 that forms the appearance of the robot 1. The case 30 may include a top cover 31 disposed on the upper side, a first middle cover 32 disposed on the lower side of the top cover 31, a second middle cover 33 disposed on the lower side of the first middle cover 32, and a bottom cover 34 disposed on the lower side of the second middle cover 33. The first middle cover 32 and the second middle cover 33 may constitute a single middle cover.
The top cover 31 may be disposed at the uppermost end of the robot 1, and may have the shape of a hemisphere or a dome. The top cover 31 may be disposed at a height below the average height for adults to readily receive an instruction from a user. Additionally, the top cover 31 may be configured to rotate at a predetermined angle.
The robot 1 may further include a control module 150 therein. The control module 150 controls the robot 1 like a type of computer or a type of processor. Accordingly, the control module 150 may be disposed in the robot 1, may perform functions similar to those of a main processor, and may interact with a user.
The control module 150 is disposed in the robot 1 to control the robot during robot's movement and by sensing objects around the robot. The control module 150 of the robot may be implemented as a software module, a chip in which a software module is implemented as hardware, and the like.
A display unit 31a that receives an instruction from a user or that outputs information, and sensors, for example, a camera 31b and a microphone 31c may be disposed on one side of the front surface of the top cover 31.
In addition to the display unit 31a of the top cover 31, a display unit 20 is also disposed on one side of the middle cover 32.
Information may be output by all the two display units 31a, 20 or may be output by any one of the two display units 31a, 20 according to functions of the robot.
Additionally, various obstacle sensors (220 in
Additionally, the robot in
The shape of the robot in
The robot in
In a state in which a plurality of the robots in
A LiDAR sensor 220 may sense surrounding objects two-dimensionally or three-dimensionally. A two-dimensional LiDAR sensor may sense positions of objects within 360-degree ranges with respect to the robot. LiDAR information sensed in a specific position may constitute a single LiDAR frame. That is, the LiDAR sensor 220 senses a distance between an object disposed outside the robot and the robot to generate a LiDAR frame.
As an example, a camera sensor 230 is a regular camera. To overcome viewing angle limitations, two or more camera sensors 230 may be used. An image captured in a specific position constitutes vision information. That is, the camera sensor 230 photographs an object outside the robot and generates a visual frame including vision information.
The robot 1, to which the present disclosure is applied, performs fusion-simultaneous localization and mapping (Fusion-SLAM) using the LiDAR sensor 220 and the camera sensor 230.
In fusion SLAM, LiDAR information and vision information may be combinedly used. The LiDAR information and vision information may be configured as maps.
Unlike a robot that uses a single sensor (LiDAR-only SLAM, visual-only SLAM), a robot that uses fusion-SLAM may enhance accuracy of estimating a position. That is, when fusion SLAM is performed by combining the LiDAR information and vision information, map quality may be enhanced.
The map quality is a criterion applied to both of the vision map comprised of pieces of vision information, and the LiDAR map comprised of pieces of LiDAR information. At the time of fusion SLAM, map quality of each of the vision map and LiDAR map is enhanced because sensors may share information that is not sufficiently acquired by each of the sensors.
Additionally, LiDAR information or vision information may be extracted from a single map and may be used. For example, LiDAR information or vision information, or all the LiDAR information and vision information may be used for localization of the robot in accordance with an amount of memory held by the robot or a calculation capability of a calculation processor, and the like.
An interface unit 290 receives information input by a user. The interface unit 290 receives various pieces of information such as a touch, a voice and the like input by the user, and outputs results of the input. Additionally, the interface unit 290 may output a map stored by the robot 1 or may output a course in which the robot moves by overlapping on the map.
Further, the interface unit 290 may supply predetermined information to a user.
A controller 250 generates a map as in
A communication unit 280 may allow the robot 1 to communicate with another robot or an external server and to receive and transmit information.
The robot 1 may generate each map using each of the sensors (a LiDAR sensor and a camera sensor), or may generate a single map using each of the sensors and then may generate another map in which details corresponding to a specific sensor are only extracted from the single map.
Additionally, the map of the present disclosure may include odometry information on the basis of rotations of wheels. The odometry information is information on distances moved by the robot, which are calculated using frequencies of rotations of a wheel of the robot, a difference in frequencies of rotations of both wheels of the robot, and the like. A wheel encoder 260 in
The robot may calculate a distance moved by the robot on the basis of the odometry information as well as the information generated using the sensors.
The controller 250 in
A plurality of LiDAR sensors 220 and camera sensors 230 may be disposed outside of the robot 1 to identify external objects.
In addition to the LiDAR sensor 220 and camera sensor 230 in
The artificial intelligence unit 255 may input information that is processed by the LiDAR sensor 220, the camera sensor 230 and the other sensors, or information that is accumulated and stored while the robot 1 is moving, and the like, and may output results required for the controller 250 to determine an external situation, to process information and to generate a moving path.
As an example, the robot 1 may store information on positions of various objects, disposed in a space in which the robot is moving, as a map. The objects include a fixed object such as a wall, a door and the like, and a movable object such as a flower pot, a desk and the like. The artificial intelligence unit 255 may output data on a path taken by the robot, a range of work covered by the robot, and the like, using map information and information supplied by the LiDAR sensor 220, the camera sensor 230 and the other sensors.
Additionally, the artificial intelligence unit 255 may recognize objects disposed around the robot using information supplied by the LiDAR sensor 220, the camera sensor 230 and the other sensors. The artificial intelligence unit 255 may output meta information on an image by receiving the image. The meta information includes information on the name of an object in an image, a distance between an object and the robot, the sort of an object, whether an object is disposed on a map, and the like.
Information supplied by the LiDAR sensor 220, the camera sensor 230 and the other sensors is input to an input node of a deep learning network of the artificial intelligence unit 255, and then results are output from an output node of the artificial intelligence unit 255 through information processing of a hidden layer of the deep learning network of the artificial intelligence unit 255.
The controller 250 may calculate a moving path of the robot using date calculated by the artificial intelligence unit 255 or using data processed by various sensors.
Additionally, the robot may store information sensed by the camera sensor in a specific spot, in the map storage 210 using the camera sensor 230 while the robot is moving in the space 40.
The backbone is information on a trajectory of the robot. Additionally, the backbone includes one or more frame nodes corresponding to the trajectory. The frame nodes further include constraint information in a relation between the frame nodes and other frame nodes. An edge between nodes denotes constraint information. The edge denotes odometry constraint information (odometry constraint) or loop constraint information (loop constraint).
The LiDAR branch of the second layer is comprised of LiDAR frames. The LiDAR frames include a LiDAR sensing value that is sensed while the robot is moving. At least one or more of the LiDAR frames are set as a LiDAR keyframe.
The LiDAR keyframe has a corresponding relation with the nodes of the backbone. In
The visual branch of the second layer is comprised of visual keyframes. The visual keyframes indicate one or more visual feature nodes that are camera sensing values (i.e., an image captured by the camera) sensed while the robot is moving. The robot may generate a plurality of visual feature nodes on the basis of the number of camera sensors disposed in the robot.
In the map structure of
Poses of the robot at the LiDAR or the visual keyframe are same, and the LiDAR or the visual keyframe is connected with each frame node. An extrinsic parameter may be added for each keyframe on the basis of a position of the robot, to which the LiDAR sensor or the camera sensor is attached. The extrinsic parameter denotes information on a relative position at which a sensor is attached from the center of the robot.
The visual keyframe has a corresponding relation with the node of the backbone. In
Edges are displayed between nodes v1 to v5 constituting the backbone of the first layer. e12, e23, e34, and e45 are edges between adjacent nodes, and e13, e35, and e25 are edges between non-adjacent nodes.
Odometry constraint information, or for short, odometry information denotes constraints between adjacent frame nodes such as e12, e23, e34, and e45. Loop constraint information, or for short, loop information denotes constraints between non-adjacent frames such as e13, e25, and e35.
The backbone is comprised of a plurality of keyframes. The controller 250 may perform an initial mapping process to add the plurality of keyframes to the backbone. The initial mapping process includes adding the LiDAR keyframe and the visual frame based on the keyframe.
The structure of
Additionally, the backbone includes two or more frame nodes in which any one or more of a LiDAR frame or a visual frame are registered. In this case, the LiDAR frame or the visual frame registered in the frame node is referred to as a keyframe. A pose graph includes the LiDAR branch, the visual branch and the backbone.
Further, the pose graph includes odometry information, loop information and the like among frame nodes. The odometry information includes information on rotations, directions, and the like of wheels, which is generated while the robot is moving between frames nodes. The loop information is based on a set of frame nodes connected using specific constraints between visual keyframes around a specific frame node within a maximum sensing distance of the LiDAR sensor 220.
The controller 250 generates the pose graph in
Below, frame nodes are referred to as nodes for short. Additionally, the LiDAR information or the visual information registered in the frame node are respectively indicated as a LiDAR keyframe and a visual keyframe. Further, the LiDAR information may all be registered in the frame node. Accordingly, the LiDAR information referred based on the frame node is indicated as a LiDAR keyframe, a LiDAR frame, LiDAR information and the like. The visual information registered in the frame node is also indicated as a visual keyframe, a visual frame, visual information and the like.
As illustrated in
Additionally, the correlation denotes conditions that have to be satisfied by the robot 1 when the robot moves between nodes, as a sort of constraint. For example, a LiDAR frame confirmed in a first node, and a LiDAR frame confirmed in a second node may be constraints that are required for confirming a position when the robot moves between the first node and the second node.
The correlation between adjacent nodes includes information required for confirming the current position of the robot when the robot moves between two adjacent nodes. The two adjacent nodes denote that another node is not disposed on a path that connects between two nodes. A correlation between non-adjacent nodes includes information required for confirming the current position of the robot when the robot moves between two non-adjacent nodes. The non-adjacent node denotes that another node is disposed on a path between two nodes or near the path.
As an example, the adjacent node indicates two nodes in which wheel odometry information is registered, and the non-adjacent node indicates two nodes in which wheel odometry information is not registered. As another example, the non-adjacent node denotes two nodes that include two or more pieces of wheel odometry information because a waypoint node is disposed between the two nodes. For example, in
Accordingly, the controller 250 of the robot may select a correlation between nodes on the basis of information generated by different types of sensors, and may store the correlation in a map such that the robot moves on the basis of fusion SLAM.
For example, when selecting a correlation between adjacent nodes in fusion SLAM and storing the correlation in a map, the robot may exactly confirm the current position of the robot while moving between the adjacent nodes. When selecting a correlation between non-adjacent nodes in fusion SLAM and storing the correlation in a map, the robot may exactly confirm the current position of the robot while moving between the non-adjacent nodes.
The robot, as described above, performs fusion SLAM on the basis of two or more types of sensors. The robot may generate the map as in
Additionally, one or two or more camera sensors 230 are also disposed in the robot 1 to overcome viewing angle limitations.
Below, a process in which the controller 250 generates a correlation between nodes of the pose graph in
The generated correlation is stored in the map storage 210 along with the LiDAR branch, the visual branch and the backbone in
A correlation may be set between adjacent nodes by combining information acquired by each sensor. As an example, the controller 250 may calculate a correlation by combining the LiDAR sensor 220, the camera sensor 230 and rotations/directions of the wheels.
Odometry applicable between nodes includes wheel odometry (WO), visual odometry (VO), and LiDAR odometry (LO). The WO indicates odometry based on the wheel encoder 260.
The LO is LiDAR odometry information based on the LiDAR sensor 220. The VO is visual odometry information based on the camera sensor 230. The controller 250 may calculate a correlation between adjacent nodes on the basis of each of the LiDAR odometry information and the visual odometry information or a combination of the LiDAR odometry information and the visual odometry information.
The controller 250 sets i to 1 (S51), and starts a process of calculating a correlation. The controller 250 confirms a keyframe commonly registered between adjacent nodes vi and v(i+1) (S52). Being commonly registered denotes that the same sort of frames (a LiDAR frame or a visual frame) are respectively registered in both nodes.
That is, when a first LiDAR frame is registered in the first node, and a second LiDAR frame and a first visual frame are registered in the second node, the LiDAR frames are commonly registered in the first node and the second node. Odometry information between the first node and the second node is calculated on the basis of the first LiDAR frame and the second LiDAR frame.
The process is branched into step 54 or step 55 on the basis of whether a commonly registered keyframe includes a visual keyframe as a result of confirmation by the controller 250.
When a commonly registered keyframe does not include a visual keyframe, the controller 250 confirms whether the keyframe commonly registered in step 54 is a LiDAR keyframe (S54). When the keyframe commonly included in step 54 is not a LiDAR keyframe, there're is no common keyframe between the two nodes (vi and v(i+1)). Accordingly, WO is set on the basis of a correlation of edges ei(i+1) of the nodes (vi and v(i+1)) (S56). This denotes that wheel odometry information that moves from between the two nodes by the wheels is set as a correlation.
When the keyframe commonly included in step 54 is a LiDAR keyframe, the LiDAR keyframes are the only common keyframe between the two nodes (vi and v(i+1)). Accordingly, WO⊙LO is set on the basis of a correlation between edges ei(i+1) of the nodes (vi and v(i+1)) (S57). This indicates that information, in which odometry information that moves from between the two nodes by the wheels and odometry information matched using a LiDAR keyframe are combined, is set as a correlation.
⊙ is a sort of operator. A⊙B denotes a combination of information A and information B. In this process, when there is a difference between information A and information B, any information having higher priority among information A and information B may be first applied.
As an example, LiDAR odometry information matched using a LiDAR keyframe is a difference between LiDAR keyframes of two nodes.
When a commonly registered keyframe includes a visual keyframe, the controller 250 confirms whether the commonly registered keyframe further includes a LiDAR keyframe in step 55 (S55). When the commonly included keyframe does not have a LiDAR keyframe in step 55, the visual keyframe is the only common keyframe between two nodes (vi and v(i+1)).
Accordingly, a correlation of edges ei(i+1) between the two nodes is set as WO⊙VO (S58). This indicates that information, in which odometry information that moves from between the two nodes by the wheels and odometry information matched using a visual keyframe are combined, is set as a correlation.
Visual odometry information matched using a visual keyframe is a difference between visual keyframes of two nodes.
When the commonly registered keyframe further includes a LiDAR keyframe in step 55, the LiDAR keyframe and the visual keyframe are registered as common keyframes between two nodes (vi and v(i+1)). Accordingly, a correlation of edges ei(i+1) between the two nodes is set as (WO⊙LO)⊙VO (S59).
This indicates that information, in which odometry information that moves from between the two nodes by the wheels, and odometry information matched using a LiDAR keyframe, and odometry information matched using a visual keyframe are all combined, is set as a correlation.
After the correlation of edges between the two adjacent nodes (vi and v(i+1)) is set, the controller 250 increases i by 1 (S60), and sets a correlation between the following adjacent nodes. If i is greater than or equal to a maximum number of nodes (MAX NODE, value of 5 in
The combination of correlations in step 56 to 59 is specifically described as follows.
When a speed of comparison of information sensed by the LiDAR sensor 220 is higher than a speed of comparison of information sensed by the camera sensor 230, the controller 250 may perform matching first on LO.
When a speed of comparison of information sensed by the LiDAR sensor 220 is lower than a speed of comparison of information sensed by the camera sensor 230, the controller 250 may perform matching first on VO. This is applied exemplarily when the LiDAR sensor 220 senses three-dimensionally. In this case, a combination may be (WO⊙VO)⊙LO.
In
The process in
Additionally, the controller 250 generates LiDAR odometry information using two LiDAR frames registered between the two nodes (S57). The LiDAR odometry information may include a differential value of the two LiDAR frames (each LiDAR frame registered in each node) between the two nodes, or differential values changed according to a distance between the two nodes may be stored as LiDAR odometry information.
Further, the controller 250 generates visual odometry information using two visual frames registered between the two nodes (S58). The visual odometry information may include a differential value of the two visual frames (each visual frame registered in each node) between the two nodes, or differential values changed according to a distance between the two nodes may be stored as visual odometry information.
In
Nodes that are connected to strong correlation visual edges (strong visual edges), and nodes that are connected to weak correlation visual edges (weak visual edges) are connected by dotted lines around the reference node in
Next, the controller 250, as illustrated in
Specifically, the controller 250 as in
Next, the controller 250 as in
For example, suppose that a first node and a second node constitute an essential graph and have a strong correlation, and that a third node does not constitute an essential graph.
A first visual frame was registered in the first node, a second visual frame was registered in the second node, and a third visual frame was registered in the third node.
The controller 250 configures nodes, in which visual frames having a high similarity are registered, as an essential graph. Thus, a similarity between the first visual frame and the second visual frame is higher than a similarity between the third visual frame and the first visual frame, or a similarity between the first visual frame and the second visual frame is higher than a similarity between the third visual frame and the second visual frame.
According to results of
Next, the controller 250 as in
As an example of the loop closing correlation, the correlation may include displacement between two nodes. In this case, when the robot moves between two nodes, the controller 250 may confirm a position of the robot by comparing a value currently confirmed by the LiDAR sensor and the above-described displacement.
As described with reference to
That is, in nodes in which LiDAR information (LiDAR keyframes) and visual information (vision information, visual keyframes) are all registered, the controller 250 may verify the accuracy of information acquired by the camera sensor 230 using the LiDAR sensor 220 and may modify the information.
The controller 250 may select a node subject to matching using the LiDAR sensor among the keyframes or frames acquired by the camera sensor 230.
Additionally, the controller 250 may calculate a correlation between non-adjacent nodes of information sensed by the LiDAR sensor 220 using the visual essential graph for calculating a correlation between non-adjacent nodes.
The process in
Next, the controller 250 generates a visual essential graph using the candidate visual edges in response to line 7 (step 65, and step 66 in
Then the controller 250 extracts a LiDAR loop correlation on the basis of nodes including LiDAR keyframes, a distance between the nodes, and whether there are LiDAR edges between the nodes in response to lines 10 and 11 (step 67 in
Topological_dist(vi, vj) function of line 11 denotes the number of edges included in the shortest path between two nodes (vi, and vj).
According to steps in
When the above-described embodiments are applied, the controller 250 calculates a correlation between nodes, in which keyframes are registered, on a map in which different types of data (LiDAR keyframes and visual keyframes) generated by different types of sensors are registered, thereby enhancing accuracy of fusion SLAM while the robot moves between the nodes.
That is, the robot 1 in fusion SLAM may calculate a correlation (constraints or edges) between nodes more accurately than a robot that performs SLAM ((LiDAR-only SLAM or visual-only SLAM) using a single sensor because errors of any one sensor may be compensated by other sensor.
The controller 250 as illustrated in
Additionally, the controller 250 as illustrated in
As a result, the robot 1 according to embodiments of the present disclosure may perform fusion SLAM using all the information generated by the LiDAR sensor 220 and the camera sensor 230, thereby enhancing accuracy at the time when the robot moves. Additionally, the robot 1 may generate a correlation between adjacent/non-adjacent nodes, and may use the correlation during performing fusion SLAM, thereby enhancing accuracy of the process of fusion SLAM.
Below, the processes in which the robot moves using a map that stores a correlation of nodes and in which the current position of the robot is calculated are described. That is, the controller 250 calculates the current position of the robot by comparing correlations of wheel odometry information, LiDAR frames sensed by the LiDAR sensor, visual frames sensed by the camera sensor, and nodes stored in the map storage 210.
When wheel odometry (WO) is the only odometry information as a result of confirmation (S83), the controller 250 calculates the current position of the robot by comparing WO registered in edges of the first frame node and the second frame node, and values sensed by the wheel encoder 260 of the robot (S84).
For example, the wheel encoder 260 calculates a distance moved by the robot while the robot is moving from the first frame node. The controller 250 may calculate the current position of the robot 1 on the basis of WO (wheel odometry information) between the first frame node and the second frame node with respect to the first frame node and the second frame node.
When there is odometry information in addition to WO in step 83, step 85 is performed. When odometry information registered in the edges of the first frame node and the second frame node belongs to WO and LO, step 86 is performed.
When odometry information registered in the edges of the first frame node and the second frame node belongs only to WO and LO (S86), step 87 is performed. Additionally, the controller 250 calculates the current position of the robot by comparing WO and LO registered in the edges of the first frame node and the second frame node, and values (wheel odometry information) sensed by the wheel encoder of the robot, and information (LiDAR frames) sensed by the LiDAR sensor (S87).
When odometry information registered in the edges belongs to WO and LO and VO in step 86, step 88 is performed. The controller 250 calculates the current position of the robot by comparing WO, LO and VO registered in the edges of the first frame node and the second frame node, and values (wheel odometry information) sensed by the wheel encoder of the robot, information (LiDAR frames) sensed by the LiDAR sensor, and information (visual frames) sensed by the camera sensor 230 (S88).
In step 88, a single frame may be first applied to calculation of the position of the robot according to the size of data of the LiDAR frames and the size of data of the visual frames. For example, in the case in which a small amount of time is spent on comparing LiDAR frames, WO and LO are first used to calculate the position of the robot 1. The controller 250 calculates the current position of the robot by first combining the wheel odometry information and the LiDAR odometry information rather than the visual odometry information.
Additionally, to enhance accuracy of calculation of the position, the controller 250 may re-calculate the position of the robot 1 using VO. When a two-dimensional LiDAR sensor generates LiDAR frames, the size of data is small. Accordingly, LiDAR frames and LiDAR odometry information may be first applied.
Conversely, in the case in which a small amount of time is spent on comparing visual frames, WO and VO are first used to calculate the position of the robot 1. Additionally, to enhance accuracy of calculation of the position, the controller 250 may re-calculate the position of the robot 1 using LO. When a three-dimensional LiDAR sensor generates LiDAR frames, the size of data is large. Accordingly, visual frames and visual odometry information may be first applied.
When odometry information, registered in the edges in step 85, does not belong to WO and LO, the odometry information belongs to WO and VO. Accordingly, step 89 may be performed. Additionally, the controller 250 calculates the current position of the robot by comparing WO and VO registered in the edges of the first frame node and the second frame node, and values (wheel odometry information) sensed by the wheel encoder of the robot, and information (visual frames) sensed by the camera sensor (S89).
The process in
As a result, the controller 250 calculates the current position of the robot by comparing the first LiDAR frame or the first visual frame, which is generated in the current position of the robot, or wheel odometry information in the current position and an extracted correlation (odometry information, WO, LO, and VO) between nodes while the robot is moving from the first frame node to the second frame node.
That is, when there is any one of LiDAR odometry information or visual odometry information, which is registered between adjacent nodes, the controller 250 may calculate the current position of the robot by combining wheel odometry information and any one of the LiDAR odometry information or visual odometry information.
The controller 250 calculates the current position of the robot by comparing wheel odometry information (WO) and a distance moved by the robot and sensed by the wheel encoder, and by comparing LiDAR odometry information (LO) or visual odometry information (VO) and LiDAR/visual frames sensed by the robot.
The robot 1 is moving from a first frame node to a second frame node (S91). The first frame node and the second frame node are non-adjacent nodes. Additionally, suppose that a LiDAR loop closing correlation is registered between the first frame node and the second frame node.
The controller 250 extracts a loop closing correlation between the first frame node and the second frame node (S92). Additionally, the controller 250 calculates the current position of the robot with respect to the first frame node and the second frame node by comparing a first LiDAR frame generated by the LiDAR sensor and the loop closing correlation (S93).
Although in embodiments, all the elements that constitute the embodiments of the present disclosure are described as being coupled to one or as being coupled to one so as to operate, the disclosure is not limited to the embodiments. One or more of all the elements may be optionally coupled to operate within the scope of the present disclosure. Additionally, each of the elements may be implemented as single independent hardware, or some or all of the elements may be optionally combined and implemented as a computer program that includes a program module for performing some or all of the combined functions in single hardware or a plurality of hardware. Codes or segments that constitute the computer program may be readily inferred by one having ordinary skill in the art. The computer program is recorded on computer-readable media and read and executed by a computer to implement the embodiments. Storage media that store computer programs includes storage media magnetic recording media, optical recording media, and semiconductor recording devices. Additionally, the computer program that embodies the embodiments includes a program module that is transmitted in real time through an external device.
The embodiments of the present disclosure have been described. However, the embodiments may be changed and modified in different forms by one having ordinary skill in the art. Thus, it should be understood that the changes and modifications are also included
This application is the National Stage filing under 35 U.S.C. 371 of International Application No. PCT/KR2019/005885, filed on May 16, 2019, the contents of which are all hereby incorporated by reference herein in their entirety.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/KR2019/005885 | 5/16/2019 | WO | 00 |