Patent Application
20210150750
This application is based upon and claims the benefit of priority from the Japanese Patent Application No. 2019-207099, filed Nov. 15, 2019, the entire contents of which are incorporated herein by reference.
Embodiments described herein relate to a moving body and a system including the same.
Development of a moving body for automatic conveyance of a cage truck at a distribution warehouse is in progress. A moving body receives a cage truck loaded with packages at a designated position, autonomously carries a received cage truck to a designated position and displays the cage truck at a designated position. Among these operations, in order to accurately perform the operations for receiving the cage truck and displaying the cage truck, an accurate position estimate of the moving body becomes necessary.
As a method for estimating the position of the moving body, there exists the use of an obtained image by a camera attached to a moving body. For example, in one method, a position information database which stores an image captured at a target position and an image captured at a plurality of positions in a traveling area are prepared in advance, and the position of the moving body is estimated by referring to an image stored in the position information database and an image obtained by the camera of the moving body.
When estimating the position by referring to an image, if a capture conditions at the time of capture, such as a shaking camera etc. are not considered, the image quality of the image to be referred will be unstable and the degree of estimation accuracy for the position will decrease.
The embodiment provides a moving body and a system including the same where the estimation accuracy of the position is improved.
According to one embodiment, a moving body includes processing circuitry. The processing circuitry is configured to collect an external image of the moving body from an external sensor and collect information associated with an inner state of the moving body from an internal sensor. The processing circuitry is configured to determine whether or not to adopt the external image collected by the external sensor as a first image for estimating a position of the moving body, based on the information associated with the inner state of the moving body collected by the internal sensor. The processing circuitry is configured to estimate a position of the moving body by comparing the first image and a second image associated with a collection position of the first image.
Hereinafter, embodiments will be described with reference to the drawings.
The moving body 50 includes a moving body control unit 51, and a sensor information collection unit 52. Further, though not shown in the drawings, the moving body 50 includes a drive mechanism necessary for moving.
The moving body control unit 51 performs control associated with the movement etc. of the moving body 50. The moving body control unit 51, for example, performs control of the drive mechanism for moving the moving body 50 to a designated target position.
The sensor information collection unit 52 includes various types of an internal sensor and an external sensor installed in the moving body 50, and collects various sensor information. The internal sensor is a sensor for enabling the moving body 50 to collect sensor information associated with its own state. In the embodiment, the internal sensor mainly collects sensor information associated with movement of the moving body 50. On the other hand, the external sensor is a sensor which collects sensor information associated with a state outside the moving body 50. In the embodiment, the external sensor mainly collects an image outside the moving body 50.
The position estimation device 1 is configured to estimate a position of the moving body 50 from an image collected by the moving body 50. The position estimation device 1 may take the form of a superior system for controlling movement of the moving body 50 based on an estimation result of the position of the moving body 50. The position estimation device 1 may be for example, a personal computer (PC). The position estimation device 1 includes a sensor information processing unit 10, a sensor information acquiring unit 21, a position estimation unit 22, a control unit 23, and a dictionary storage unit 30.
The sensor information acquiring unit 21 acquires sensor information from the sensor information collection unit 52 of the moving body 50. For example, the sensor information acquiring unit 21 extracts sensor information from a signal communicated from the moving body 50.
The sensor information processing unit 10 processes sensor information acquired at the sensor information acquiring unit 21. The sensor information processing unit 10 includes an evaluation information calculating unit 11, and a determination unit 12.
The evaluation information calculating unit 11 calculates evaluation information for determining by the determination unit 12, based on sensor information of the internal sensor of the moving body 50 acquired by the sensor information acquiring unit 21. The evaluation information is, e.g., information associated with a movement of the moving body 50, and includes for example, velocity, angular velocity, and acceleration of the moving body 50.
The determination unit 12 determines whether or not an image outside the moving body 50 acquired by the sensor information acquiring unit 21 can be adopted for the estimation of the position of the moving body 50. The determination unit 12 determines to adopt, for the estimation of the position of the moving body 50, an image of evaluation information with similar evaluation information at the time of collecting images stored in the dictionary storage unit 30.
The position estimation unit 22 estimates a position of the moving body 50 based on the image determined to be adopted for the estimation of the position by the determination unit 12 out of the images outside the moving body 50 acquired by the sensor information acquiring unit 21, and the images registered to the dictionary storage unit 30.
The control unit 23 controls the operation of the moving body 50 in accordance with the position of the moving body 50 estimated at the position estimation unit 22.
The dictionary storage unit 30 stores a dictionary for storing an image of a target position of the moving body 50, images collected at a plurality of capture positions near the target position, feature points and feature amounts extracted from each image, the degree of correspondence between each image, information on the collection position for each image, and evaluation information for each image.
The CPU 101 is a processor for controlling the entire operation of the position estimation device 1. The CPU 101, by executing a program stored in, e.g., the storage 105, operates as the sensor information processing unit 10, the sensor information acquiring unit 21, the position estimation unit 22, and the control unit 23. The CPU 101 may be replaced with MPU, GPU, ASIC, FPGA etc. The CPU 101 may be a single CPU etc. or a plurality of CPU etc.
The input device 102 is an input device such as a joystick, a touch panel, a keyboard, and a mouse etc.
When the input device 102 is operated, a signal corresponding to the operation matter is input to the CPU 101 via the bus 106. The CPU 101 performs various processes in accordance with this signal.
The display 103 is a display such as liquid crystal display or organic EL display. The display device 103 displays various images.
The communication device 104 is a communication device for performing, for example, a wireless LAN communication. The communication device 104 communicates with the moving body 50. The communication device 104 need not necessarily be a communication device for wireless LAN communication.
The storage 105 is a storage device such as a hard disk drive or a solid state drive. The storage 105 operates as the dictionary storage unit 30. Further, the storage 105 stores various programs executed by the CPU 101.
The bus 106 is a data transfer path for data exchange between the CPU 101, the input device 102, the display 103, the communication device 104, and the storage 105.
The following explains in more detail the matters regarding each configuration shown in
The moving body control unit 51 performs control of the drive mechanism for moving the moving body 50 to a designated target position. The moving body control unit 51 recognizes its position by the position estimated at the position estimation device 1, and may control the drive mechanism by determining the necessary direction and distance for moving towards the designated target position. The target position of the moving body 50 may be acquired from any one of: the host system of the moving body 50, e.g. the position estimation device 1, invoking a prior setting, or a setting directly performed by a person. The target position is not only a spatial position of the moving body 50, and may include a posture of the moving body 50 at the target position. Unless otherwise stated in the embodiment, the position and the posture of the sensor mounted to the moving body 50 are the position and the posture of the moving body. When the target position is designated, specific position coordinates may be specified and, through use of the map information etc. of the operation, areas such as “position A” or “operation area B” may also be designated. Further, the instruction of movement is not limited to the instruction for designating the absolute position, and may be an instruction for designating a relative position from the current position. For example, the instruction of movement may be an instruction such as circling 30 degrees clockwise at a position 1m ahead. When moving a predetermined location, a movement path may also become important. Even if the moving body tries to simply move linearly to a predetermined position, there are cases where the movement becomes difficult due to obstacles etc. The movement path may be decided by the moving body control unit 51 or provided by a superior system. Thus, the movement path may be provided by invoking a preset matter or by a direct setting made by a person.
The internal sensor of the sensor information collection unit 52 for example, includes, a rotary encoder, an acceleration sensor, and an angular velocity sensor such as a gyro sensor. The movement amount and the posture of the moving body 50 may be measured by the internal sensor. From the movement amount and the posture of the moving body 50, an estimated position of the moving body 50 can be acquired.
The external sensor of the sensor information collection unit 52 collects the images outside of the moving body 50. It is preferable if the external sensor collects, in addition to the images outside the moving body 50, sensor information for measuring or calculating a peripheral distance of the moving body 50. It is further preferable if the external sensor collects sensor information for measuring or calculating the peripheral distance of the moving body 50 at the surface. For example, Depth Camera and LIDAR (Light Detecting and Raging) etc. can collect planar distance images. The distance image is an image generated by converting the value of distance to a value of brightness. The Depth Camera may be any method such as ToF (Time of Flight) method, pattern irradiation method etc. Further, even a line measurement laser range finder, may collect planar distance images by altering the mechanical measurement direction or a measurement direction for multiple units. In addition, it is possible to utilize a stereo camera or single lens camera etc. as a method of indirect measurement. In the case of a stereo camera, images may be converted to distance information by a stereo matching method. Even in the case of a single lens camera, changes to the moving body position and posture can realize capture akin to that of a stereo camera, rendering it possible to calculate the distance information. The external sensor may be something other than the above. Unless otherwise stated in the embodiment, the external sensor is a stereo camera.
Next, the sensor information processing unit 10 will be further explained. The evaluation information calculating unit 11 calculates evaluation information associated with movement of the moving body 50 from internal sensor information acquired at the sensor information acquiring unit 21.
The information associated with the movement of the moving body 50 is, for example, the velocity of the moving body 50. In this case, the evaluation information calculating unit 11 calculates the amount of movement of the moving body 50 from a measurement value of the rotary encoder as an internal sensor, attached to for example, wheels of the moving body 50. In addition, the evaluation information calculating unit 11 calculates the velocity of the moving body 50 by differentiating the calculated moving amount by time. The evaluation information calculating unit 11 may calculate the velocity of the moving body 50 by converting the rotation speed of the drive mechanism, such as wheels etc., to a velocity of the moving body 50, or by calculating the velocity of the moving body 50 by converting the angular velocity of the drive mechanism, such as wheels etc., to the velocity of the moving body 50. Further, if an acceleration sensor is attached as an internal sensor to the moving body 50, the evaluation information calculating unit 11 may calculate the velocity of the moving body 50 by integrating the acceleration of the main body of the moving body 50. Further, the evaluation information calculating unit 11 may calculate the velocity of the moving body 50 from a drive current value etc. input from the moving body control unit 51 to the drive mechanism of the wheels etc. In other words, the evaluation information calculating unit 11 may calculate the velocity using sensor information of any type of internal sensor.
The information associated with the movement of the moving body 50 is not limited to the velocity of the moving body 50. For example, the information associated with the movement of the moving body 50 may be a rotational speed or angular velocity of the drive mechanism of wheels etc. provided to the moving body 50, or acceleration of the moving body 50.
The determination unit 12, by referring to the evaluation information associated with movement of the moving body 50 calculated at the evaluation information calculating unit 11, and evaluation information associated with an image of the target position registered in the dictionary storage unit 30, determines whether or not to adopt an image collected by the moving body 50 as an image for estimating a position of the moving body 50. More specifically, the determination unit 12, when the difference in the evaluation information is small, determines that the image collected at the moving body 50 is a suitable image for estimating a position of the moving body 50. At this point, the determination unit 12 determines to adopt the image collected at the moving body 50 as the image for estimating the position of the moving body 50.
For example, the determination unit 12 determines if the image collected at the moving body 50 is the suitable image for estimating the position of the moving body 50, when a difference between the velocity of the moving body 50 calculated at the evaluation information calculating unit 11 and the velocity associated with the image of the target position registered in the dictionary storage unit 30 is within a certain range. This range of velocity difference may be designated by a person, or defined from the ratio to the velocity associated with the image of the target position registered to the dictionary storage unit 30. For example, the range of velocity difference may be set to 0±0.1 [m/s] for a state when the moving body 50 is stopped. In addition, for example, if the image of the target position registered in the dictionary storage unit 30 is captured when the velocity is 3 [m/s], the range of the velocity difference is set to ±0.3 [m/s] which is its 10%.
The determination unit 12 may convert the velocity registered in the dictionary storage unit 30 to sensor information of the internal sensor such as the rotary encoder or the gyro sensor etc. and compares such converted sensor information with sensor information of the internal sensor. In this case, the determination unit 12 may determine that the image collected by the moving body 50 is the suitable image for estimating the position of the moving body 50, when the difference between the sensor information acquired at the sensor information acquiring unit 21 and sensor information converted from the velocity registered in the dictionary storage unit 30 is within a predetermined range. For example, when the angular velocity converted from the velocity registered to the dictionary storage unit 30 is 10 [rad/s], the determination unit 12 determines that the image should be used if the difference of the angular velocity acquired by the sensor information acquiring unit 21 being ±1 [rad/s].
In addition, when a plurality of images of the target position are registered with the dictionary storage unit 30, the determination unit 12 may determine the target position as either that designated by the superior system, or one set in advance.
Next, the dictionary storage unit 30 is explained. The dictionary storage unit 30 stores a reference dictionary in which information necessary to the target position at the sensor information processing unit 10 and the position estimation unit 22 is registered. The registered information of the reference dictionary, for example, is the following.
There are two ways of registering with the reference dictionary when a plurality of target positions exists, 1) a method of collecting the registration information for all target positions and registering to one dictionary, or 2) a method separating the dictionary for each target position. The method 1) will not require the selection of a dictionary; however, as it becomes necessary to match the feature points for all images registered in the reference dictionary, the process of estimating the position will become time-consuming. Further, as it can be completed merely by the process of registering the necessary target position with the reference dictionary, the process time for estimating the position in method 2) is shorter. On the other hand, in the method 2), it is necessary to designate the reference dictionary to be used by a host system or person etc. As can be seen from the above reference dictionary registration methods 1) and 2), both have their advantages and disadvantages. Thus, it is preferable if the registration methods 1) and 2) for the reference dictionary are used in different situations when necessary.
In step S101, the control unit 23 of the position estimation device 1 starts moving the moving body 50 to one designated position. The designated position includes the target position and its surrounding capture positions. The control unit 23 selects one of the above and instructs the moving body 50 to move. With this instruction, the moving body 50 starts autonomous movement. The movement of the moving body 50 may be performed manually by the operation of a joystick etc.
In step S102, the control unit 23 determines, from sensor information acquired at the sensor information acquiring unit 21, whether or not the moving body 50 has stopped. The control unit 23, for example, calculates the velocity of the moving body 50 from the sensor information acquired at the sensor information acquiring unit 21, and determines whether the moving body 50 has stopped when the calculated velocity is equal to or lower than a threshold. For the velocity of the moving body 50, the velocity calculated at the evaluation information calculating unit 11 may be used. Here, the stoppage of the moving body 50 is not limited to the arrival of the target position or the capture positions. For example, the moving body 50 may be configured stop at a corner etc. when moving towards the target position or the capture positions. In the determination of step S102, even for the above case, if the velocity of the moving body 50 is equal to or lower than the threshold, it is determined that the moving body 50 has stopped. In step S102, the process is on standby until it is determined that the moving body 50 has stopped. In step S102, when it is determined that the moving body 50 has stopped, the process proceeds to step S103.
In step S103, the control unit 23 instructs the moving body 50 to capture and acquires an image from the moving body 50.
In step S104, the control unit 23 determines whether or not to a designated number of images have been acquired. In an exemplary instance, the control unit 23 determines that the designated number of images have been acquired when the images of the target position and all capture positions have been acquired. In step S104, if the designated number of images are not acquired, i.e., if there is still the capture position in which the image has not been acquired, the process will return to step S101. In this case, the control unit 23 will designate a new capture position and start the movement of the moving body 50. In step S104, if a designated number of images are acquired, the process proceeds to step S105.
In step S105, the position estimation unit 22 detects the feature points of the acquired images. The position estimation unit 22 may detect the feature points using SIFT (Scale Invariant Feature Transform), AKAZE (Accelerated KAZE) etc.
In step S106, the position estimation unit 22 extracts feature amounts from the respective detected feature points. In the position estimation unit 22, the feature amounts may be extracted in accordance with the method used in the feature point detection.
In step S107, the position estimation unit 22 performs feature point matching for the target position image and the image at each capture position. More specifically, the position estimation unit 22 associates the feature points between the target position image and the image at each capture position so that the difference in the feature amounts will be the smallest. The position estimation unit 22 may perform feature point matching at the method of NN (Nearest Neighbor), k-NN, kd-Tree, humming distance etc.
In step S108, the position estimation unit 22 performs estimation of correspondence between a target position image and an image of each capture position. For example, the position estimation unit 22 performs estimation of relative position and relative posture of the moving body 50 at each capture position to the target position from a correspondence of feature points between a target position image and each capture position image. In addition, the position estimation unit 22 generates three-dimensional information for each image by a triangulation principle using the designated relative position and relative posture.
In step S109, the position estimation unit 22 registers to the reference dictionary: feature points and feature amounts for every image, correspondence between the images of each capture position and the target position, coordinates of the target position and the capture positions, and evaluation information associated with the movement of the moving body 50 when collecting respective images, e.g., velocity of the moving body 50.
Here, the number of images registered to the reference dictionary, i.e., the number of capture positions, is not limited to a specific value. Furthermore, in principle, no problem is posed by any type of positional relationship between the target position and each capture position. In a practical sense, it is desirable that the capture positions are positioned within a vicinity range NB of the target position.
Next, the position estimation unit 22 is explained. The position estimation unit 22 estimates a position of the moving body 50 by comparing the image among those acquired at the sensor information acquiring unit 21 that is determined, by the determination unit 12 of the sensor information processing unit 10, as being adoptable for estimation of a position of the moving body 50, with an image registered to the reference dictionary of the dictionary storage unit 30.
In step S201, the sensor information acquiring unit 21 acquires sensor information, e.g., a measurement value of a rotary encoder, of an internal sensor from the sensor information collection unit 52 of the moving body 50, and sensor information, e.g., an image obtained by a stereo camera, of an external sensor. The sensor information acquiring unit 21 acquires sensor information, from the sensor information collection unit 52, of the moving body 50 at, for example, every constant interval.
In step S202, the evaluation information calculating unit 11 calculates evaluation information from sensor information of the internal sensor acquired at the sensor information acquiring unit 21. For example, the evaluation information calculating unit 11 calculates the velocity of the moving body 50 from the measurement value of the rotary encoder.
In step S203, the determination unit 12 determines whether or not the image acquired from the sensor information acquiring unit 21 is suitable for estimating the position of the moving body 50. For example, the determination unit 12 determines if the image collected at the moving body 50 is the suitable image for estimating the position of the moving body 50, when a difference between the velocity of the moving body 50 calculated at the evaluation information calculating unit 11 and the velocity associated with the image of the target position registered in the dictionary storage unit 30 is within a certain range. In step S203, the process of
In step S203, when the image acquired from the sensor information acquiring unit 21 is determined as suitable for estimating the position of the moving body 50, the process proceeds to step S204.
In step S204, the position estimation unit 22 acquires, from the sensor information acquiring unit 21, the image determined for adoption as the position estimate deemed suitable for such a purpose in step S203. Further, the position estimation unit 22 detects the feature points from the acquired image. The position estimation unit 22 may detect the feature points using SIFT (Scale Invariant Feature Transform), AKAZE (Accelerated KAZE) etc.
In step S205, the position estimation unit 22 extracts feature amounts from the respective detected feature points. In the position estimation unit 22, the feature amounts may be extracted in accordance with the method used in the feature point detection.
In step S206, the position estimation unit 22 performs feature point matching for the target position image and images of surrounding capture positions registered to the reference dictionary stored in the dictionary storage unit 30, as well as for the image acquired at the sensor information acquiring unit 21. More specifically, the position estimation unit 22 performs the feature point matching so that the difference in feature amount will be at the minimum. When a plurality of target positions are registered to the reference dictionary, it is necessary to select the target position. The target position may be selected by either 1) a method of acquiring a target position from a host system mounted on the moving body, or 2) a method of estimating positions of the moving body 50 to all target positions registered to the reference dictionary and selecting the best estimation result.
In step S207, the position estimation unit 22 calculates the relative position using PnP (Perspective n-Point) from the relationship between three-dimensional point groups and its corresponding two-dimensional points of feature points registered to the reference dictionary. Further, the position estimation unit 22 estimates a position of the moving body 50 from the target position. Afterwards, the process of
In the first embodiment as explained above, when the difference between evaluation information based on sensor information collected at the internal sensor included by the moving body 50 and evaluation information based on sensor information at the time of collecting the image registered to the reference dictionary is small, the image collected by the external sensor included to the moving body 50 is adopted as the image for estimating the position of the moving body 50.
The precision of the position of the moving body 50 estimated at the position estimation unit 22 changes greatly depending on whether or not the capture conditions are considered.
Further, in the embodiment, the image registered to the reference dictionary is captured when the velocity of the moving body 50 is equal to or lower than the threshold. Thus, it is considered that the blurring of images registered to the reference dictionary is kept to a minimum. Furthermore, if the moving body 50 is assumed to move autonomously, loss of speed often means it is at a position to be focused on, such as, for example, near the target position, a corner, or a dead end etc. A meaningful position estimation of the moving body 50 is performed by performing capture at such positions.
In the first embodiment, the moving body 50 and the position estimation device 1 are described as separate systems. However, the moving body 50 and the position estimation device 1 may be integral. Further, a part of the position estimation device 1 may be separate from the position estimation device 1. For example, the dictionary storage unit 30 may be provided with an external server outside the position estimation device 1.
The processing unit 24 compares an image of the target position registered to the reference dictionary of the dictionary storage unit 30 with an image of the target position collected at a different time in the moving body 50, and detects a defective part in the target position. The defective part is a part that does not originally exist in the target position such as holes, cracks, and scratches etc. on the walls.
As previously explained, the position estimation unit 22 of the embodiment estimates the position of the moving body 50 using an image adopted by the determination unit 12. Therefore, when the position of the moving body 50 estimated by the position estimation unit 22 is the target position, the capture conditions of an image SI, acquired at the sensor information acquiring unit 21, are similar to the capture conditions of an image RI of the target position registered to the reference dictionary, meaning that image RI and the image SI matches basically match. On the other hand, when the defective part is in the target position, as shown in
As a detection process of the defective part by the processing unit 24, various processes may be applied, such as: corresponding two images using feature point matching, and having the part with the greatest difference indicated as the defective part; detecting a part with a low degree of similarity found by template matching as the defective part; detecting a part with the biggest difference in brightness by a background subtraction method.
As explained above, according to the second embodiment, the detection of a defective part is performed using high quality images. Thus, various applications of the moving body 50 using the maintenance and inspection can be anticipated.
Similar to the first embodiment, the moving body 50 and the position estimation device 1 may be integral in the second embodiment. Further, a part of the position estimation device 1 may be separate from the position estimation device 1. For example, the dictionary storage unit 30 may be provided with an external server outside the position estimation device 1.
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2019-207099 | Nov 2019 | JP | national |
