This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2017-020462, filed on Feb. 7, 2017, the entire contents of which are incorporated herein by reference.
The embodiments discussed herein are related to a moving-object position estimating system, an information processing apparatus and a moving-object position estimating method.
In recent years, development of autonomous moving objects, such as autonomous vehicles and robots, is under way.
Japanese Laid-open Patent Publication No. 2007-263657, International Publication Pamphlet No. 2012/172870, or Japanese Laid-open Patent Publication No. 2004-12429 discloses related technologies.
According to an aspect of the embodiments, a moving-object position estimating system includes: a moving-object position estimating terminal apparatus that estimates a position a moving object; and an information storing apparatus that stores first environment map information transmitted from the moving-object position estimating terminal apparatus, wherein the moving-object position estimating terminal apparatus: acquires an image in surroundings of the moving object; obtains feature point information regarding feature points of the acquired image; obtains photography environment information in surroundings of the feature points in synchronization with acquiring the image; associates the feature point information with the photography environment information to set the associated feature point information and photography environment information as the first environment map information; transmits, to the information storing apparatus, a request for retrieving second environment map information corresponding to the photography environment information in the first environment map information from the environment map information prior to the first environment map information and sending back the second environment map information; receives the second environment map information; and compares the feature point information included in the second environment map information with the feature point information included in the first environment map information to estimate the position of the moving object.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed.
An autonomous moving object moves to a destination autonomously while recognizing the position of the autonomous moving object, based on map information. Thus, the position of the autonomous moving object in the map information is continuously recognized in real time.
For example, a three-dimensional image of the surroundings of a vehicle is acquired using a distance-measuring camera mounted on the vehicle, and feature points in the acquired three-dimensional image are compared with feature points in a known three-dimensional image to thereby estimate the position of the moving object. In a moving object used inside a room, an image acquired by photography in a ceiling direction by using a camera mounted on the moving object is compared with a large number of images acquired by pre-photography in the ceiling direction under a different condition, such as lighting, to thereby estimate the position of the moving object.
For example, in the aforementioned technologies, how feature points look vary depending on a condition, such as weather, which may make it difficult to compare feature points in a newly acquired three-dimensional image with feature points in a known three-dimensional image and may, thus making it difficult to perform position estimation. Since an image acquired by the camera mounted on the moving object is compared with all images pre-acquired under a different condition, it may take a computational processing time.
For example, a moving-object position estimating system that is capable of estimating the position of a moving object in a short period of time even when the environment changes may be provided.
Since the “moving-object position estimating system” includes a “moving-object position estimating terminal apparatus” and an “information storing apparatus”, a description will also be given of the “moving-object position estimating terminal apparatus” and the “information storing apparatus” in conjunction with the entire “moving-object position estimating system”. Control performed by the controller in the moving-object position estimating terminal apparatus may be synonymous with executing a moving-object position estimating method. The moving-object position estimating terminal apparatus may be referred to as a “vehicle-mounted apparatus”, and the information storing apparatus may be referred to as a “server apparatus”.
For example, in the technology called simultaneous localization and mapping (SLAM), an environment map can be created based on feature points or the like in an image acquired by photographing the surroundings of a moving object, and the position of the moving object in the created environment map can be estimated. However, in a case in which the weather is sunny and bright when the environment map is created for the first time, and then the weather is cloudy and dark when the moving objects moves back to the same position, feature points in images acquired at the same position may differ from each other, which may make it difficult to compare the feature points and make it difficult to estimate the position of the moving object. For example, in the moving-object position estimating system 10, even when the photography environment, such as weather, changes, the position of a moving object is estimated in a short period of time.
For example, in the moving-object position estimating system 10, the moving-object position estimating terminal apparatus 100 acquires an image of the surroundings of the vehicle 200 and obtains the first environment map information in which photography environment information, such as ambient brightness and weather, at the time of the image acquisition and the feature point information in the acquired image are associated with each other.
The feature point information in the image of the surroundings of the vehicle 200 includes position information including features at feature points and three-dimensional coordinate information of the feature points. The feature points (like those illustrated in
The moving-object position estimating terminal apparatus 100 transmits, to the information storing apparatus 300, a request for sending back second environment map information corresponding to the photography environment information in the first environment map information, and compares feature point information included in the second environment map information sent back from the information storing apparatus 300 with the feature point information included in the first environment map information to estimate the position of the vehicle 200. The “second environment map information corresponding to the photography environment information in the first environment map information” means “second environment map information for an image acquired in a photography environment that is the same as or similar to the photography environment where the image corresponding to the first environment map information was acquired”. Thus, it is preferable that the information storing apparatus 300 extract the second environment map information including photography environment information that matches photography environment information in the first environment map information exactly. However, it is sufficient to be able to extract most-similar second environment map information, and second environment map information in a predetermined similar range may be extracted.
The moving-object position estimating terminal apparatus 100 transmits the first environment map information to the information storing apparatus 300. The information storing apparatus 300 stores the received first environment map information in a database to update the database. This makes it possible to accumulate environment map information in various photography environments and makes it possible to update the environment map information to most-recent environment map information. This makes it easier to compare the first and second feature point information even when the photography environment changes, thus making it possible to reliably estimate the position of the vehicle 200 in a long period of time.
Upon receiving, from the moving-object position estimating terminal apparatus 100, a request for sending back the second environment map information corresponding to the photography environment information in the first environment map information, the information storing apparatus 300 transmits the second environment map information to the moving-object position estimating terminal apparatus 100, the second environment map information being retrieved from a database of the second environment map information, based on the photography environment information in the first environment map information. Since the information storing apparatus 300 retrieves the second environment map information from the database of the second environment map information, based on the photography environment information in the first environment map information, as described above, it is possible to estimate the position of the vehicle 200 in a short period of time without referring to all the second environment map information in the database of the second environment map information, which makes it easier to realize real-time processing.
It is preferable that, from the standpoint of increasing the amount of obtainable environment map information to enhance the update frequency of the environment map information, the number of vehicles 200 on which the moving-object position estimating terminal apparatuses 100 are mounted be two or more and be increased. Also, the number of information storing apparatus 300 may be two or more. Examples of the vehicle 200 include a large-size automobile, a mid-size automobile, a normal-size automobile, a large-size special automobile, a small-size special automobile, a large-size motorcycle, and a normal-size motorcycle, and it is preferable that vehicle 200 is an autonomous vehicle in an autonomous driving system.
The image acquirer 101 acquires an image by photographing the surroundings of the vehicle 200 by using a camera installed on the vehicle 200 (S101 in
The photography environment information obtainer 102 obtains photography environment information in the surroundings of feature points in synchronization with the image acquirer 101 (S102 in
The feature point information obtainer 103 obtains feature point information regarding the feature points in the image acquired by the image acquirer 101 (S103 in
The communication controller 104 associates the feature point information in an image of the surroundings of the vehicle 200 with the photography environment information in the surroundings of feature points, sets the associated feature point information and photography environment information as the first environment map information, and performs control on transmitting, to the information storing apparatus 300, a request for retrieving the second environment map information corresponding to the photography environment information in the first environment map information and sending back the second environment map information (S106 in
In this case, the frequency at which the communication controller 104 transmits, to the information storing apparatus 300, the request for sending back the second environment map information may be at predetermined time intervals or for each predetermined area. A case in which the request is transmitted for each predetermined area is, for example, a case in which a request for sending back environment map information of an entire parking place is transmitted when the vehicle 200 has moved to an entrance of the parking place. In addition, when the photography environment changes, for example, when the sun sets while in a parking place, a request for sending back the environment map information again may be issued.
The position estimator 105 compares the feature point information included in the second environment map information received from the information storing apparatus 300 with the feature point information included in the first environment map information to estimate the position of the vehicle 200 (S107 in
Based on instructions from the communication controller 104, the communicator 106 transmits/receives various types of information and requests to/from the information storing apparatus 300 through the network 400.
Based on instructions from the controller 110, the storage 107 associates the photography environment information with the feature point information in images and stores the associated information as environment map information. Based on instructions from the controller 110, the storage 107 stores various programs installed in the moving-object position estimating terminal apparatus 100, data generated by executing the programs, and so on.
Based on instructions from the controller 110, the input unit 108 receives various instructions for the moving-object position estimating terminal apparatus 100.
Based on instructions from the controller 110, the output unit 109 displays an internal state of the moving-object position estimating terminal apparatus 100.
The controller 110 executes the various programs stored in the storage 107 to control the entire moving-object position estimating terminal apparatus 100. The image acquirer 101, the photography environment information obtainer 102, the feature point information obtainer 103, the communication controller 104, and the position estimator 105 have been described above as being independent from the controller 110, at least one of the functions of the image acquirer 101, the photography environment information obtainer 102, the feature point information obtainer 103, the communication controller 104, and the position estimator 105 may be included in the controller 110.
The camera 101 is not particularly limiting, as long as it is capable of acquiring images by photographing the surroundings of the vehicle 200. The camera 101 is, for example, a distance-measuring camera, such as a stereo camera or a monocular camera that is capable of measuring a distance, and may be implemented by a product on the market. The camera 101 may or may not outhave the distance-measuring function and may be a general camera. In the case of a general camera, when the amount of movement when the vehicle 200 on which the camera 101 is mounted moves is known, it is possible to determine the distance from the camera 101 to a stationary object by using the principle of motion stereo utilizing the fact that motion parallax occurs between a plurality of images acquired by the same camera at different time points.
The communicator 106 is not particularly limiting, as long as it is communicably connected to a communicator 301 in the information storing apparatus 300 illustrated in
The storage 107 may be appropriately selected depending on the purpose, as long as it is capable of storing various types of information. The storage 107 may be, for example, a solid-state drive or a hard-disk drive, may be a portable storage device, such as a compact disk (CD) drive, a digital versatile disc (DVD) drive, or a Blu-ray® disc (BD) drive, or may be a part of a cloud that is a group of computers on a network.
The input unit 108 is not particularly limiting, as long as it is capable of receiving various requests for the moving-object position estimating terminal apparatus 100 and may be implemented by a known output unit as appropriate. Examples of the input unit 108 include a keyboard, a mouse, a touch panel, and a microphone.
The output unit 109 is not particularly limiting and may be implemented by a known output unit as appropriate. Examples of the output unit 109 include a display and a speaker.
One example of the controller 110 is a central processing unit (CPU). A processor that executes software is hardware.
The ROM 111 stores therein various programs, data, and so on that the controller 110 uses to execute various programs stored in the storage 107. Specifically, the ROM 111 stores boot programs, such as a Basic Input/Output System (BIOS) and an extensible firmware interface (EFI) therein.
The RAM 112 is a primary storage device and serves as a work area in which the various programs stored in the storage 107 are loaded when they are executed by the controller 110. Examples of the RAM 112 include a dynamic random-access memory (DRAM) and a static random-access memory (SRAM).
Based on an instruction from the controller 306, the communicator 301 receives, from the moving-object position estimating terminal apparatus 100, a request for retrieving and sending back the second environment map information corresponding to the photography environment information in the first environment map information (S201 in
The storing unit 302 stores, in an environment map information database 303a in the storage 303, the first environment map information received from the moving-object position estimating terminal apparatus 100 (S204 in
The storage 303 includes the environment map information DB 303a, and stores therein various programs installed in the moving-object position estimating terminal apparatus 100, data generated by executing the programs, and so on, based on instructions from the controller 306.
The “position information of feature points” may be information of absolute position coordinates on the earth which are defined by latitudes, longitudes, and altitudes. The “features” may be information of features obtained by, for example, a scheme, such as SURF or RIFF.
The “photography date and time” may be information of the date and time when the camera 101 acquired an image by photographing the surroundings of the vehicle 200. The “photography position information” may be the position of the camera 101 when an image from which the feature point information was obtained was acquired, and is defined by, for example, the values of a latitude, a longitude, and an altitude. The “photography position information” is represented by relative position coordinates calculated based on the position information of the camera 101 and the position information of the vehicle 200 and indicates a positional relationship between a feature point and the vehicle 200 in three-dimensional coordinates. In the present embodiment, the “brightness information of photographic object” is information obtained as a numerical value of 0 to 100 based on the luminance of an acquired image. In the present embodiment, the “weather information” is information obtained from external weather information and information or weather in the surroundings of a feature point at the time of image acquisition. One of items “sunny”, “cloudy”, “rainy”, and “snowy” is selected based on the external weather information. Although the photography environment information has been described above as being the information of the items “photography date and time, photography position information, brightness information of photographic subject, and weather information”, the present disclosure is not limited thereto. Example of the photography environment information include a luminance average value of an entire image, luminance histogram of an entire image, and color histogram of an entire image.
The input unit 304 receives various requests from a user in accordance with instructions from the controller 306.
The output unit 305 outputs images, sound, and so on to be used by the user, in accordance with instructions from the controller 306.
The controller 306 retrieves the second environment map information corresponding to the photography environment information in the first environment map information. For example, the controller 306 retrieves the second environment map information for an image acquired in a photography environment that is the same as or similar to a photography environment where an image corresponding to the first environment map information was obtained. A method for determining whether or not the photography environment information in the first environment map information and the photography environment information in the second environment map information are the same or similar to each other is, for example, a method for calculating the degree of similarity based on pieces of information included in the photography environment information and determining that they are similar to each other when the degree of similarity of the photography environment information is higher than or equal to a predetermined value. The controller 306 executes various programs stored in the storage 303 to control the entire moving-object position estimating terminal apparatus 100. Although the storing unit 302 and the controller 306 have been described above as being independent from each other, the controller 110 may have the functions of the storing unit 302.
The communicator 301 is not particularly limiting, as long as it is communicably connected to the communicator 106 in the moving-object position estimating terminal apparatus 100 illustrated in
The storage 303 may be appropriately selected depending on the purpose, as long as it is capable of storing various types of information. The storage 303 may be, for example, a solid-state drive or a hard-disk drive, may be a portable storage device, such as a CD drive, a DVD drive, or a BD drive, or may be a part of a cloud that is a group of computers on a network.
The input unit 304 is not particularly limiting, as long as it is capable of receiving various requests from a user and may be implemented by a known input unit as appropriate. Examples of the input unit 304 include a touch panel and a microphone.
The output unit 305 is not particularly limiting and may be implemented by a known output unit as appropriate. Examples of the output unit 305 include a display and a speaker.
One example of the controller 306 is a CPU.
The ROM 307 stores therein various programs, data, and so on that the controller 306 uses to execute the various programs stored in the storage 303. Specifically, the ROM 307 stores a boot program, such as a BIOS or an EFI, therein.
The RAM 308 is a primary storage device and serves as a work area in which the various programs stored in the storage 303 are loaded when they are executed by the controller 306. Examples of the RAM 308 include a DRAM and an SRAM.
In operation S101, the image acquirer 101 acquires an image by photographing the surroundings of the vehicle 200 by using a camera. Thereafter, the process proceeds to S102.
In operation S102, the photography environment information obtainer 102 obtains photography environment information of the surroundings of the vehicle 200 in synchronization with the image acquirer 101. Thereafter, the process proceeds to S103. Photography environment information other than position information may be determined based on an image of the surroundings of the vehicle 200, the image being acquired by the image acquirer 101, or based on information obtained externally. The photography environment information of the surroundings of the vehicle 200 means photography environment information of the surroundings of feature points in an image acquired by photographing the surroundings of the vehicle 200.
In operation S103, the feature point information obtainer 103 obtains feature point information in the image acquired by the image acquirer 101. Thereafter, the process proceeds to S104.
In operation S104, in accordance with an instruction from the controller 110, the photography environment information in the surroundings of the feature points and the feature point information in the image are associated with each other and are stored in the storage 107 as environment map information.
In operation S105, the controller 110 determines whether or not the second environment map information corresponding to the photography environment information in the first environment map information has been received from the information storing apparatus 300. If the controller 110 determines that the second environment map information has been received from the information storing apparatus 300, the process proceeds to S107, and if the controller 110 determines that the second environment map information has not been received from the information storing apparatus 300, the process proceeds to S106.
In operation S106, the communication controller 104 performs control on transmitting, to the information storing apparatus 300, a request for retrieving and sending back the second environment map information corresponding to the photography environment information obtained in S102 and on receiving the second environment map information sent back from the information storing apparatus 300. Thereafter, the process proceeds to S107.
In operation S107, the position estimator 105 compares feature point information included in the second environment map information received from the information storing apparatus 300 with feature point information included in the first environment map information obtained in S103 to estimate the position of the vehicle 200. Thereafter, the process proceeds to S108.
In operation S108, the communication controller 104 performs control on transmitting the first environment map information for storage in the information storing apparatus 300. Thereafter, the process proceeds to S109.
In operation S109, the controller 110 determines whether or not the processing is to be continued. If the controller 110 determines that the processing is to be continued, the process returns to S101, and if the controller 110 determines that the processing is not to be continued, the processing ends.
In operation S201, the controller 306 determines whether or not the communicator 301 has received, from the communication controller 104 in the moving-object position estimating terminal apparatus 100, a request for sending back the second environment map information corresponding to the photography environment information in the first environment map information. If the controller 306 determines that the communicator 301 has received the request for sending back the second environment map information, the process proceeds to S202, and if the controller 306 determines that the communicator 301 has not received the request, the process proceeds to S203.
In operation S202, the controller 306 instructs the communicator 301 to send back the second environment map information requested by the moving-object position estimating terminal apparatus 100. Thereafter, the process proceeds to S203.
In operation S203, the controller 306 determines whether or not the communicator 301 has received the first environment map information from the moving-object position estimating terminal apparatus 100. If the controller 306 determines that the communicator 301 has received the first environment map information, the process proceeds to S204, and if the controller 306 determines that the communicator 301 has not received the first environment map information, the process proceeds to S205.
In operation S204, the storing unit 302 stores, in the environment map information DB 303a in the storage 303, the first environment map information received from the moving-object position estimating terminal apparatus 100. Thereafter, the process proceeds to S205.
In operation S205, the controller 306 determines whether or not the processing is to be continued. If the controller 306 determines that the processing is to be continued, the process returns to S201, and if the controller 306 determines that the processing is not to be continued, the processing ends.
Although the moving object has been described above as being the vehicle 200, the moving object is not limited thereto and may be, for example, a robot or a drone.
All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
Number | Date | Country | Kind |
---|---|---|---|
2017-020462 | Feb 2017 | JP | national |