Patent Application
20210343143
The invention relates to an autonomous or semiautonomous robot. The invention also relates to a method and system for traffic light detection and usage.
Autonomous and semi-autonomous robots travelling on sidewalks and other pedestrian walkways are on the rise. Such robots may need to cross public roads to get to their destination. To do so safely and efficiently, such robots may need to comprise sensors configured to sense the robot's surroundings and generate a picture of the world for the robot. More particularly, WO 2017/064202 A1 discloses a method and a system for delivering a delivery to a delivery recipient by using a robot traveling partially autonomously. While the method and system may be satisfactory in some regards, they still have certain drawbacks and limitations. Generally, there is a need to optimize systems and methods relating to robots travelling partly autonomously as regards the following aspects: It is desirable that the part of the robot travelling autonomously is as large is possible, i.e., it is a desire that the robot requires as little human control as possible. Further, there is a paramount desire to have such robots travelling as safely as possible. Further still, it is desirable to reduce the time necessary for the robot to travel from a first location to a second location, e.g., to reduce energy consumption and traffic.
It is an object of the present invention to overcome or at least alleviate the shortcomings and disadvantages of the prior art. In particular, it is an object of the present invention to enable robots, or, more generally, (partly) autonomously operating vehicles to be improved as regards travelling time (and thus, reduction of traffic and energy consumption), percentage of autonomous operation, and/or safety. These objects are met by the present invention.
According to a first aspect, the present invention relates to a method comprising a data processing device predicting a time for a future state change of a first traffic light.
That is, in the first aspect of the present invention, a data processing device predicts a time for a future state change of a first traffic light before the state change actually happens. That is, the data processing device may “know” the time of the future state change before this state change actually occurs. This may be advantageous, as it may thus be possible to prepare for the future state change, i.e., to trigger actions that are beneficial to have been completed before the state change actually happens. Such a preparation for a future state change may render, e.g., the operation of a mobile robot (which may comprise the data processing device) particularly safe and efficient.
The method may comprise: capturing a plurality of images of the first traffic light; and the data processing device determining occurring states of the first traffic light based on the plurality of images of the first traffic light and the data processing device determining a time of an occurring state change of the first traffic light based on the determined occurred states of the first traffic light; wherein the time of the occurring state change of the first traffic light is taken into account for predicting the time for the future state change of the first traffic light.
That is, in embodiments of the present invention, a differentiation is made between future state changes and occurring state changes. A future state change of a traffic light is a state change that did not yet happen, while an occurring state change is one that occurs (or already has occurred).
Embodiments of the present technology thus utilize occurring state changes to predict future state changes. To do that, images of the first traffic light may be captured, and a state change (e.g., from green to red) may be determined on these images. That is, a data processing device may analyze these images and determine an occurring state change. For example, the data processing device may analyze the images and arrive at the finding that a state change from green to red occurred at a time of 12:13:21. Thus, a time for an occurring state change may be determined by the data processing device.
This time of the occurring state change may then be used to predicting the time of the future state change.
That is, the first traffic light may be observed and it may be determined when a state change has occurred. Based on this state change, the data processing device may then predict the time for a future state change. For example, the data processing device may have access to data relating to rules of the state change. In a very simple example, such a rule could be “the traffic light turns from red to green 60 seconds after turning from green to red”. However, it will be understood that the rules may also be more sophisticated, and, e.g., include other parameters (such as a day of the week and a time of the day, as the rules may also be time dependent). In any case, a future state change of the first traffic light may be determined by capturing images of the first traffic light and by determining occurring an occurring state change of the first traffic light in these images.
The method may comprise capturing a plurality of images of a second traffic light; and the data processing device determining occurring states of the second traffic light based on the plurality of images of the second traffic light and the data processing device determining a time of an occurring state change of the second traffic light based on the determined occurred states of the second traffic light; wherein the time of the occurring state change of the second traffic light is taken into account for predicting the time for the future state change of the first traffic light.
That is, not only the occurring state changes of the first traffic light may be taken into account for predicting the time for the future state change of the first traffic light, but also occurring state changes of other traffic lights.
The first traffic light may be located on a first crossing.
The second traffic light may be located on the first crossing.
Alternatively, the second traffic light may not be located on the first crossing.
That is, the second traffic light may be located on the crossing where the first traffic light is located or may be located at a different crossing. For example, in cases where the second traffic light is located on the same crossing where the first traffic light is located, the first and second traffic light may be synchronized with a time delay. As a mere example, the first traffic light may be a traffic light at a pedestrian crossing and the second traffic light may be a traffic light of a road perpendicular to the pedestrian crossing. That is, when the first traffic light is green (and thus gives way to the pedestrians), the second traffic light is red and prohibits vehicles from driving, and vice versa. In such instances, there may typically be a time delay between the traffic lights to ensure that the intersection is empty before giving way to the other traffic participants. As a mere example, the second traffic light (governing the vehicles) may turn red and 10 s afterwards, the first traffic light may turn green. Again, the data processing device may have access to data comprising information relating to such a rule, may capture images of the second traffic light and may, when the second traffic light turns red, predict that the first traffic light will turn green 10 s afterwards.
Furthermore, also in instances where the second traffic light is not located on the first crossing, the first traffic light and the second traffic light may be synchronized. In some instances, traffic lights on subsequent crossings are phased with respect to one another, such that, e.g., a traffic light on an initial crossing is green between the time of 0 s and the time of 20 s, and a further traffic light on a subsequent crossing is green between the time of 15 s and the time of 35 s, i.e., the traffic light on the subsequent crossing changes its state always 15 s after the state change of the traffic light on the initial crossing. Again, the data processing device may have access to data indicating such rules and may thus predict a time for a future state change based on an occurring state change.
The method may comprise: capturing a plurality of images of a third traffic light; and the data processing device determining occurring states of the third traffic light based on the plurality of images of the third traffic light and the data processing device determining a time of an occurring state change of the third traffic light based on the determined occurred states of the third traffic light; wherein the time of the occurring state change of the third traffic light is taken into account for predicting the time for the future state change of the first traffic light.
The third traffic light may be located on the first crossing.
Alternatively, the third traffic light may not be located on the first crossing.
The method may comprise a first mobile robot driving, wherein the first mobile robot comprises a first camera system.
The plurality of images of the first traffic light may be captured by the first camera system.
The plurality of images of the second traffic light may be captured by the first camera system.
The plurality of images of the third traffic light may be captured by the first camera system.
The method may comprise a second mobile robot driving, wherein the second mobile robot comprises a second camera system.
The plurality of images of the second traffic light may be captured by the second camera system.
That is, in embodiments of the present technology, it may be possible that a second robot captures the images of the second traffic light, and that these images are then used to predict a time for a state change of the first traffic light, which may be relevant for the first robot.
Consider, e.g., the situation that the first and the second traffic light are not located on the same intersection, but at different intersections, which may be separated by a separation distance, e.g., exceeding 30 m. In such a scenario, the first robot may observe the first traffic light, but may be unable to observe the second traffic light. However, the first and second traffic lights may still be synchronized with one another. In such a scenario, it may be possible that the second robot observes the second traffic light and generates a notification once a state change of this traffic light has occurred. Thus, a time for a future state change of the first traffic light (relevant for the first robot) may be predicted.
The plurality of images of the third traffic light may be captured by the second camera system.
The first mobile robot may comprise at least a part of the data processing device.
An external server may comprise at least part of the data processing device.
The future state change of the first traffic light may be a state change of the traffic light from a state prohibiting the first mobile robot to cross a road to a state allowing the first mobile robot to cross the road.
The method may further comprise notifying an operator terminal of the future state change before the predicted time of the future state change.
The operator terminal may be notified 1 s to 20 s before the predicted time, preferably 1 s to 10 s before the predicted time, such as 1 s to 5 s before the predicted time.
The first mobile robot may be controlled by the operator terminal after the notification.
That is, in some embodiments, the predicted time for a future state change may be used to hand over operation of the robot to a human operator. It will be understood that partly autonomous travelling robots may travel autonomously most of the time, and only be controlled by a human operator in certain instances. These instances may be difficult or hazardous situation. When considering a robot operating at a sidewalk, it will be understood that the instances when the robot has to cross a road may generally be more hazardous than the mere driving on a sidewalk. Thus, it may advantageous to have a human operator controlling the robot while the robot crosses the road. However, it will further be understood that it is generally be desirable to reduce the percentage of time the robot is controlled by a human operator. Thus, one human operator may be responsible for more robot than would be the case when having a high percentage of the time where the human operator controls an individual robot.
The present technology may be used to facilitate that. More particularly, the robot may travel to a traffic light at a road the robot intends to cross. When the robot arrives, the traffic light may be red, for example. That is, the robot may not be allowed to cross the respective road and may wait at the road crossing. During this time, the robot may be stationary, which may render this situation relatively safe. Thus, during the waiting time, it may be superfluous that a human operator controls the robot.
However, once the traffic light turns green and the robot crosses the road, it may be desirable that the robot be controlled by a human operator. Thus, it may be advantageous that the human operator, and more particularly, the terminal used by the human operator, is notified when the traffic light turns green. However, it may even be more advantageous that the operator terminal is notified shortly before the state change actually occurs. Thus, the human operator may assume control of the robot shortly before the traffic light turns green. This may give the human operator the possibility to establish a full understanding of the situation the robot is presently in shortly before the traffic light turns green. Thus, once the traffic light turns green, the operator may promptly trigger the robot to cross the street. Thus, the presently discussed embodiment allows for a very efficient and safe operation of the robot.
A route of the first mobile robot may be altered in response to the predicted time for a future state change.
The method may comprise determining a first pattern of time delays between occurring state changes of the first traffic light based on the plurality of images of the first traffic light; wherein the first pattern is taken into account for predicting the time for the future state change of the first traffic light.
The first pattern may be time dependent.
In other words, the first pattern may be different for different times of the day and/or for different days of the week.
The method may comprise determining a second pattern of time delays between an occurring state change of the first traffic light and an occurring state change of the second traffic light; wherein the second pattern is taken into account for predicting the time for the future state change of the first traffic light.
The second pattern may be time dependent.
In other words, the second pattern may be different for different times of the day and/or for different days of the week.
The step of determining the second pattern may be based on synchronized images of the first traffic light and the second traffic light.
The method may comprise determining a third pattern of time delays between an occurring state change of the first traffic light and an occurring state change of the third traffic light; wherein the third pattern is taken into account for predicting the time for the future state change of the first traffic light.
The third pattern may be time dependent.
In other words, the third pattern may be different for different times of the day and/or for different days of the week
The step of determining the third pattern may be based on synchronized images of the first traffic light and the third traffic light.
The method may further comprise the data processing device determining a state of an additional traffic light based on the determined occurring states of the first traffic light.
The additional traffic light may be a traffic light allowing traffic participants to turn into the road.
That is, from the state of traffic lights (e.g., from the determined states of the first traffic light), embodiments of the present technology can predict when cars can turn into the road to be crossed by the robot. For example, it may be possible that the traffic lights (for pedestrians and the robot to cross a pedestrian crossing on the one hand, and cars to turn into the road the robot intends to cross) are synchronized, but that the cars can only turn into the road for a limited amount of time. E.g., in some crossings, it may be possible that cars can make a left turn only in the beginning or only in the end. That is, the Our full green cycle for a pedestrian crossing does not mean that it will be all the time the same for all the different cars. That is, in embodiments of the present technology, it might be possible to predict that in that particular crossing from the moment it gets green: cars can come from left only at first 30 s (or that they not can come from left on first 30 s, depending on the cycles of the traffic light). That is, embodiments of the present invention may utilize the occurring states of the first traffic light (e.g., the traffic light governing the way of right for the robot) to determine the state of other traffic lights.
In some instances, the pedestrian green has different cycles, e.g., one green cycle of the pedestrian cycle might last for 2 minutes but another one for 6 minutes. Then using other traffic lights would be helpful. That is, in such embodiments it may also be possible that the robot observes the additional traffic light to determine its state. However, it may also be possible that the robot determines the state of the other traffic light by using the occurring states of the first traffic light for differing cycles of the first traffic light.
The method may further comprise the data processing device assigning the first traffic light to a mapped traffic light.
The method may further comprise the data processing device assigning the second traffic light to a mapped traffic light.
The method may further comprise the data processing device assigning the third traffic light to a mapped traffic light.
That is, in embodiments of the present invention, there may be captures images of the first, second, and/or third traffic light. E.g., to predict the future state changes, the captured images have to be assigned to mapped traffic lights, for which the cycles are known. Thus, in embodiments of the present invention, after capturing the images of the traffic light, this traffic light is assigned to a mapped traffic light, i.e., it is determined to which mapped traffic light (which operates with certain rules) a captured traffic light and its occurring state changes belong.
Generally, it will be understood that the first aspect of the present invention relates to a method (and a corresponding system) for detection and use the states of traffic lights. This aspect can be used to traverse roads, more particularly for a mobile robot to safely traverse car roads.
A further aspect of the present invention may also relate to a method comprising a data processing device generating a map of traffic lights, wherein the map of traffic lights comprises a location of at least one localization traffic light.
It should be understood that the term “localization traffic light” is used to denote a traffic light that is localized on the map. In other words, the term “localization traffic light” is used to denote that a traffic light is used in the map creating method. That is, any traffic light whose location is on the map (or is intended to be put on the map) may be referred to as a “localization traffic light”.
Generally, it should be understood that some embodiments relating to generating a map of traffic lights start with an initial or “raw” map not including the traffic lights (but other features, such as roads) and supplement such an initial or “raw” map with the location of the traffic lights. That is, in such embodiments, e.g., the robot may have access to such an initial or “raw” map, may localize itself on the map and may thus supplement the map (e.g., based on captured images) with locations of the traffic lights.
The method may comprise capturing a plurality of images of the at least one localization traffic light; determining a capture location where at least one image from the plurality of images of the at least one localization traffic light is captured; and determining the location of the at least one localization traffic light based on the capture location of the at least one image of the at least one localization traffic light.
It should be understood that the capture location denotes the location where the respective image has been captured. In embodiments of the present invention, the images are captured by a camera system comprised in the mobile robot. Thus, a location of the mobile robot while capturing the image is the capture location. To determine the capture location and thus determine the location of the at least one localization traffic light, it is advantageous that the robot has access to a precise map and may localize itself utilizing such a map. For example, the robot may localize itself using the technology described in WO 2017/076929 A1, incorporated herein by reference in its entirety. Thus, the capture location may be determined.
The step of capturing a plurality of images of the at least one localization traffic light may comprise capturing a plurality of images; and the data processing device determining the presence of at least one localization traffic light on each of the plurality of captured images.
The step of determining the presence of at least one localization traffic light on each of the plurality of captured images may comprise utilizing a machine learning algorithm, such as a neural network algorithm.
The machine learning algorithm may be trained using a plurality of images of traffic lights.
The step of capturing a plurality of images of the at least one localization traffic light may comprise capturing a plurality of images of at least one road crossing.
The method may comprise the data processing device determining the location of the at least one localization traffic light on at least one image; wherein the location of at least one localization traffic light on at least one image is taken into account for determining the location of the at least one localization traffic light.
The map of traffic lights may be generated by adding the location of at least one localization traffic light to an initial map.
The map of traffic lights may comprise information related to the location of roads and/or buildings, such as, information regarding the relative position of roads and/or buildings to the location of at least one localization traffic light.
The method may comprise adding to the map of traffic lights a respective pattern of time delays between occurring state changes of the at least one localization traffic light.
The pattern of time delays between occurring state changes of the at least one localization traffic light may be added manually by a human operator.
The step of adding to the map of traffic lights a respective pattern of time delays between occurring state changes of a localization traffic light may comprise: capturing a plurality of images of the localization traffic light; the data processing device determining occurring states of the localization traffic light based on the plurality of images of the localization traffic light; and the data processing device determining a time of an occurring state change of the localization traffic light based on the determined occurred states of the localization traffic light; wherein the time of an occurring state change of the localization traffic light is taken into account for determining the pattern of time delays between occurring state changes of the localization traffic light.
The step of the data processing device determining occurring states of the localization traffic light based on the plurality of images of the localization traffic light may comprise at least one of utilizing a brightness thresholding image processing algorithm and utilizing a color identification image processing algorithm.
The method may comprise determining patterns of time delays between occurring state changes of two or more localization traffic lights based on the respective pattern of time delays between occurring state changes of the two or more localization traffic lights; and adding to the map of traffic lights the patterns of time delays between occurring state changes of two or more localization traffic lights.
The method may comprise a mobile robot driving, wherein the mobile robot comprises a camera system.
The plurality of images may be captured by the camera system of the mobile robot. The mobile robots may comprise at least a part of the data processing device. An external server may comprise at least part of the data processing device.
The method may further comprise annotating the at least one localization traffic light according to a type of road crossing.
Furthermore, the above discussed method relating to predicting a future state change of a traffic light and the map creation can also be combined. That is, a further aspect of the present invention relates to a method of generating and utilizing a map of traffic lights, wherein the method comprises the method relating to the map creation as discussed above and the method relating to predicting a future state change as discussed above.
The at least one localization traffic light may comprise the first traffic light of the predicting method.
The method may comprise adding to the map of traffic lights the location of the first traffic light.
That is, the combined method may comprise to first generate a map comprising the location of the first traffic light and then utilizing this map to predict a future state change of this first traffic light.
The at least one localization traffic light may comprise the second traffic light of the predicting method.
The method may comprise adding to the map of traffic lights the location of the second traffic light.
The at least one localization traffic light may comprise the third traffic light of the predicting method.
The method may comprise adding to the map of traffic lights the location of the third traffic light.
That is, the combined method may also comprise to generate a map comprising the location of the second and/or third traffic light and then to utilize this map to predict a future state change of the first traffic light.
The method may comprise adding the first pattern to the map.
The method may comprise adding the second pattern to the map.
The method may comprise adding the third pattern to the map.
That is, in the combined method, any of the above discussed patterns (the first pattern relating to time delays in state changes of the first traffic light, the second pattern relating to time delays of state changes of the second traffic light with respect to the first traffic light, and the third pattern relating to time delays in state changes of the third traffic light with respect to the first traffic light) may also be added to the map and may then subsequently be used to determine a future state change of the first traffic light.
In this method, data comprised in the map of traffic lights may be used to predict a time for a future state change of the first traffic light.
In a still further aspect, the present invention also relates to a system configured to perform at least one of the methods discussed above.
The system may comprise the data processing device.
The data processing device may be configured to process at least one image of a traffic light for detecting at least one occurring state of the traffic light.
The data processing device may be configured to process at least two images of a traffic light for detecting an occurring state change of the traffic light.
The data processing device may be configured to detect a first occurring state of the traffic light on a first image from the at least two images of the traffic light; detect a second occurring state of the traffic light on a second image from the at least two images of the traffic light; determine an occurring state change based on the detected first and second occurring states; and determine a time of the occurring state change based on capturing times of the at least two images.
The data processing device may be configured to process at least one image for detecting at least one localization traffic light on the at least one image.
The data processing device may be configured to execute a machine learning algorithm for detecting at least one localization traffic light on the at least one image.
The data processing device may be configured to determine a capturing location of at least one image comprising a localization traffic light; and determine a location of a localization traffic light based on the location of the at least one image comprising a localization traffic light.
The system may comprise a mobile robot comprising a camera system.
The at least one image may be captured by the camera system.
The at least two images may be captured by the camera system.
The mobile robot may comprise a part of the data processing device.
The system may comprise an external server and the external server may comprise part of the data processing device.
The system may comprise an operator terminal.
The data processing device may be configured to notify the operator terminal of the future state change before the predicted time of the future state change, such as, 1 s to 20 s before the predicted time, preferably 1 s to 10 s before the predicted time, such as 1 s to 5 s before the predicted time.
The operator terminal may be configured to control the mobile robot after the notification.
The system may comprise the first mobile robot.
The system may comprise the second mobile robot.
That is, the present invention may relate to building (as in the map creation realization) and/or using a map of traffic lights using existing high resolution footage. The high resolution images may be generated, e.g., from “incident reports” that happen near intersections. Such high resolution images may be automatically collected in order to get high resolution images towards the same intersection from different locations to facilitate the mapping of traffic lights.
As discussed before, in embodiments of the present invention, the robot is switched from autonomous mode to operator control right before the traffic light turns green for the robot, which may require the prediction of a future state change of the traffic light.
The present invention is also defined by the following numbered embodiments.
Below, predicting method embodiments will be discussed. These embodiments are abbreviated by the letter “M” followed by a number. Whenever reference is herein made to predicting method embodiments, these embodiments are meant. The respective method will also be referred to as a predicting method.
That is, in embodiments of the present invention, a differentiation is made between future state changes and occurring state changes. A future state change of a traffic light is a state change that did not yet happen, while an occurring state change is one that occurs (or already has occurred).
Embodiments of the present technology thus utilize occurring state changes to predict future state changes. To do that, images of the first traffic light may be captured, and a state change (e.g., from green to red) may be determined on these images. That is, a data processing device may analyze these images and determine an occurring state change. For example, the data processing device may analyze the images and arrive at the finding that a state change from green to red occurred at a time of 12:13:21. Thus, a time for an occurring state change may be determined by the data processing device.
This time of the occurring state change may then be used to predicting the time of the future state change.
In other words, the first pattern may be different for different times of the day and/or for different days of the week.
In other words, the second pattern may be different for different times of the day and/or for different days of the week.
In other words, the third pattern may be different for different times of the day and/or for different days of the week
Below, map creation method embodiments will be discussed. These embodiments are abbreviated by the letter “A” followed by a number. Whenever reference is herein made to map creation method embodiments, these embodiments are meant. The respective method will also be referred to as a map creation method.
It should be understood that the term “localization traffic light” is used to denote a traffic light that is localized on the map. In other words, the term “localization traffic light” is used to denote that a traffic light is used in the map creating method. That is, any traffic light whose location is on the map (or is intended to be put on the map) may be referred to as a “localization traffic light”.
A11. The method according to the preceding embodiment, wherein the pattern of time delays between occurring state changes of the at least one localization traffic light is added manually by a human operator.
A12. The method according to the penultimate embodiment, wherein the step of adding to the map of traffic lights a respective pattern of time delays between occurring state changes of a localization traffic light comprises
A18. The method according to any of the preceding map creation embodiments, wherein an external server comprises at least part of the data processing device.
Below, embodiments relating to a method of generating and utilizing a map of traffic lights will be discussed. These embodiments will also be referred to as generate and use embodiments and will be abbreviated by the letter B followed by a number. Whenever reference is herein made to generate and use embodiments (or GU embodiments), these embodiments are meant. The method will also be referred to as a GU method.
Below, system embodiments will be discussed. These embodiments are abbreviated by the letter “S” followed by a number. Whenever reference is herein made to system embodiments, these embodiments are meant.
In the following, exemplary embodiments of the invention will be described, referring to the figures. These examples are provided to provide further understanding of the invention, without limiting its scope.
In the following description, a series of features and/or steps are described. The skilled person will appreciate that unless required by the context, the order of features and steps is not critical for the resulting configuration and its effect. Further, it will be apparent to the skilled person that irrespective of the order of features and steps, time delays between steps can be present between some or all of the described steps.
The description of the figures first provides a general overview of embodiments of the present invention, before providing further details of more specific embodiments, features and steps of the exemplary embodiments of the present invention.
In general terms, embodiments of the present invention relate to methods and systems relating to a mobile robot 10 travelling on sidewalks 110 (see
The robot 10 may travel autonomously (i.e., without a human operator controlling it) or semi-autonomously (i.e., with a human operator controlling it part of its travelling time).
A typical scenario for human operation of the semi-autonomous robot 10 is the road crossing. It will be understood that the robot 10 crossing the road 102 may be more hazardous than the robot 10 just travelling on the sidewalk 110.
Embodiments of the present invention relate to handing over control of the robot 10 to the human operator shortly before the traffic light 410 relevant for the robot 10 crossing the road 102 switches from red to green.
In simple words, embodiments of the present technology relate to creating a map including traffic lights and patterns of their state changes, and utilizing this map with the patterns of traffic light state changes to infer a future state change of a traffic light.
Generally, embodiments of the present invention may comprise the following steps:
Initially, steps relating to creating a map of traffic lights will be described.
In such an embodiments relating to creation of a map of traffic lights, images are collected at the intersections. The images may be high-resolution images. A couple of images per square meter can be collected. The image collection is typically performed by the robots 10. A track record of where such images have been collected may also be created for checking where additional images need to be collected.
On these images, traffic lights and preferably their angle are detected. For this detection, neural networks may be used. In this step, there may also be utilized a similarity score to indicate a similarity between an object on an image and a traffic light (which may correspond to the probability of a traffic light being present on the image). This detection step is typically performed on a data processing device, e.g., on a server. That is, the images can be uploaded to the data processing device and traffic lights can be detected offline on such images.
After the detection, the traffic lights may be mapped. That is, a map may be created including these traffic light. Again, this step may be performed by the data processing device (e.g., the server) and offline. It will be understood that the map is an (e.g., two-dimensional) data set comprising information relating to positions of objects (such as roads, sidewalks), the objects also including the traffic lights. Further, it will be understood when creating the map of traffic lights, it may be possible to start with an initial or “raw” map comprising, e.g., the streets, and supplementing this map with the location of the traffic lights.
In an optional step, the traffic lights may be annotated. The annotation may differentiate the traffic lights for autonomous crossings or operator-based crossings.
In a further step, states of the traffic lights may be recorded. This may be done by the robots 10. Exemplary states may include permitting to cross (usually indicated by a green light), prohibiting a crossing (usually indicated by a red light), and optionally also an intermediate state (usually indicated by an orange light).
The recorded state changes for the traffic lights can also be used to build a model to predict future state changes of traffic lights. That is, the recorded state changes (i.e., information relating to the state of the traffic light at different times) may be provided to the data processing device, and the data processing device may create a prediction model for future state changes. A very simple exemplary prediction model may be that a traffic light turns green 20 seconds after it turns red. Other prediction models may also take into account the state changes of other traffic lights (on the same intersection or on different intersections).
Further embodiments of the present technology relate to predicting future state changes of traffic lights, e.g., by utilizing the map of traffic lights that has been created.
That is, in embodiments of the present technology, the created map, or generally, the model for the state changes of the traffic lights may then be used to predict a future state change of a traffic light, e.g., the traffic light 410 relevant for the robot 10 (see
Further details of the present technology will be discussed below.
Embodiments of the present invention relate to methods and systems comprising a mobile robot 10 (see
Some embodiments of the present invention concern building and/or using a map of traffic lights. Said map may comprise information regarding the location and position of the traffic lights, such as the relative position between the traffic lights of an area of operation. Said map may also comprise information regarding the location of roads, such as the relative position of roads with respect to each other. The map may also comprise information regarding the roads and the traffic lights, such as, the relative position of the traffic lights relative to the roads. Said map may also comprise information regarding the pattern on the traffic light state changes, such as the synchronization between the traffic lights. The map may be created based on images captured near at least one intersection. Such images may be captured by one or more mobile robots 10.
In some embodiments, a map can be obtained by the mobile robot 10, such as OpenStreetMap by the OpenStreetMap Foundation, Sutton Coldfield, UK. This map can comprise information about the roads, buildings and other physical features of the environment. Such a map can be used by the mobile robot 10 for localization and/or navigation. Further, the map can facilitate the creation of a map of traffic lights. For example, the mobile robot 10 can use the map for localization and navigation, and take images that can then subsequently used for creating the map of the traffic lights (these aspects may thus relate to the creation of the map of traffic lights). In some embodiments, an initial map (i.e. the map comprising information about roads, buildings and other physical features of the environment) can be extended with information relating to traffic lights.
Further, such a map of traffic lights can then also be used to predict a future state change of traffic lights. In embodiments relating thereto, the mobile robot can take images of traffic lights that can be used for locating traffic lights and/or other features related to the traffic lights (such as the patterns on the traffic lights state changes).
In some embodiments, the mobile robot can have access to a memory device that can comprise images or videos of roads, particularly of places in the roads comprising traffic lights, such as intersections or road crossings (for example zebra crossings). It is common and probable that the intersections (i.e. road intersections) and the road crossings are equipped with traffic lights. Thus, images or videos at an intersection or at road crossings are advantageous in the present invention.
Further, it is advantageous that more than one image is taken at an intersection and such images are taken in consecutive steps of time, such as, for example every 5 ms, 10 ms, 100 ms, or 1 s. For example, the images can be part of a video (each frame of the video can be an image). In another example, the images are taken by at least one camera configured in burst mode (also referred continues shooting mode), wherein more than one image are captured successively one after another. The speed at which successive images are captured can depend on the camera or can be a programmable parameter. In yet another example, the images are taken by at least one camera that records a video at an intersection. Thus, the images are realized as video frames. The videos can comprise different frame rates (i.e. number of still images per unit of time), such as 1, 6, 8, 24, 60, 120 or more frames per second.
To put it simply, it can be advantageous that multiple successive images are taken for a particular scene wherein a time delay between the images is present. A small time delay, meaning a high frame rate, can allow for a better and more accurate detection of the movements in the images. For example, if the frame rate is 1 frame per minute and the movement of an object captured by the images lasts only 1 second, then it can be hard to infer an accurate time during which the movement happened. With a higher frame rate, such as 60 frames per second, in the above example, a more accurate inference about the timing of the movement can be made. However, higher frame rate means that more images need to be stored on the memory, thus more memory need to be allocated for such a purpose.
In the below, further details are provided of how a map comprising traffic lights can be created.
As stated in the above embodiments, the mobile robot 10 can purposefully (i.e. for the purpose of creating a map of traffic lights and/or for detecting state change patterns of such traffic lights) capture images at an intersection. That is, the mobile robot can localize itself and can deduce that it is located near or at an intersection, such as less than 50 m away from the center of the intersection. Further, the mobile robot can capture images or videos at the intersection and can store such images or videos in a memory. In some embodiments, the mobile robot 10 can use existing databases of images. For example, the mobile robot can have access to a memory comprising images of the environment it is operating. For example, the images may have been stored in the memory for the purpose of using them for visual based localization. In another example, the images may have been stored in the memory by the mobile robot during a hazardous situation, such as during a malfunction of the robot, wherein images or videos were recorded by the mobile robot as part of the incident report. Generally, it can be any reason that the images may have been stored in the memory. Further, such images or the memory comprising such images can be shared by several mobile robots. That is, information based on the images captured by a mobile robot can also be accessed by another mobile robot. For example, the mobile robots 10 may share or have access to the memory device storing information relating to a map of the environment.
In a further step, a data processing device can process the images for detecting traffic lights on the images. This step may be performed by a server, i.e., not locally on the robot. That is, the images that the mobile robot can obtain are captured at an intersection. However, some of the images may not contain a traffic light. Thus, the images may be processed, using a data processing device, to detect if each of them comprises a traffic light as well as the position of the traffic light on the image. Once a traffic light is detected on the image and its position on the image is found, the position, location and/or orientation of the traffic light in the real world, or relative to the mobile robot's position, can be inferred. For example, the position of the traffic light relative to an object in a map can be inferred, such as, relative to the intersection, or to a road in the map. Using such information, a map of traffic lights can be created or a current map can be extended, to thus further comprise information regarding the position of the traffic lights.
The processing of the images for detecting the traffic lights on the images that the mobile robot can obtain, can be facilitated by neural network algorithms. For example, the data processing device can search in the images for patterns or shapes similar to a traffic light. For this purpose, the data processing device can use (or can be trained with) a plurality of photos of traffic lights captured in different angles or light conditions, or different types of traffic lights. In other words, the higher the diversity of traffic light images used for training the data processing device, the better the results on detecting a traffic light on a processed image. By comparing the shapes and the patterns on the image being processed with different images of the traffic lights, the data processing device can assign a probability based on the likelihood that a found pattern represent a traffic light. A threshold level can be defined for deducing if a traffic light is detected on the image or not. Thus, a map of traffic lights can be built, and/or an existing map can be supplemented with information relating to traffic lights.
In the below, further details of determining patterns of state changes of traffic lights will be explained.
A further embodiment of this invention relates to determining a timing between state changes for at least one traffic light in an intersection or more than one intersection and switching the robots from autonomous mode (i.e. without the control of a human operator) to operator control (i.e. with the control or assistance of a human operator and/or an operator terminal and/or an control interface device) right before the traffic light turns green for the robot.
Generally, it will be understood that the mobile robot 10 may travel autonomously, i.e., without a human operator controlling the robot 10. However, sometimes, the robot 10 may require help from a human operator, e.g., in hazardous situations or in situations unknown to the robot 10. In such instances, a communication connection (e.g., via a mobile network) may be established between the robot 10 and an operator terminal, e.g., a terminal (such as a computer) controlled by a human. The human operator may then enter instructions into the operator terminal (such as: stop, turn left, turn right, accelerate, decelerate), and the instructions may control the robot 10. This may allow a safe operation in hazardous situations and situations unknown to the robot 10.
In some embodiments, the images of the traffic lights can be used for inferring the timing between state changes. Thus, a plurality of images, taken at successive instances, such as every 10 ms, 100 ms, 1 s, can be used and can be advantageous to detect the pattern of the traffic light state changes. For example, the data processing device (which may be part of a server) can detect that the light of the traffic light stays red for 30 seconds and green for 10 seconds. The data processing device can infer this and can store such information on a memory. Next time the mobile robot detects the particular traffic light it “knows” the duration of the states of the traffic light. Thus, if the mobile robot detects a change from green to red, the mobile robot “knows” that after 30 seconds the traffic light will turn green again.
To put it simply, the mobile robot 10 can obtain images at an intersection, such as videos at an intersection, and based on this images, the duration of each state (i.e. the pattern on state changes) for a traffic light can be inferred. Further, the information related to the pattern of state changes can be used to make predictions about future states of the traffic light. It will be understood that these steps may be performed by a server building the map of traffic lights and equipping this map with additional information, and in particular rules that are followed by the traffic lights.
The state change patterns can also be detected between at least two traffic lights. That is, it is often the case that traffic lights are related (or synchronized) with each other. For example, a pedestrian traffic light and a car traffic light for the same road cannot be on the same time on the same state (such as on the green state), but they are synchronized such that one can be on the green state and the other on the red state. Further, a time delay can be present between the state changes. That is, if one changes from green to red after a time delay, for example 10 s, the other will change from red to green. The time delay can be present to make sure that the road is clear for the traffic participants that are allowed to pass the road by the traffic lights. To put it simply, it is often the case that traffic lights are synchronized with each other and the pattern of the traffic light state changes can be detected by the presently described technology. Such information regarding the pattern of state change between two or more traffic lights can be used to make predictions about a future state of a traffic light. For example, the mobile robot is required to traverse a road. The mobile robot 10, as any other pedestrian, can pass the road when the pedestrian traffic light for that road is in the green state. Further, the road is equipped with another traffic light that guides the vehicles to pass the road or not. As discussed, the present technology can detect the pattern of state changes between multiple traffic lights. Thus, it is “known” that when the vehicle traffic light changes from green to red, the pedestrian traffic light changes from red to green after specific time delay. Thus, once the vehicle traffic light changes from green to red and the mobile robot can detect such a change, it can be predicted that after said specific time delay the pedestrian traffic light will change from green to red.
The pattern on the traffic light state changes can also be inferred for traffic lights that belong to different roads or different intersections. Thus, based on the states of the traffic lights on a current intersection, the state of the traffic lights on another intersection can be predicted.
In some embodiments of the present invention, the mobile robot 10 uses the predicted traffic light state to execute an action. The mobile robot can be configured to require operator assistance while traversing a road, as usually traversing the road imposes a more complex or dangerous scenario. Thus, once predicting the next state change of a traffic light, the mobile robot knows when to ask for assistance. For example, the mobile robot asks for assistance from the operator a few seconds, such as 2 to 10 seconds, before the traffic light state changes.
Exemplary embodiments of the present invention will now be described with reference to
The mobile robot 10 can comprise at least one camera that can facilitate, among others, the localization and navigation of the robot 10. The cameras may be directed towards the front of the mobile robot 10 (front defined as the side of the robot towards the direction of movement of the mobile robot in a normal operation mode of the robot), wherein the cameras can provide to the mobile robot a front field of view as indicated by the bounding lines 112, 113. It should be noted that the front field of view herein refers to the smaller sector that is defined by the lines 112 and 113—i.e. area defined by the internal or smaller angle of lines 112 and 113 and the lines 112 and 113. Further, the mobile robot 10 can comprise another camera (or other cameras) directed towards the back of the mobile robot 10, wherein the cameras provide to the robot 10 a rear field of view as indicated by lines 116, 117. The mobile robot can also comprise other cameras that provide to the mobile robot 10 other fields of view.
That is, the mobile robot 10 can comprise at least one camera. The cameras can provide different fields of view to the mobile robot 10. Using the cameras, the mobile robot can capture images of its surroundings. For example, the mobile robot 10 can be configured to capture images of its surrounding to facilitate its localization and navigation based on visual localization and navigation techniques. In another example, the mobile robot 10 can be configured to capture images or record videos of its surroundings during a hazardous situation, such as during a malfunction of the robot 10, wherein said images or videos can be used as part of the incident report. While in the previous two examples the mobile robot captures images and/or videos without the direct purpose of capturing traffic lights, in yet another example, the mobile robot 10 can be configured to capture images and/or record videos of the surroundings to detect traffic lights in an environment.
The mobile robot 10 can localize itself on an initial or “raw” map. Thus, the mobile robot 10 can comprise information about its position when capturing photos. Thus, the images stored on the memory can also be associated with the respective position of the location where they were captured. This may also be referred to as the capturing location.
In other words, the mobile robot 10 during its operation can capture images of the surroundings. The images can be stored on a memory that the mobile robot 10 can have access to. Such images can be used by the mobile robot 10 to detect traffic lights and a map of traffic lights can be created using these traffic lights.
The area of interest 220 for an intersection or a road can be defined based on the topology of the road or intersection. For example, a large intersection would have a larger area of interest 220 compared to a smaller intersection.
The area of interest 220 can be used to facilitate or optimize the process of detecting traffic lights. That is, since the area of interest 220 can comprise the highest likelihood of comprising traffic lights, only images captured within such an area can be processed for detecting traffic lights. For example, in
Visual features 320 are schematically depicted in the environment in
Thus, in the depicted scenario of
In
It should be noted that
By the above means, traffic lights may be detected and may be added to a map, to thereby generating a map comprising traffic lights. Thus, a map of traffic lights may be generated.
In order to create a map comprising traffic lights, such as the one depicted in
In
With reference to
It should also be understood, that the described embodiments with reference to the figures relate to an environment comprising an intersection. It should be noted that the embodiments of the present technology can also be applied on other environments that can comprise a traffic light, such as for example, at a road crossing.
Furthermore, in
As discussed, the mobile robot 10 can comprise at least one camera. The at least one camera can provide to the mobile robot a field of view, depicted in
To be able to detect patterns on the state changes of the traffic lights of the intersection, it can be advantageous that more than one image of the traffic lights can be captured by the mobile robot. It can further be advantageous that the images are captured successively, with a small time delay between them, such as for example 5 ms to 1 s. In some embodiments, the mobile robot 10 can be configured to record videos with a frame rate between 1 to 200 frames per second.
The images (or the frames of the video) can be used to detect the patterns on the light changes of the traffic light. By processing the images, the position of the traffic light on each image can be detected. In some embodiments, the mobile robot can be configured to not change the focus and/or view of its cameras while capturing said images (or videos) which can facilitate the process of detecting the traffic lights on the images. That is, a traffic light on the first image can be detected and the position found can be used to predict the position of the same traffic light on the next image. The mobile robot can comprise or have access to a map similar or identical to the one depicted in
During or after the determination of the position of a traffic light on an image, its state can be detected, that is, it can be detected which of the lights of the traffic light is on. For example, using brightness thresholding the position of the light that is on can be determined. Then, the relative position between the light that is on and the traffic light can be used to determine which of the lights is on. That is, a bounding box (or any shape) can be defined representing the area with the highest probability of comprising the traffic light. Further, the position of the light that is on (using, for example, brightness thresholding) relative to the bounding box can be determined. Based on such information the state of the traffic light can be inferred. For example, if the light that is turned-on is positioned on the most lower section of the bounding box, this can provide a hint that the green light is on and the traffic light is on the green state. Additionally, or alternatively the color of the light that is on can be analyzed, to determine the state of the traffic light.
As discussed, the state of the traffic lights on each (or some) of the images that have be captured can be detected. Since the images can be captured in a successive manner, a traffic light state progress over time can be created.
The diagram of
With reference to
On an image of the traffic light 410 on time instance I a data processing device can determine the traffic light as being in the red state (i.e. the red light of the traffic light 410 is turned on). On an image of the traffic light 410 captured at time instance II the data processing device can determine the traffic light as being in the green state (i.e. the green light of the traffic light 410 is turned on). The data processing device can detect the state change from red to green of the traffic light 410 and can save the time of the time instance II on a memory. Proceeding on time instance III, it can be detected that the traffic light 410 stays in the green state. Also, at time instance IV and V it can be detected that the traffic light 410 stays in the green state. At time instance VI it is detected that the state of the traffic light is red, inferring that a state change from green to red happened between time instance V and time instance VI. Thus, two state transitions of the traffic light 410 are detected, the first one being from red to green and the second one being from green to red. By subtracting the time of the second transition and time of the first transition, the duration that the green light of the traffic light 410 stays on can be determined. That is, the difference between the time of the time instance VI from the time of the time instance II provides an approximate for the time that the traffic light 410 stayed on the green state. By continuing the observation further, that is beyond time instance VII until at least one other transition from red to green is observed the duration that the traffic light 410 stays in the red state can be determined. Or by having started the observation before time instance I, since at least the first transition from green to red before time instance I is observed, the duration that the traffic light 410 stays in the red state can be determined. In this manner, the duration of each of the states of the traffic lights can be detected and thus the pattern of state changes of a traffic light can be determined.
In a similar manner the pattern of state changes of traffic light 420 can be determined by the data processing device.
In the following, the determination of the pattern of state changes between two or more traffic lights will be described. It is often the case that traffic lights are synchronized with each other. For instance, in
Thus, for a better guidance of movements of the traffic participants the traffic lights 410 and 420 can be synchronized with each other. That is, the traffic light 410 and 420 have to be on opposite states, i.e. cannot be both on the green state or both on the red state (except for some safety interval). In some instances, the traffic lights 410 and 420 can be synchronized such that when one of them changes state from red to green the other instantly changes from green to red. However, it is often the case with such traffic lights that a time delay is present between the state change of the first traffic light to the state change of the second traffic light (for example, to make sure that the road is clear for the traffic participants that will be allowed crossing).
It can be seen from
In the above description, the traffic lights transitioned only between two states: red state and green state, while the orange state was neglected. This was done for sake of brevity. In a similar manner the pattern of state change including the orange state can be determined.
As discussed, the pattern of state changes of one traffic light or the pattern of state changes between multiple traffic lights at one intersection (refer to
Thus, an additional manner is required that can provide the state of the traffic lights on different intersections, in order to determine the pattern of state changes between the traffic lights of such different intersections.
In some embodiment, a second mobile robot can be positioned at another intersection, for example on intersection 6. The mobile robots can be configured to both communicate with a data processing device by exchanging data through a communication channel, such as for example wirelessly though the internet. The mobile robot positioned on intersection 6 can transmit (e.g., on live time) to the data processing device the status of the traffic lights of intersection 6, while the mobile robot 10 observes the traffic lights of intersection 2 (or vice versa) and transmits data concerning the traffic lights of intersection 2 to the data processing device. Since the data processing device can comprise the status of traffic lights of both intersections, the data processing device can create a timing diagram similar to the one depicted in
In another embodiment, instead of a second mobile robot a human operator can provide the status of the traffic light on a further intersection. The human operator can determine the status of the traffic light on another intersection either by observing images or videos, such as, live images or videos, of the traffic light on another intersection or by being on the said intersection and directly observing the status of the traffic light on that intersection.
Thus, the found patterns between traffic lights of different intersections can be used to infer patterns between different intersections. For example, if the pattern between traffic lights of intersection 2 and intersection 6 is known and the pattern between intersection 6 and intersection 11 is known the pattern between intersection 2 and intersection 11 can be inferred.
In general, as described in the above, embodiments of present technology provide a manner of creating a map of traffic lights. Further, the pattern of state changes of at least one traffic light, or more than one traffic light that can belong to the same intersection or that can belong to different intersections, could be detected and inferred. In some embodiments, images taken at successive time instants and/or video recording(s) can be used.
A map comprising such traffic lights can then be used to later predict future state changes of traffic lights. That is, detecting the light patterns can then allow to predict the time of the next state change of one or more traffic lights. Knowing the next state change of a traffic light before it happens, such as 2-10 seconds ahead, can allow the mobile robot to execute particular actions. A general description of the method of predicting a state change and executing an action based on the predicted state change is provided in the schematic flowchart of
Referring to
Further, there may be provided a data processing device that can be configured to process images, for example, for the purpose of detecting and/or extracting different features or patterns from processed images. The mobile robot 10 can comprise the data processing device configured to process images (i.e. configured to execute an image processing algorithm). Thus, in a step S3 the mobile robot 10 can detect traffic lights from the images captured in step S2. More particularly, the mobile robot 10 can detect a traffic light on the captures images and can determine a location of the traffic light on the images. Based on the location of the traffic light on the images, a capture location (i.e., the location of the robot 10 while capturing the images), and optionally other parameters (such as the size of the traffic light on the image), the location of the traffic light can be determined. Thus, the detected traffic light may be assigned to a mapped traffic light. This may then also further facilitate the localization of the traffic light in the image, and the subsequent determination of the state of the traffic light.
That is, the mobile robot 10 can identify the presence and/or position of at least one traffic light or at least of the traffic light that provides (or denies) permission for crossing the road that the mobile robot 10 intends to cross. The detection of position of the traffic light (or traffic lights) on the images can allow the mobile robot to infer the position of the traffic light (or traffic lights) on a map the mobile robot 10 has access to and/or the position of the traffic light (or traffic lights) in the real world. Thus, the mobile robot can be able to identify the traffic lights that can be present and detected on the images (or videos) the mobile robot 10 can capture.
Generally, it will be understood that the robot 10 may have access to a map of traffic lights that has been created previously, e.g., by the steps described above. Further, it will be understood that the creation of the map typically is performed by a server. That is, while mobile robots 10 may collect the images of the traffic lights, the step of creating the map (e.g., adding the location of the traffic lights to an existing map), may be performed by a server. Thus, a map of traffic lights may be created by the server.
When travelling in an environment and intending to use the map (to predict future state changes), the mobile robot 10 may have access to the map. That is, the robot may access the map and may thus be able to obtain information relating to the locations of traffic lights.
This information may be used to facilitate the detection of a traffic light. By accessing the map, the robot 10 may obtain information on the location of traffic lights. When the robot then captures images at (or close to) the location of a mapped traffic light, the robot may use this information to search the images captures at such location for the presence of traffic lights. Thus, the map of traffic lights may also facilitate the live detection of traffic lights. Further, once a traffic light is found by the robot, it may be assigned to one of the mapped traffic lights. That is, for a traffic light encountered by the robot, it is determined to which mapped traffic light it corresponds.
The mobile robot 10 can further be configured to detect a state change of a traffic light based on the captured images (step S4). Again, it will be understood that the detection of such state changes is also easier when having access to the map of traffic lights and therefore being able to determine where exactly to look for such state changes. More particularly, to detect state changes of a traffic light, the mobile robot 10 can adapt its own orientation or its cameras orientation such that the traffic lights of the intersection or at least some of the traffic lights of the intersection are within the field of view of the cameras, i.e. are captured in all or at least some of the captured images. Using brightness thresholding or color identification or other similar algorithms the mobile robot can detect which of the lights of a traffic light is turned-on and thus can infer the state of the traffic light. The mobile robot 10 can infer the state of the traffic light for each of the images (or for each of the frames of the videos) that captured the traffic light. For optimizing such a process some of the images (or frames of the video) may be neglected. For example, one of every two frames of the video may be considered. Usually the higher the number of considered images the higher can be the accuracy on inferring the time the state change happened. Thus, by inferring the state of the traffic light on an image (or on the frame of a video) and by comparing such a state with the state of the traffic light on the previous image the state change can be detected.
The detection of the first state change can be advantageous as it can allow the mobile robot 10 to synchronize with the traffic light, more particularly with the internal timer of the traffic lights. That is, by simply detecting the state of the traffic light it can be hard (or even impossible) to predict the next state changes, as the mobile robot cannot know how long the traffic light was on that state (and thus cannot predict how long the traffic light will be on that state). In contrast, by observing a state change (step S4) the mobile robot can comprise more accurate information about the state of the traffic light and how long the traffic light has been on that state.
However, the information regarding how long the traffic light has been on a particular state cannot be enough for inferring an accurate prediction about the next state change of the traffic light. Further information regarding how much time the traffic light stays on that particular state can be advantageous. In other words, the pattern of state changes of the traffic light has to be known in advance before inferring the time of the next state change. In some embodiments, the state change patterns of a traffic light as described with reference to
Thus, in a step S5, the mobile robot matches the first change with a known state change pattern of the traffic light. As discussed, the mobile robot typically assigns a detected traffic light to a mapped traffic light. For the mapped traffic lights, the robot has access to information as regards their timing, i.e., rules governing their state changes. For example, the mobile robot 10 on step S4 detected that the traffic light changed its state from green to red. Further, the mobile robot “knows” (by having access to the corresponding information) that the traffic light it is observing stays on the red state for e.g. 75 seconds. Thus, the mobile robot 10 can infer that the traffic light will go to the green state 75 seconds after the first state change. In other words, based on a first state change and based on state change pattern information the mobile robot 10 can comprise or have access to, the mobile robot can predict a second state change of the traffic light, such as the next state change.
In the above steps only one traffic light was considered for predicting the state of a traffic light. However, it should be understood that more than one traffic light can be used for predicting the state of a traffic light. Furthermore, the traffic lights can be part of the same intersection or road or can belong to different intersections and roads (refer to
In the above, it has been exemplarily described that it was observed that the time of a traffic light in the red state is 75 seconds. It should be understood that this is to be regarded as a simple example. In fact, the traffic lights may not have a fixed frequency for changing their states. Instead, the timing of the state change may depend, inter alia, on the weekday, on the time of the day, and/or the traffic density. It will be understood that such complex patterns may also be taken into consideration by the present technology. That is, independent of the actual dependencies of the state changes, it may be possible to predict the next stage change, e.g., based on historical data of the subject traffic light and optionally also of other traffic lights.
Further, while in the above, it is said that the data processing device performed the estimations/predictions as regards the next change of a status of a subject traffic light, it should be understood that this is merely exemplary.
In a step S6 the mobile robot 10 is configured to execute an action based on the predicted traffic light second state change. Example of such actions will be provided in the following description.
In some embodiments, the mobile robot 10 is configured to drive fully autonomously on part of their traveling and on the other part of their travelling the mobile robots can drive partially autonomously and partially controlled by a human operator or fully controlled by the human operator. That is, in some cases a human operator can take control of the mobile robot and can control part of the mobile robot functionalities, such as the human operator can control the driving of the mobile robot among others. It is usually the case that the human operator takes control of the mobile robot 10 particularly in situations that impose more danger, or that are considered more dangerous, and that can be prone to possible accidents or other incidents. One such situation can be the mobile robot crossing a road.
It is often the case that when the mobile robot is in a hazardous situation, such as when crossing a road, an operator takes control of the mobile robot. In U.S. Pat. No. 9,741,010 B1 “System and method for securely delivering packages to different delivery recipients”, among others, embodiments are discussed relating to a mobile robot travelling to a delivery location comprising the robot semi-autonomously and/or autonomously driving from an initial location to another location. During the semi-autonomous driving the mobile robot is controlled by an operator. This can be particularly advantageous in hazardous situations, such as the robot crossing a street with car traffic. In other words, it is often the case in the state of the art that during hazardous situations such as road crossings an operator terminal can take control of the mobile robot. That is, the operator terminal can take control of the mobile robot before the initiation of the road crossing. In such cases it is the operator that checks the traffic light status and when the green light turns on the operator controls the robot to cross the road. Thus, the attention of the operator is required not only during the road crossing but also before the road crossing for checking the traffic light status.
In some embodiments of the present technology, the mobile robot can use the predicted traffic light state change to require assistance from the operator (refer to step S6 in
Furthermore, it can be advantageous that the mobile robot notifies the operator for the next traffic light state change some moments, such as a few seconds, before the state change happens. For example, the autonomous time during road crossing can be increased if the mobile robot notifies the operator at the instant the traffic light state change happens. That is, the instant the light of the traffic light goes green the mobile robot can signal the operator to take control of the mobile robot for crossing the road. Though indeed the autonomous time is increased in such a scenario, it may cause further problems. The operator may need some time, to prepare for taking control of the mobile robot. Thus, the mobile robot cannot start passing the road instantly when the light goes green, if the mobile robot notifies the operator about the state change at the instant the state changes. This may be problematic, particularly in cases where green light stays on for short periods of time, which may result on either the mobile robot to miss the green light and thus have to wait for the next cycle, or the light goes red while the robot has not crossed the road yet, which can cause accidents or other incidents. Furthermore, since the mobile robot cannot instantly start crossing the road it may increase the total time that the mobile robot needs to travel to its destination.
Yet another problem accompanied with the case when the mobile robot does not start crossing the road instantly when the light of the traffic light goes green, is illustrated in
Put simply, on one hand the prediction of the next state change of a traffic light is used to increase the autonomous time of a mobile robot during a road crossing and thus to decrease the time required for an operator to control or assist a mobile robot during a road crossing. On the other hand, the prediction of the next state change of a traffic light is used to be able to notify the operator moments before the state change, such as 3-10 seconds before the light of the traffic light goes green, to provide time to the operator to prepare for taking control or assisting the mobile robot. In this manner, the mobile robot controlled or assisted by the operator can start crossing the road immediately when the light goes green which may result in a faster time for arriving to the destination and can also avoid the hazardous situation described in
In some embodiments, one operator can control more than one mobile robot 10. In such embodiments notifying the operator for crossing the road not on the instant the traffic light changes can be advantageous as it can provide to the operator time for planning which mobile robot to operate. It can also provide time to the operator for switching control from one mobile robot to the other. This may decrease or avoid the delay between the time the light of the traffic light goes green and the time the mobile robot starts crossing the road, which contributes on a faster arrival at the destination.
In yet some other embodiments, wherein the mobile robot 10 is configured to predict the traffic light state change of an intersection based on patterns between said intersection and other intersection, said prediction can be used to determine a shortest time route. For example, reference is made to the multiple intersection scenario of
Thus, in some embodiments, the states of the traffic lights on intersections of the operating area (such as depicted in
Whenever a relative term, such as “about”, “substantially” or “approximately” is used in this specification, such a term should also be construed to also include the exact term. That is, e.g., “substantially straight” should be construed to also include “(exactly) straight”.
Whenever steps were recited in the above or also in the appended claims, it should be noted that the order in which the steps are recited in this text may be accidental. That is, unless otherwise specified or unless clear to the skilled person, the order in which steps are recited may be accidental. That is, when the present document states, e.g., that a method comprises steps (A) and (B), this does not necessarily mean that step (A) precedes step (B), but it is also possible that step (A) is performed (at least partly) simultaneously with step (B) or that step (B) precedes step (A). Furthermore, when a step (X) is said to precede another step (Z), this does not imply that there is no step between steps (X) and (Z). That is, step (X) preceding step (Z) encompasses the situation that step (X) is performed directly before step (Z), but also the situation that (X) is performed before one or more steps (Y1), . . . , followed by step (Z). Corresponding considerations apply when terms like “after” or “before” are used.
While in the above, preferred embodiments have been described with reference to the accompanying drawings, the skilled person will understand that these embodiments were provided for illustrative purpose only and should by no means be construed to limit the scope of the present invention, which is defined by the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 18190327.9 | Aug 2018 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2019/070808 | 8/1/2019 | WO | 00 |
