This application claims the priority benefit of European Patent Application No. 22 166 079.8 filed on Mar. 31, 2022. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification.
The disclosure concerns an assistance method and a system for operating an agent in a dynamic and dense environment. In particular, the method assists in behavior planning for operating the agent using importance filtering with driving risk models.
Vehicular automation is a technical field, which uses technologies such as mechatronics or multi-agent systems to assist an operator of a vehicle in operating the vehicle in an environment. The term vehicles encompasses road vehicles, which may include an increasing number of advanced driving assistance systems, aircraft with advanced autopilots or unmanned aerial vehicles (UAV, drones), autonomously operating planetary rovers, or watercraft moving above the water as well as submersibles.
A vehicle using automation for addressing difficult tasks, for example navigation, to ease but not entirely replace human input by the operator, may be referred to as a semi-autonomous vehicle, whereas a vehicle relying solely on automation is called a robotic or autonomous vehicle. Similar techniques from automation are present in the field of robotics, in which robotic devices carry out a series of actions automatically. Subsequently, the generic term agent is used, which includes moving devices, e.g. vehicles and robots, which are able to perceive their environment, to determine and to execute actions autonomously in order to perform tasks.
Characteristic examples of tasks addressed by assistance systems for land vehicles include monitoring of blind spots around the vehicle or assisting in lane changes in a road traffic environment. The capabilities of assistance systems operating vehicles in real time depend on an availability of affordable sensors for perceiving the environment of the vehicle and of the computing power required for processing, both of which improved considerably in recent times. The road traffic environment is a particular application area, in which the increased capabilities of current assistance systems enable to move from assisting in less complex traffic scenarios on roads between cities to coping with congested and highly dynamic urban traffic scenarios.
The design of assistance systems has to address the handling of safety issues and avoiding potential collisions with other agents in the common environment. An aspect of the planning of future actions in the assistance system is the management of risks associated with the actions of the agent and ensuring compliance with safety margins. Current assistance systems may perform planning of future actions of the ego-agent using risk maps applying risk models. Analytical risk models may be applied to the domains of predicting a future evolvement of a perceived scenario in the environment, of the planning of potential actions to be performed by the ego-agent, and of warning an assisted operator or the ego-agent or autonomously performing a determined action by the ego-agent. Driving risk models predict movement of vehicles on trajectories along paths and may include risk types, e.g. collisions with other agents in the environment, collisions with static objects, risk resulting from sharp curves or regulatory risk. Risk includes the probability of an event to occur and the consequences of the event, e.g. the severity of the event. Regarding of risk during action planning for the ego-agent improves the behavior selection for the ego-agent.
The specific example of driving risk models in a road traffic scenario may use stochastic risk models for agent-to-agent collisions. The driving risk model may assume agents that move on predicted trajectories on paths in the environment. The predicted trajectories of the agents may include general Gaussian and Poisson distributed uncertainties. The generic risk models may be extended beyond modelling the collision risk to integrate further risks, e.g. curve risk or regulatory risk. For the purpose of action planning, the ego-agent may perform a cost evaluation based on the risk model.
The ego-agent may employ the risk model for performing cooperative planning with multiple trajectories and taking into account the paths of a plurality of other agents. Current planning algorithms employing risk models are capable of handling scenarios of few agents, for example up to five agents, in real time. While this may be sufficient for many scenarios encountered in the current traffic environment, the task for an urban traffic scenario including a plurality of other agents in addition to the ego-agent may be entirely different. Even more, in an urban traffic environment the other agents may also include pedestrians with their comparatively high agility, however at low velocities. A densely populated urban environment may confront the ego-agent with traffic scenarios, which include 15, 30 or even 50 other agents in the environment of the ego-agent, which the ego-agents sensors may perceive, and which the assistance system may have to process for behavior planning for the ego-agent. Predicting the future behavior of the other agents, may have to take different potential future behavior of each of the other agents, illustrated in
DE 10 2015 200 215 A1 proposes a driving assistance system, which uses a dynamic allocation of computation resources for perceiving or predicting at least one of the vehicles preceding to the ego-vehicle, or a portion of the environment of the ego-vehicle in greater detail than the other vehicle or the other portions of the environment. The driving assistance system attempts to achieve early results in perceiving and predicting objects in the environment by using a behavior prediction of perceived objects and a relevance assessment for the predicted behavior of the objects for addressing the question, which objects and which portions of the environment are indeed relevant in the scenario. The driving assistance system, which DE 10 2015 200 215 A1 discusses specifically using a lane change assistant, allocates and re-allocates computation and perception resources dynamically based on the assessed relevance of perceived objects or of areas. While using the prediction and the prediction assessment for resource allocation offers an efficient use of available resources to ensure real-time assessment of the traffic scene, densely populated traffic environments involving complex interactions between numerous individual agents in the scenario will exceed the capabilities of the described lane change assistant. DE 10 2015 200 215 A1 lacks an efficient and generally applicable approach to assess relevance of perceived objects or of areas beyond the application in the lane change assistant.
Thus, the aspect of scaling the implementation of an action-planning algorithm to more dynamic and more densely populated scenarios in the environment of the ego-agent becomes an increasingly important task.
The assistance method according to a first aspect, the program according to a second aspect and the assistance system according to a third aspect address these and other issues.
The assistance method according to the first aspect assists operation of an agent operating in a dynamic scenario involving at least one other agent in an environment of the agent. The computer implemented method comprises acquiring sensor data from the environment including data on the at least one other agent and on physical structures in the environment. The data conveys information on the environment. The method further acquires data comprising information on position and motion of the agent. It is to be noted that the term “data” is used hereinafter as a synonym for the information conveyed by using the data and vice versa. The method calculates an importance score for each agent of the at least one other agent based on the acquired sensor data and the acquired position and motion data. The importance score is a quantitative physical value describing a relevance of the particular agent for the operated agent based on the agent's dynamics. Then, the method generates a list comprising the at least one other agent based on the calculated importance scores of the at least one other agent, and partitions the generated list in at least one partition based on the importance score of the at least one other agent. Each partition includes at least one other agent. The method generates a prediction result by predicting a behavior of the at least one other agent included in the at least one partition based on the sensor data and the position and motion data using a selected behavior prediction model. The selected behavior prediction model is selected from a plurality of behavior prediction models based on the importance score of the at least one agent included in the at least one partition. The method outputs the prediction result to a planning and control system for operating the agent.
The method according to the first aspect computes an importance score for each other agent in the environment of the agent (ego-agent, operated agent) and arranges the other agents perceived in the environment in a ranked list ordered according to their respective importance scores. The method partitions the generated list of the other agents into partitions depending on the other agent's importance scores. The method predicts the other agent's future behavior based on the other agents membership in a respective partition of the ordered list. The method thereby enables to apply a specifically selected behavior prediction model for each partition, which may, for example, differ from the behavior prediction model selected for another partition with respect to its prediction accuracy and its computational complexity. The method may, for example select the accuracy and computational complexity of the behavior prediction model according to the importance scores of the other agents included in that partition. This may even include filtering out, and thereby neglecting those of the other agents, which have a particular low importance score. By partitioning and ranking the other agents according to their importance score enables to optimize use of processing resources by concentrating the scarce resources to those other agents, which have a high importance for the agent in the future evolvement of the traffic scenario. The method according to the first aspect thereby generates a processing environment for handling of highly dynamic scenarios in the environment of the agent, which includes a large number of other mobile agents present in the environment.
The second aspect concerns a computer program product comprising a non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device, causes the computing device to perform the assistance method according to the first aspect that assists operation of an agent operating in a dynamic scenario involving at least one other agent in an environment of the agent. The computer implemented method comprises acquiring sensor data from the environment including data on the at least one other agent and on physical structures in the environment. The data conveys information on the environment. The method further acquires data comprising information on position and motion of the agent. The method calculates an importance score for each agent of the at least one other agent based on the acquired sensor data and the acquired position and motion data. Then, the method generates a list comprising the at least one other agent based on the calculated importance scores of the at least one other agent, and partitions the generated list in at least one partition based on the importance score of the at least one other agent. Each partition includes at least one other agent. The method generates a prediction result by predicting a behavior of the at least one other agent included in the at least one partition based on the sensor data and the position and motion data using a selected behavior prediction model. The selected behavior prediction model is selected from a plurality of behavior prediction models based on the importance score of the at least one agent included in the at least one partition. The method outputs the prediction result to a planning and control system for operating the agent.
The program according to the second aspect comprises instructions, which, when a computer executes the program, cause the computer to carry out the method according to the first aspect.
The assistance system according to the third aspect assists in operating an agent that operates in a dynamic scenario involving at least one other agent in an environment of the agent. The system comprises at least one sensor configured to acquire sensor data from the environment including data on the at least one other agent and on physical structures in the environment. The system further comprises at least one ego-sensor configured to acquire ego motion data on the agent, and at least one processor. The at least one processor is configured to calculate an importance score for each of the at least one other agent based on the sensor data on the environment and the acquired ego motion data. The importance score is a quantitative physical value describing a relevance of the other agent for the agent based on the agent's dynamics. The at least one processor generates a list comprising the at least one other agent based on the calculated importance score of the at least one other agent, and partitions the generated list in at least one partition based on the importance score of the at least one other agent. Each partition includes at least one other agent. The at least one processor is configured to generate a prediction result by predicting a behavior of the at least one other agent of the at least one partition based on the sensor data using a selected behavior prediction model. The behavior prediction model for predicting the behavior of the at least one other agent included in the at least one partition is selected from plural behavior prediction models based on the at least one partition the other agent belongs to. The at least one processor is further configured to provide the prediction result to a planning and control system for operating the agent.
The assistance system according to the third aspect achieves the corresponding advantages as the computer—implemented method according to the first aspect.
The planning and control system may directly operate the agent by generating and outputting control signals for the agent operating in a dynamic scenario involving the at least one other agent. Additionally or alternatively, the planning and control system may generate and output further control signals to output devices for communication to an operator, which assist the operator in operating the agent in the dynamic scenario involving the at least one other agent.
The aspects and features of embodiments refer to the attached drawings, in which
The discussion of the figures uses same reference signs in different figures to denote same or corresponding elements. The discussion dispenses with discussing same reference signs for different figures wherever considered possible without adversely affecting understanding.
The method according to the first aspect computes an importance score for each other agent in the environment of the agent (ego-agent, operated agent) and arranges the other agents perceived in the environment in a ranked list ordered according to their respective importance scores. The method partitions the generated list of the other agents into partitions depending on the other agent's importance scores. The method predicts the other agent's future behavior based on the other agents membership in a respective partition of the ordered list. The method thereby enables to apply a specifically selected behavior prediction model for each partition, which may vary, for example, differ from the behavior prediction model selected for another partition with respect to its prediction accuracy and its computational complexity. The method may, for example select the accuracy and computational complexity of the behavior prediction model according to the importance scores of the other agents included in that partition. This may even include filtering out, and thereby neglecting those of the other agents, which have a particular low importance score. By partitioning and ranking the other agents according to their importance score enables to optimize use of processing resources by concentrating the scarce resources to those other agents, which have a high importance for the agent in the future evolvement of the traffic scenario. The method according to the first aspect thereby generates a processing environment for handling of highly dynamic scenarios in the environment of the agent, which includes a large number of other mobile agents present in the environment.
The method may be extended to involve at least two agents in the environment of the agent (ego-agent) operating in the dynamic scenario involving the at least one other agent.
Furthermore, the method may partition the generated list of the other agents into partitions depending on the other agent's importance scores, wherein one other agent may be in a first partition (e.g. “at least a bit important other agents”) and in a second partition (e.g. “very important other agents”) simultaneously. This may result in the other agent being considered in three different parallel planning steps, one planning step for each partition. Thus, an advantageous effect of a redundant planning is achieved.
Generating at least two partitions, each partition comprising a subset of a plurality of other agents, and performing prediction for the other agents included in each partition separately introduces redundancy and thereby also safety and robustness to the method for assisting in operating in agent. This may further be improved by adding the communication capability between the individual partitions in order to inform each partition on other agents that are included in both partitions.
Furthermore, the method may implement a switchover probability which enables to perform processing of generating a prediction result from a lower partition to a higher partition in case the higher partition already succeeds in finding a solution for the prediction problem.
Filtering the agents with particular low importance score may further reduce computational speed of the evaluation due to reducing the number of other agents having to be taken into consideration.
The dependent claims define further advantageous embodiments.
According to an embodiment, the plurality of behavior prediction models comprises individual behavior prediction models, which differ with respect to at least one of prediction accuracy and computational complexity.
The at least one partition may comprise at least a first partition including the other agents having a first range of the importance score and a second partition including the other agents having a second range of the importance score. The first range includes importance scores, which are smaller than the importance scores included in the second range, and the method comprises selecting the behavior prediction model for the first partition with a lower computational complexity than the behavior prediction model for the second partition.
The method according to an embodiment comprises discarding other agents with a calculated importance score below a threshold when generating the list.
According to one embodiment, the method comprises calculating the importance score by applying a risk shadowing process to the data on the at least one other agent and on the physical structures in the environment.
The method may apply the risk shadowing process including a step of performing a reachability analysis based on the sensor data and the position and motion data for determining occupied areas for the agent and the at least one other agent in the environment over time. The risk shadowing process determines overlapping areas from the occupied areas of the agent and the at least one other agent, and determines a relevance of each of the at least one other agent based on the determined overlapping areas.
According to an embodiment, determining the relevance of each at least one other agent includes disregarding the other agent in case the occupied areas of the agent and the at least one other agent do not exist. Additionally or alternatively, determining the relevance of each of the at least one other agent includes computing a continuous relevance score of the other agent by calculating a size of the overlapping area or a distance of the occupied areas.
Calculating the importance score according to one embodiment comprises decreasing an influence of a first other agent of the at least one other agent based on an influence of at least one second other agent of the at least one other agent.
According to one embodiment, calculating the importance score includes applying a risk model that regards an influence of the at least one other agent on the agent for computing the importance score of the at least one other agent.
The method may comprise, in one particular embodiment, calculating the importance score as a first importance score for each of a plurality of other agents by applying a first method. The method then proceeds by discarding a subset of the plurality of other agents based on a first filter criterion for the calculated first importance score for generating a first filtered list of the plurality of other agents. The method may then calculate the importance score as second importance score by applying a second method for the plurality of other agents, discarding a second subset of the plurality of other agents included in the generated list based on a second filter criterion for the calculated second importance score for generating a second filtered list of the plurality of other agents.
Advantageously, the first method for calculating the first importance score may have a lower computational complexity than the second method for calculating the second importance score.
The method according to an embodiment may calculate the importance score using a path distance model, a trajectory distance model, a Gaussian model, or a survival analysis model.
The agent may be a mobile robotic device, and the at least one other agent may include other mobile devices, in particular other mobile robotic devices.
Additionally or alternatively, the agent is an ego-vehicle, and the at least one other agent is another traffic participants in a road traffic scenario, in particular at least one of other vehicles, cyclists and pedestrians.
The operated agent and the at least one other agent may be vessels in a maritime traffic scenario.
The operated agent and the at least one other agent may be air vehicles in an air traffic scenario.
The operated agent and the at least one other agents may be space vehicles in a space environment.
The following discussion of embodiments predominantly references a road traffic scenario including a plurality of vehicles, bicycles and pedestrians as specific examples of the other agents in the environment of an ego-vehicle as a specific example for the ego-agent. It suffices to mention that the road traffic scenario is only one application example, which is of particular commercial interest and useful for illustrating an embodiment. Application of the method and system as defined in the attached claims is by no means restricted to the discussed road traffic scenario.
The processing illustrated in
The at least one sensor further acquires sensor data including data on the other agents and on objects and structures in the environment of the ego-agent. The method acquires position and motion data of the ego-agent. In particular, the position and motion data on the ego-agent includes data on a current state of the go-agent. The current state of the ego-agent may include a current position of the ego-agent, a heading of the ego-agent, a velocity of the ego-agent, and an acceleration or deceleration of the ego-agent. The method acquires data on the current state of the ego-agent relative to a map of the environment.
The method acquires the sensor data relative to the map of the environment. Thus, a common processing of the sensor data and the position and motion data by at least one processor becomes possible.
The processing depicted in
Filter 1, filter 2, and filter 3 form part of a first processing sequence for filtering the other agents perceived by the sensors based on a risk model selected for the particular filter 1, filter 2, and filter 3.
The method may use a plurality of risk models as indicated in the lower right portion of
Alternatively, at least one filter 1, 2, and 3 may perform risk shadowing using a particular risk shadowing processing.
For example, filter 2 may apply risk shadowing for filtering.
The input to filter 1 as first filtering step are the perceived other agents. In the shown example, filter 1 receives filter input data including data on 50 perceived agents as input. Filter 1 performs filtering using a first selected risk model of the plurality of risk models on the filter input data. Filtering the filter input data with the first risk model provides filter output data including data on 31 other agents at the filter output of filter 1.
Filter 1 performs filtering of the 50 perceived other agents to provide 31 filtered other agents as first filter output data. The first filter output data is subsequently fed as second filter input data to filter 2. According to
The second filter output data provided by filter 2 is subsequently provided to filter 3 representing the third and final step of filtering other agents.
The input to filter 3 as third filtering step are the 21 filtered other agents output by filter 2. In the example, filter 3 receives filter input data including data on 21 other agents as input. Filter 3 performs filtering using a third selected risk model of the plurality of risk models on the filter input data provided to filter 3. Filtering the third filter input data with the third risk model provides third filter output data including data on 15 other agents at the filter output of filter 3.
The process of filtering other agents reduces the number of other agents for further processing from 50 perceived other agents to 15 filtered other agents. Filtering out, or discarding, 35 other agents from the original 50 perceived other agents results in a corresponding increase in computation speed and reduction in processing complexity of a subsequent processing in a behavior prediction processing in an action planning module.
Reducing the number of other agents from the original number of perceived agents to the reduced number of filtered agents enables to use a planning algorithm which takes into account interactions between different agents including the ego-agent and the remaining other agents after filtering even in crowded and highly dynamic scenarios in the environment. The filtering approach bases on applying risk models, and therefore suppresses possibly fatal effects of the achieved reduction in complexity achieved by the process of filtering the other agents.
Using a filter cascade, comprising in
Filter 4, filter 5, and filter 6 form part of a second processing sequence for partitioning the filtered other agents obtained by the first processing sequence for filtering. Filters 4, 5, 6 are arranged in series.
Each filter 4, 5, 6 used for the partitioning processing employs a selected risk model for generating a set of filter output data from a set of filter input data based on the respectively selected risk model.
The selected risk model for each of the filters 4, 5, 6 is selected from the plurality of risk models, which are available.
Filter 4 obtains the third filter output data provided by the third filter as fourth filter input data. Thus, the fourth filter input data includes data on a filtered set of 15 other agents. Filter 4 applies a fourth selected risk model to the fourth filter input data and generates a first ranked list of other agents based on the calculated importance score. The calculated importance score is a first importance score calculated by applying the fourth risk model.
Filter 4 generates a first partition of the other agents included in the fourth filter input data and outputs the sorted agents included in the first partition. The first partition of the first ranked list of the other agents correspond to the other agents included in the first ranked list with the lowest importance score. The first partition is generated by discarding a first subset of the other agents from the first ranked list of other agents, whose respective calculated importance score is smaller than a fourth threshold.
In
Filter 5 obtains fifth filter input data, which corresponds to the third filter output data, and uses a fifth selected risk model of the plurality of risk models to generate a second ranked list of other agents based on the calculated importance score. The calculated importance score is a second importance score calculated by applying the fifth risk model.
Filter 5 generates a second partition of the other agents included in the fifth filter input data and outputs the sorted agents included in the second partition. The second partition of the second ranked list of the other agents correspond to the other agents included in the second ranked list with the lowest second importance score. The second partition is generated by discarding a second subset of the other agents from the second ranked list of other agents, whose respective calculated importance score is smaller than a fifth threshold.
In
Filter 6 obtains sixth filter input data, which corresponds to the third filter output data, and uses a sixth selected risk model of the plurality of risk models to generate a third ranked list of other agents based on the calculated importance score. The calculated importance score is a third importance score calculated by applying the sixth risk model.
Filter 6 generates a third partition of the other agents included in the sixth filter input data and outputs the sorted agents included in the third partition. The third partition of the third ranked list of the other agents correspond to the other agents included in the third ranked list with the lowest third importance score. The third partition is generated by discarding a third subset of the other agents from the third ranked list of other agents, whose respective calculated importance score is smaller than a sixth threshold.
In
Filter 4, filter 5, and filter 6 perform the second processing sequence for partitioning the filtered other agents obtained by the first processing sequence for filtering. Filters 4, 5, 6, provide three partitions of other agents based on calculated importance scores. The importance scores are calculated using at least one selected risk model.
The generated first partition of other agents includes other agents after the filtering, which have a calculated importance score value exceeding the fourth threshold.
A subsequent behavior-planning algorithm may apply a regular risk-based planning method with lower computational cost. Other agents included in the first partition may be used with one predicted trajectory with an associated uncertainty.
The generated second partition of other agents includes other agents after the filtering, which have a calculated importance score value exceeding the fifth threshold.
A subsequent interactive planning algorithm may apply an interactive model of medium complexity in order to solve more advanced driving situations. Other agents included in the second partition may be used with one single predicted trajectory with an associated fallback intention trajectory.
The generated third partition of other agents includes other agents after the filtering, which have a calculated importance score value exceeding the sixth threshold.
A cooperative planning algorithm may apply a complex cooperative behavior model, other agents included in the third partition may be used with multiple predicted trajectories and paths.
The cooperative planning algorithm may have a significantly higher computation complexity than the interactive planner applying a model of medium complexity and is therefore only used for a small number of other agents included in the third partition, which have an importance level exceeding the importance levels of the first partition and the second partition.
The partitioning processing in filters 4, 5, 6 in particular includes a redundancy, which adds safety and robustness to the method, which is advantageous and important for a situation analysis of a traffic scenario. A planning process will use results of the situation analysis in the next step to determine suitable actions for performing by the ego-agent in the analyzed scenario. The redundancy and robustness of the processing structure of
In particular,
In the depicted example the filter input data includes 32 perceived other agents. The filter output data includes a filtered set of other agents, which includes 14 other agents.
The filter performance of the filter depends on the risk model applied for calculating the importance score and the filter threshold.
Selecting the risk model from the plurality of risk models takes into account, that different risk models show different characteristics.
For example, risk models with an increased accuracy suffer from an increased computation time.
Risk models with decreased accuracy on the other hand will not have the capability to filter out a partition that includes a small number of other agents with a high importance score.
Designing a specific filter stage will set the filter threshold to a value so that the filter stage provides no or almost no false negative errors, and only some false positive errors occur during filtering.
In the exemplary case shown in
A false negative is an agent that is not included in the output of the filter but is actually important. Thus, a false negative agent is filtered out and therefore might generate safety issues, as the planner would not consider the false negative agent, and might create an accident as a consequence.
A false positive is an agent that is still included in the output of the filter although it is actually not important and could be neglected safely, without inducing adverse consequences. In order to ensure safety, it is more important to avoid inducing false negatives than to induce false positives. This aspect is considered when setting the respective threshold accordingly.
The design of the filter using a selected risk model takes into regard the computation speed of the filtering process on the one hand, which is of particular importance for real time application and for coping with highly dynamic scenarios in the environment of the ego-agent. On the other hand, design of the filter using a selected risk model takes into regard the number of filtered other agents of the filtering process, which is of particular importance for coping with highly dynamic scenarios involving a significant number of other agents in the environment of the ego-agent.
The embodiment copes with these design targets of the filtering approach by using different selected risk models from the plurality of risk models in subsequent filter stages.
Generally, filtering of a set of other agents using a risk model to generate a filtered set of other agents has two particular aspects. A first aspect of the filtering approach using the risk model is that a contribution of single agents to the complete risk in the perceived dynamic scenario in the environment of the ego-agent is regarded. The second aspect of the filtering approach concerns the filtering out or disregarding of other agents from further processing whose importance score not exceeds a risk threshold value. The effect of this approach is that, other agents are filtered out and not considered for further processing, because they do not essentially influence the future behavior of the ego-agent.
The 32 agents include the ego-agent and 31 other agents.
The filtering stage implemented with filters 1, 3, 4, 5, 6 each use a selected risk model selected out of a plurality of risk models.
The table arranges each shown type of risk model in one column. Within one type, the risk models are arranged from top to bottom of the table according to increasing accuracy of the of the resulting importance score that respective risk model provides. An increasing accuracy of the risk model corresponds to an increase in computational cost and thereby a respective use of processing resources by the risk model as indicated on the right side of the table.
Generally, the risk model provides a risk value, which is in range between 0 and 1 and describes the probability of a collision between agents. Severity outcome of risk is neglected in present case. The importance score is defined as the risk value provided by the risk model. A high risk value means a high risk and describes a high importance since the other agent will influence the ego-behavior of the operated agent. Distance models and time models can be transferred into a risk value, in particular by regarding the inverse of the distance or the time. The first type of risk models are distance models in the left column.
The second type of risk models are time models in the center column of
The third type of risk models are stochastic models in the right column of
Neither the number of types of risk models, nor the number of risk models of
The distance model determines the importance score for the interaction between the ego-agent and the other agent, the importance score representing a collision probability between the ego-agent and the other agent, by computing a current distance between a determined current position of the ego-agent (ego-position) and the position of the other agent (other position).
The path distance model determines the importance score for the interaction between the ego-agent and the other agent by computing a path distance between a path of the ego-agent (ego-path) and a path of the other agent (other path). In particular, the path distance corresponds to a minimum of the distance between the path of the ego-agent and the path of the other agent determined on a map of the environment.
In the particular example of intersecting paths, the distance is zero, ant he risk is at a maximum, which also means the importance is maximum.
The closest encounter model determines the importance score for the interaction between the ego-agent and the other agent by computing a distance for the point of closest encounter between a trajectory of the ego-agent moving with a velocity (ego-velocity) along the ego-path and a trajectory of the other agent moving with its velocity (other velocity) along the other path. The closest encounter distance is the minimum of the distance between the ego-agent and the other agent, which is predicted for future time steps.
The closest encounter with headway model determines the importance score for the interaction between the ego-agent and the other agent by combining the closest encounter distance according
The closest encounter with 2D headway model determines the importance score for the interaction between the ego-agent and the other agent by combining the closest encounter distance according
The circle approximation model takes an uncertainty of the predicted position of the ego-agent on the predicted path of the ego-agent and an uncertainty of the predicted position of the other agent on the predicted path of the other agent into account. Circles approximate the uncertainties of the position of respective agent. The circle approximation model determines the importance score for the interaction between the ego-agent and the other agent by determining the minimum of the distance (circle distance) between the circle approximation of the uncertainty of the position of the ego-agent and the circle approximation of the uncertainty of the corresponding position of the other agent along their predicted trajectories. The circles in
The 2D-Gaussian model takes an uncertainty of the predicted position of the ego-agent on the predicted path of the ego-agent and an uncertainty of the predicted position of the other agent on the predicted path of the other agent into account. 2D-Gaussian distributions model the uncertainties of the position of ego-agent and the position of the other agent. The 2D-Gaussian model determines the importance score for the interaction between the ego-agent and the other agent by determining the minimum of the overlapping area between the 2D-Gaussian distribution of the uncertainty of the position of the ego-agent and the 2D-Gaussian distribution of the uncertainty of the corresponding position of the other agent on their respective predicted trajectories. The 2D-Gaussian models in
The 2D-Gaussian model takes an uncertainty of the predicted position of the ego-agent on the predicted path of the ego-agent and an uncertainty of the predicted position of the other agent on the predicted path of the other agent into account. 2D-Gaussian distributions approximate the uncertainties of the position of ego-agent and the position of the other agent. The survival analysis model determines the importance score for the interaction between the ego-agent and the other agent by integrating all Gaussian overlaps using a Poisson-dependent survival function.
The scenario may represent a road traffic scenario including the ego-agent (ego-vehicle) and 47 other agents (other vehicles). The ego-agent perceives therefore 47 other agents (perceived other agents).
The first route (road) 12 shows two lanes per driving direction 2, 3. The second route (road) 13 has two lanes per driving direction 14, 15.
The road traffic scenario shown in
The set of perceived other agents represents the filter input data. In a first step S11, a risk for each single other agent is determined. In particular, in step S11, the method computes an importance score for the other agent and the ego-agent by applying the selected risk model.
In step S12, the determined risk for the single agent is compared with a predetermined risk threshold. If the determined risk does not exceed the risk threshold, the method determines that the other agent may be neglected. Given this case, the other agent is removed from the set of perceived other agents for generating the set of filtered agents in step S13.
For determining the risk represented by the other agent by computing the importance score in step S11, the determined importance score is compared to respective importance threshold in step S12. In case, the computed importance score is equal or above the importance threshold, the other agent is maintained in the filtered set of other agents. In case, the computed importance score smaller than the importance threshold in step S12, the other agent is removed from the set of agents in step S13 for generating the filtered set of other agents.
The method performs the steps S11 to 13 for each agent included in the set of agents included in the filer input data. The generated filtered set of agents is output by the filter in the filter output data.
A performance analysis of the importance filtering process using a selected risk model includes a step S21 of acquiring a set of other agents in the filter input data, selecting a specific risk model of the plurality of risk models, and generating the filtered set of agents by performing the filtering process according to steps S11 to S13. For obtaining a reference for the performance analysis of the filter, in a step S21, a fixed set of agents is obtained, for example by using a survival analysis as the baseline risk model.
In step S23, the performance analysis of the filter is performed by comparing the filtered set of agents from step S21 with the fixed set of agents obtained from the survival analysis in step S23.
The first risk threshold for filtering the set of other agents is set to a high threshold value in
The traffic scenarios shown in
The ego-agent is depicted with a circle with a hatched interior area. Circles with a monotonously-colored area represent the other agents in
The shade of the colored area representing the other agents in
The darker shade of the other agents close to the ego-agent indicates that the importance score computed by using the current distance risk model results in respectively high calculated importance score values. Contrary thereto, the faint shade of the other agents remote to the ego-agent indicates that the importance score computed by using the current distance risk model results in respectively low calculated importance score values in the example of
Thus, the current distance risk model weights those other agents with a higher importance than those of the other agents, which are currently at a more remote position relative to the ego-agent.
The darker shade of the other agents on the same path as the ego-agent and on the paths crossing the path of the ego-agent at the intersection indicates that the importance score computed by using the survival-analysis risk model results in respectively high calculated importance score values for interactions of the ego-agent with these other agents. Contrary thereto, the faint shade of the other agents on paths remote to the path of the ego-agent, and of other agents that already passed the intersection that the ego-agent is approaching indicates that the importance score computed by using the survival-analysis risk model results in respectively low calculated importance score values in the example of
The relevant elements within the area 18 of the plane 17 are the false negative elements and the true positive elements. For analyzing the filter performance, expression (1) defines a true positive rate:
and expression (2) a false negative rate:
The true positive rate according to expression (1) and the false positive rate according to expression (2) are determined for a single stage filter for filtering a set of other agents using one specific selected risk model. This enables to compare the characteristic performance of the different risk models of the plurality of risk models with each other.
On the abscissa axis of
The plurality of risk models shown by the respective curves include the current-distance model, the path-distance model, the trajectory-distance model, closest-encounter model, the encounter-and-headway model, the encounter-and-2d-headway model, the circle-approximation model, the Gaussian model, and the survival-analysis model.
For reference purposes,
When comparing the curves shown in
The current-distance model, for example, shows to a similar performance as a random filtering approach. The current distance model is far away from the ideal point (true positive rate=1 and false positive rate=0).
The path-distance model stands out with high false positive rates. The path distance model has either low true positive rate values or high false positive rate values, which results in a large distance to the ideal point.
The closest-encounter-and-headway and the closest-encounter model result in a design of the filter with high false positive rates compared with filters designed with most other risk models.
Contrary thereto, closest-encounter-and-2D-headway and the circle—approximation model both show significantly better filter performance, which for those two risk models is also similar.
Closest-encounter model, closest-encounter-with-headway model and closest-encounter-with-2D-headway model are risk models with a good performance due to their closeness to the ideal point in the top left area of the
The circle approximation model shows a higher robustness and a better performance than the closest encounter models.
The filter design based on the trajectory-distance model shows an advantageously high true positive rate for a rather low false positive rate. The good filtering characteristics of the trajectory-distance model go along with an extraordinarily low computation time for calculating the importance scores, thereby having good computation time and computation complexity.
The survival-analysis model represents the baseline model, which shows ideal filtering performance concerning the true positive rate.
The risk model coming closest to the survival-analysis model with respect to the filtering performance in
Based on
Taking into regard the aforementioned examples for individual risk models of the plurality of risk models, an advantageous implementation of the assistance system may use multiple filter stages of the embodiment of
It is particularly advantageous, when the sequential filters use computationally fast models as selected models first in order to reduce the time required for filtering over all three filter stages.
For example, filter 1 may filter the first filter input data based on the path-distance model as the first selected risk model.
For example, filter 2 may subsequently filter the second filter input data based on the trajectory-distance model as the second selected risk model.
For example, filter 3 may proceed by filtering the third filter input data based on the 2D-Gaussian model as the first selected risk model.
The filtering step is followed by a partitioning step (sorting step), which uses three filters, filter 4, filter 5, and filter 6.
For example, filter 4, filter 5, and filter 6 may filter the first filter input data based on the survival analysis model, using respectively differing thresholds in order to partition the other agents into different groups, which are then handled respectively different in a subsequent planning step. The subsequent planning then enables planning and determining actions with interaction aware planning algorithms, which, based on the previous filtering step in sequential filtering stages, and the partitioning step, enable to determining a suitable action or suitable actions for crowded and highly dynamic scenarios in the environment of the ego-agent.
A respective one of the filter stages of the filtering step, for example filter 2 as depicted in
The risk-shadowing process provides such a heuristic for an embodiment of the assistance method.
Risk models only regard a risk for a single agent-to-agent interaction. There exist scenarios in the environment, which involve a plurality of moving agents. The following example uses a simple scenario involving the ego-agent and two other agents. The following examples of dynamic scenarios use road traffic scenarios with one ego-agent and two other agents each for illustrative purposes. It is evident that a road traffic environment with dynamic traffic scenario including a plurality of other agents will benefit from the risk shadowing approach to a particular extent.
The first example shows an intersection in a road traffic scenario, where two roads intersect. The ego-agent is an ego-vehicle 21, which approaches the intersection and intends to follow its driving path 21.1 crossing over the intersection.
On the same road as the ego-vehicle 21, but with an opposing driving direction, one first other agent approaches the intersection. The first other agent is a truck 22, which intends to turn left on the intersection, as the driving path 22.2 shown in
On the further road, which crosses the road on which the ego-vehicle 21 and the truck 22 travel, a second other agent approaches the intersection on its respective driving path 23.1. The second other agent is the other vehicle 23, which intends to continue traveling on its current road straight over the intersection, as the driving path 23.1 indicates. A predicted driving behavior of the other vehicle 23 may pose a collision risk for a collision with the truck 22 on its driving path 22.1 and subsequently a collision risk for a collision involving the ego-vehicle 21 on its driving path 21.1.
Applying risk shadowing in the traffic scenario of
The second exemplary road traffic scenario shows the intersection of
On the further road, which crosses the road on which the ego-vehicle 21 travels, a first and a second other agent approach the intersection on its respective driving paths 23.1 and 24.1. The first other agent is the other vehicle 23, which intends to continue traveling on its current road straight over the intersection, as the driving path 23.1 indicates. A predicted driving behavior of the other vehicle 23 may pose a collision risk for a collision with the ego-vehicle 21 on its driving path 21.1. The second other agent is a second vehicle 24 following the vehicle 23, wherein the vehicle 24 intends to turn right at the intersection. The vehicle 24 has a driving path 24.1, which only may represent a collision risk to the ego-vehicle 21, when first colliding with the vehicle 23.
Applying risk shadowing in the traffic scenario of
The third example, which illustrates the advantages of risk shadowing includes a perceived road structure in the environment, which may for example be a traffic barrier 27. The traffic barrier 27 depicted in
The structure of the assistance method or system of
The input data to the risk shadowing process may be the data on the perceived other agents and the position and motion data of the ego-agent. The input data may be acquired from the sensor suite of the ego-agent, and may be preprocessed. Alternatively or additionally, the input data to the risk shadowing process has been subject to a filtering process before performing the risk shadowing process on the input data.
The input data in particular includes data on the current dynamic scenario in the environment of the ego-agent. The input data includes also data on a predicted future evolvement of the traffic scenario. The predicted future evolvement of the scenario may include data on motion paths of other agents present in the environment and predicted trajectories of the other agents. The data may also include data on stationary structures in the environment, such as road geometry, or traffic barriers 27. The data may include regulatory data on applicable traffic rules.
The risk shadowing process includes a first step S41 of determining critical points 28 from the input data, which defines a traffic scenario in the environment of the ego-agent. The system may determine the critical points 28 using a risk model, in particular using the closest encounter model.
Determining the critical points 29 comprises computing future collision points from the perspective of each of the agents present in the traffic scenario. Each of the agents includes the ego-agent and the perceived other agents in the environment of the ego-agent.
Computing the future collision points uses time, distances and points of closest encounter and bases on the paths and the predicted trajectories of the ego-agent and the other agents.
With respect to
Step S41 proceeds with determining the critical points 28 from the perspective of the truck 22 (other agent). Based on the perceived scenario, a critical point 28.3 from the perspective of the truck 22 is a collision risk with the ego-vehicle 21. A first critical point from the perspective of the truck 22 might be a collision point resulting from intersecting trajectories 22.1 and 23.1. However, this potential critical point is left out, since only the critical points, which are located beyond the other agents' current position, are considered.
The critical points 28 from the perspective of the ego-vehicle 21 based on the perceived scenario include a first critical point 28.4 resulting from the ego-vehicle 21 colliding with the truck 22. A second collision point of the ego-vehicle 21 with the other vehicle 23 determines a second critical point 28.5 from the perspective of the ego-vehicle 21.
Having determined the critical points 28, 28.1, 28.2, 28.3, 28.4, 28.5 in step S41, the method proceeds with step S42. The critical points 28, 28.1, 28.2, 28.3, 28.4, 28.5 form the basis for the processing in subsequent step S42.
In step S42, the process of risk shadowing proceeds with generating reachability areas based on the determined critical points 28 from step S41 and the input data. A reachability area of an agent starts at the agent's current position in the environment and ends at a closest point of the agent with any further agent, wherein the closest point is also a collision point of the agent with the further agent.
A reachability area extends along the trajectory of the agents for which the reachability area is determined. The reachability area may have a width corresponding to a width of the agent, whose reachability area it is. The reachability area may correspond to a ground projection of the agent moving along its trajectory from the current position to the first critical point on its trajectory.
Step S42 determines a reachability area for each agent of the ego-agent and the perceived other agents present in the input data to the risk shadowing process.
With regard to
Turning to the perspective of the truck 22, step S42 determines a second reachability area 29.2 for the truck 22. The second reachability area 29.2 starts at the current position of the truck 22 and extends to the first critical point 28.3 on the predicted trajectory 22.1 of the truck 22, which is the only critical point 28 on the trajectory of the truck 22.
Concerning the perspective of the ego-vehicle 21, step S42 determines a third reachability area 29.3 for the ego-vehicle 21. The third reachability area 29.3 starts at the current position of the ego-vehicle 21 and extends to the first critical point 28.4 on the trajectory 21.1 of the ego-vehicle 21. The second critical point 28.5 comes after the first critical point 28.4 on the trajectory 21.1 of the ego-vehicle 21 and accordingly remains disregarded.
Thus, step S42 generates one reachability area for each of the ego-vehicle 21, the truck 22, and the vehicle 23. The method of risk shadowing now proceeds to step S43.
Step S43 searches for an overlap area 30 for the reachability area 29 of the ego-agent and the respective reachability areas 29 of each other agent of the perceived other agents present in the input data to the risk shadowing process.
Alternatively or additionally, Step S43 may include computing a surface area value of each determined reachability area 29 of the ego-agent and the respective reachability areas 29 of each other agent of the perceived other agents for generating an importance score for each interaction of the ego-agent with one other agent with a continuous value.
Additionally or alternatively, Step S43 may include computing respective distances between the reachability area 29 of the ego-agent and the respective reachability areas 29 of each other agent of the perceived other agents for generating an importance score for each interaction of the ego-agent with one other agent with a continuous value.
In the particular example of
Concerning the ego-agent 21 and the truck 22, Step S43 searches for an overlapping area 30 for the second reachability area 29.2 of the truck 22 and the third reachability area 29.3 of the ego-agent 21. The reachability areas 29.2 and 29.3 do indeed overlap, so that Step S42 determines an overlapping area 30 for the interaction between the ego-agent 21 and the truck 22.
Thus, the step S42 determines that the truck 22 has an influence on the ego-vehicle 21 as indicated by the existence of the overlapping area 30, whereas the vehicle 23 does not influence the ego-vehicle 21.
In a subsequent step S44 depicted in
Step S44 may include comparing a continuously valued importance score determined in step S43 with a predetermined threshold value for determining whether to disregard another agent for further processing. Thus, the risk shadowing process enables to reduce the number of the perceived other agents in the dynamic scenario in the environment of the ego-agent taking second order interactions between the other agents for filtering into regard. Thereby, the risk shadowing processing is capable to decrease the complexity of the dynamic scenario for further processing, and to improve the possibilities of assisting the ego-agent when implementing further planning and action execution steps based on the result of the risk shadowing.
The risk shadowing process S4 bases on the perceived scenario in the environment of the ego-agent. The risk shadowing uses a perception stack that is capable to detect the other agents and their movement characteristics. Risk shadowing therefore comes after a step S0 of perceiving the dynamic scenario in the environment based on sensor data acquired from the environment.
The process S4 of risk shadowing generates and provides a filtered scenario to the subsequent planning process. The filtered scenario includes other agents, which are of high importance for the predicted further evolvement of the current traffic scenario. Other agents, which, based on their future geometrical driving constraints, are not considered further in the filtered traffic scenario generated by performing risk shadowing of step S4, are neglected.
The risk shadowing process S4 proposes a simple and efficient way to compute bounds of the reachability area 29 in step S42. The risk shadowing process S4 determines the reachability area 29 for the future time steps of the scenario, in particular all future time steps up to a prediction time horizon. The risk shadowing process S4 uses the computed reachability areas 29 for determining an importance score for filtering of other agents in the scenario.
The process of risk shadowing of step S4 may be applied to any dynamic scenario, for example to any driving situation in a traffic environment. The risk shadowing process considers dynamics of the scenario in a computationally efficient manner.
The process of risk shadowing reduces the required computation time required in a subsequent planning step S5, which determines actions and/or warnings based on the filtered scenario. The planning step S5 may include determining actions, which subsequently form a basis for generating and outputting a control signal to actuators of an autonomously or partially autonomously operating ego-agent.
Additionally or alternatively, the planning step S5 may include determining information, which subsequently forms the basis for generating and outputting an information signal or warning signal via a human machine interface to an operator of an autonomously or partially autonomously operating ego-agent.
In the exemplary traffic scenario depicted in the lower part of
In this dynamic scenario, the first vehicle 31 exerts a strong influence 37 on the further evolvement of the current dynamic scenario from a perspective of the ego-vehicle 21. On the other hand, the second vehicle 33 has only a weak influence on the further evolvement of the current dynamic scenario from the perspective of the ego-vehicle 21.
Performing the risk shadowing process S4 and filtering of the scenario of
The filtering process uses a selected risk model for determining the importance value used for filtering the perceived dynamic scenario and the perceived other agents in order to reduce the complexity of the representation of the dynamic scenario.
The selected risk model is a one of plurality of driving risk models. A driving risk model is a stochastic model for an agent-to-agent collision in the environment. Determining an agent-to-agent collision as an example for an interaction between two agents may be determined using predicted trajectories for both agents involved in the collision.
A future position each of the ego-agent and of the other agent may be predicted with an uncertainty. The predictions of the trajectories of the ego-agent and of the other agent each include an uncertainty, which may be considered as a Gaussian and a Poisson distributed uncertainty.
The uncertainty may result from the uncertainty to predict a velocity of the other agent over an extended period of time into the future. The velocity at the current point in time may be known, e.g., by sensing or measuring. For future points in time, actions performed such as braking or accelerating performed by the other agent may each influence the velocity, and therefore the future positions of the other agent.
The risk model enables to predict a probability P (collision) between the ego-agent and the other agent based on the respective trajectories of the ego-agent and of the other agent.
In the example of the ego-vehicle 21, the ego-vehicle 21 mounts at least one sensor 41 for sensing the environment of the ego-vehicle 21. Such sensor 41 might form part of extensive sensor suite including, for example, one or more of a camera sensor, a LIDAR sensor, a RADAR sensor, a car-to-x communication, an ultrasonic sensor or any combination of these sensors 41. Based on sensor signals provided by the sensors 41, preferably the entire environment of the ego-vehicle 21 is observed. The sensor 41 forwards the sensor data including information of the sensed environment of the ego-vehicle 21 to a processor 42 forming part of an electronic control unit (ECU) of the ego-vehicle 21. The processor 42, alternatively an array of processors 42 arranged in combination with memory performs the method steps or the assistance method.
The ego-vehicle 21 further comprises at least one further sensor 43 that generates position data of the ego-vehicle 21 and motion data of the ego-vehicle 21. The sensor 43 may include a Global Navigation Satellite System (GNSS) sensor. Additionally or alternatively, the sensor 43 includes a vehicle odometer module. The motion data may include data on a current velocity of the ego-vehicle 21, a current acceleration or deceleration (braking) of the ego-vehicle 21, and a turning rate of the ego-vehicle 21.
The processor 42 generates a representation of the current situation in the sensed environment based on the sensor data from the sensor 41, position data of the ego-vehicle 21 and motion data of the ego-vehicle 21.
Starting from the representation of the current situation, a potential ego-vehicle behavior is determined.
Based on the current situation and taking into consideration the potential ego-vehicle behavior, the processor 42 performs a prediction for generating predicted data including predicted future behavior for the other vehicles 22, 23 in the environment.
Based on a selected trajectory and behavior, the processor 42 generates a control signal, and outputs the control signal to a human machine interface 44 (HMI). The control signal may control output of information via the HMI 44 to an operator to inform them of the determined trajectory and behavior that needs to be performed in the current traffic scenario in order to safely operate the ego-vehicle 21.
Additionally or alternatively, the processor 42 provides the control signal to a controller for controlling at least one actuator 46 of the ego-vehicle 21. In case of fully or partially-autonomous driving the ego-vehicle 21, the controller 45 receives the control signal and, based on the control signal, controls the actuators 46 of the ego-vehicle 21 to implement the determined trajectory and behavior that needs to be performed in the current traffic scenario in order to safely operate the ego-vehicle 21.
Such actuators 46 may include at least one of a brake system, an accelerator pedal, a steering system, indicator lights and a vehicle-to-x communication system of the ego-vehicle 21.
An application of the processor 42 for outputting control signals for autonomously operating the ego-vehicle 21, illustrates another advantageous application of the method, in which corresponding structural elements implement the method for autonomously controlling a robotic device as the ego-agent in a dynamic scenario in the environment.
In the specification and the claims, the expression “at least one of A and B” may replace the expression “A and/or B” and vice versa due to being used with the same meaning. The expression “A and/or B” means “A, or B, or A and B
Number | Date | Country | Kind |
---|---|---|---|
22166079.8 | Mar 2022 | EP | regional |