Patent Application
20220032951
The present disclosure relates to control agents for robots in simulation environments.
Research and Development (R&D) in the field of autonomous robot control relies heavily on simulation for testing, evaluation, validation and machine learning (ML) based training. In general, the more realistic and diverse the simulation, the more useful it is. Realism of simulation keeps the simulated environment true to the real world. Diversity of the simulated behavior, especially diversity in the simulated interaction between the ego vehicles and social vehicles, ensures coverage of vehicle behavior variations in the real world. Key to realism and diversity is the behavior of traffic participants, especially social vehicles, during their interaction with the ego vehicle and with each other. Meanwhile, behavior of social vehicles needs to be combinable, controllable, configurable, and automatable to allow expressive and repeatable simulations crucial for effective validation and training. Hence, autonomous driving simulation needs to provide realistic and diverse interactive behaviors of social vehicles and appropriate mechanism to combine, control, configure and automate the use of such behaviors. This in turn means that even though the primary goal of autonomous driving R&D is to develop a single agent that is a competent autonomous driver, autonomous driving simulation needs to flexibly combine multiple diverse agents to develop such a single agent.
Existing simulation systems provide limited options for independent control of social vehicles in a simulated environment.
Accordingly, there is need for systems and methods that enable flexible control of multiple simulated vehicles, including social vehicles, in autonomous driving simulations.
The present disclosure describes methods and systems that enable control of an object to be transitioned from a first agent that applies a first behavior policy to a second agent that applies a second behavior policy. The control is transitioned during a transition period that can enable the second agent to be initialized so as to facilitate a smooth transition. Example embodiments may enable an object to be controlled in diverse ways across diverse scenarios using agents that are specialized for such scenarios. The use of specialized agents may reduce the computation resources (e.g., processor operations and/or memory access and capacity) required for controlling the object in some applications, including simulated environments where several agents may need to be controlled simultaneously.
In at least one example aspect, the present disclosure describes a computer implemented method for controlling the behavior of an object, comprising: controlling the behavior of the object during a first time period by using a first agent that applies a first behavior policy to map observations about a state of the object in the first time period to a corresponding control action that is applied to the object; transitioning control of the behavior of the object from the first agent to a second agent during a transition period following the first time period; and controlling the behavior of the object during a second time period following the transition period by using a second agent that applies a second behavior policy to map observations about a current state of the object in the second time period to a corresponding control action that is applied to the object. During the transition period the first agent applies the first behavior policy to map observations about a state of the object in the transition period to a corresponding control action that is applied to the object and the second agent applies the second behavior policy to map observations about the state of the object in the transition period to corresponding control actions that are not applied to the object.
In at least some example of the preceding aspect, the observations mapped by the first behavior policy and the observations mapped by the second behavior policy are each from respective, different, observation spaces.
In at least some example of the preceding aspects, during the transition period the method includes modifying observations generated in respect of the object to include observations required by the second behavior policy.
In at least some example of the preceding aspects, first time period corresponds to a time that the object is present in a first bubble defined by a first spatiotemporal boundary, the second time period corresponds to a time when the object is present in a second bubble defined by a second spatiotemporal boundary, and the transition period corresponds to a time when the object in present within a transitional zone between the first bubble and the second bubble, the method including performing the transitioning upon detecting presence of the object in the transition zone following presence of the object in the first bubble.
In at least some example of the preceding aspects, the method further comprises: transitioning control of the behavior of the object from the second agent to the first agent during a further transition period following the second time period. During the further transition period the second agent applies the second behavior policy to map observations about a state of the object in the second transition period to a corresponding control action that is applied to the object and the first agent applies the first behavior policy to map observations about the state of the object in the further transition period to corresponding control actions that are not applied to the object.
In at least some example of the preceding aspects, the method is applied during a simulation run, the object is a simulated object and the observations are simulated observations.
In at least some example of the preceding aspects, the object is a simulated social vehicle operating in a simulated environment that also includes a simulated ego vehicle that is controlled throughout the first time period, transition period and second time period by a respective ego agent that applies an ego behavior policy to map ongoing observations about the state of the ego vehicle to corresponding ego vehicle control actions that are applied to the ego vehicle.
In at least some example of the preceding aspects, the second bubble and the transition bubble are fixed in a virtual position that moves with the virtual location of the simulated ego vehicle within the simulated environment.
In at least some example of the preceding aspects, the second bubble and the transition bubble are fixed in a virtual position that is stationary with a virtual physical location within the simulated environment.
In at least some example of the preceding aspects, the first behavior policy is less computationally intensive than the second behavior policy.
In at least some example of the preceding aspects, the second behavior policy is configured to map observations from an observation space that is larger than an observation space that the first behavior policy is configured to map observations from.
In at least some example of the preceding aspects, the second behavior policy is configured to map observations to control actions from an action space that is larger than an action space that the first behavior policy is configured to map control actions from.
According to a further example aspect is a computer system comprising a processor and a non-transitory memory coupled to the processor, the memory storing instructions that, when executed by the processor, configure the computer system to perform the method of any of the preceding aspects.
According to a further example aspect is computer program product comprising a non-transitory computer medium storing instructions for configuring a computer system to perform the method of any of the preceding aspects.
For a more complete understanding of example embodiments, and the advantages thereof, reference is now made to the following detailed description taken in conjunction with the accompanying drawings, in which:
Similar reference numerals may have been used in different figures to denote similar components.
The AI controller for a vehicle can incorporate one or more trained agents. An agent is a computer-implemented program or program module that applies a learned behavior model to map observations about a state to a respective action. In a real world application, the subject vehicle includes a set of sensors for sensing data, which collectively provide observations about the state of the vehicle, and a set of controllers for controlling vehicle actuators in response to a respective action generated by the agent. The observations about the state includes sensed information about operating characteristics of the vehicle (e.g., state of vehicle actuators, vehicle pose, vehicle linear and angular speed and acceleration) as well as sensed information about the environment the vehicle is operating in (e.g., image derived from LIDAR, image and/or radar units).
Simulator system 100 is configured to generate real-world simulations 120 for training agents across a range of simulation scenarios before the trained agents are transferred to real world applications.
In this regard, simulator system 100 is configured to run simulations 120 that include a simulated ego vehicle 122 and one or more simulated social vehicles 124. In an example embodiment, the simulator system 100 includes an ego agent 102 for controlling simulated ego vehicle 122. The ego vehicle 122 is the primary object of interest in the simulation as the ego agent 102 is what is being trained for transfer to real-world vehicle control applications. The ego agent 102 receives simulated observations about the state of simulated ego vehicle 122, and maps those observations about the state of the simulated ego vehicle 122 to actions for the simulated ego vehicle 122 to perform in the simulated environment. During a simulation run, this process is repeated over several simulation time steps. Simulated social vehicles 124 are provided to interact with simulated ego vehicle 122 in simulations 120. Social vehicles 124 are provided to simulate the behavior of vehicles that a real world ego vehicle would interact with.
In example embodiments, simulator system 100 is configured to implement instances of social agents 104 that apply respective behavior policies to control behavior of one or more social vehicles 124 such that they function independently of the ego vehicle 122. The simulator system 100 is configured to activate instances of social agents 104 that control the simulated behavior of social vehicles 124.
In some simulation environments a possible solution can be to use a single social agent to manage all the behavior of all social vehicles in all situations. However, such a solution can face challenges in terms of computational requirements and an inability to provide a diversity of experiences. Such a ubiquitous agent would require a great deal of computational resources such as processor time and memory space. For computational scalability of simulation, it may not be desirable or practical to simulate social vehicle behaviors and interaction at the highest level of fidelity in all scenarios.
Accordingly, in example embodiments, the social agents 104 include different types of social agents (e.g. social agents 104-A to 104-n) that can respectively apply different behavior polices (BP 106) for controlling social vehicle behavior. For example, social agent 104-B may apply a different behavior policy 106 than social agent 104-A and so on. In at least some examples, this can enable computationally efficient social agents 104 that are specialized at some aspects of social vehicle control but not suitable for other aspects. For example, a social agent 104-B may be a powerful and compute-intensive agent that can be used to control a social vehicle 124 where fine-grained interaction matters, such as dealing with unprotected left turn, busy intersection, or on-ramp merge. Where fidelity of intersection does not matter as much, such as in a constant-speed lane following situation, a much simpler and less computationally intensive social agent 104-A could be used.
In example embodiments, the different social agents 104 may have different observation spaces and different action spaces than each other. For example, one social agent 104 may be configured to receive simulated image data, whereas another may be configured to receive simulated radar data.
The use of multiple social agents 104 can also enable a diverse range of social vehicle behavior, thereby enabling the ego agent 102 to be presented with a wide and diverse range of training scenarios. Behavioral diversity of the social vehicles may provide a realistic simulation of the different driving style and different abilities of human drivers that contribute to the complexity of real interaction on a real world road.
In example embodiments, control of a social vehicle 122 during a simulation run of a simulation 120 may be transitioned from one social agent 104-A to a different social agent 104-B as the social vehicle moves from a simulation experience requiring one level of control to a simulation experience that requires a different level of control. A specific social agent 104 can be associated with a specific ego vehicle 124 in view of a specific operational scenario to effect a specific behavior. The agent-vehicle-scenario-behavior match can change as required by the simulation scenario. Accordingly, as will be described below, example embodiments are directed towards dynamically changing and managing agent-vehicle associations during a simulation run.
In example embodiments, simulator system 100 may use a heterogeneous computing configuration to implement social agents that apply respective behavioral policies. The various social agents 104 may be based on behavior policies 106 that are scripted, based on model predictive control or similar classical methods, or data-driven and trained through imitation learning or reinforcement learning.
At any time, a single instance of a social agent 104 may control a single social vehicle 122 or may control multiple social vehicles 122 together as a spatial or logical group in batch mode. Different types of social agents 104 may be specifically designed for particular scenarios or tasks such as highway merge, following a lane or handling stop signs, but may not be suitable for other scenarios. As previously suggested, different types of social agents 104 may assume different observation spaces and action spaces.
In some example embodiments, at specific point in space and time or when specific conditions are met during the course of a simulation run, control of a subset of social vehicles 142 may be transferred from one social agent 104-A to another social agent 104-B, so as to use the most suitable type of social agent 104 to provide the most suitable interaction, without wasting unnecessary computing resources to simulate every detail of the interaction where it does not matter. In example embodiments, simulator system 100 is configured to flexibly choose from a set of diverse social agents 104 to control social vehicles 124.
In example embodiments, simulator system 100 is configured to recognize constraints in respect of agent-vehicle-scenario-behavior matches when making agent-vehicle assignments. For example, a social agent 104 may be configured to expect specific types of simulated observations to be delivered from an assigned social vehicle 124 and expect the social vehicle 142 to be able to perform specific types of actions (in some examples, via intermediate controllers). Accordingly, a specific social agent 104 may be only suitable for some scenarios and behaviors but not others. Consequently, simulator system 100 is configured to make agent-vehicle assignments to satisfy compatibility in terms of matching observation and control spaces. In some examples, a change in social agent 104 also requires modifications to the simulated social vehicle 124. For example, the simulated social vehicle 124 may need to be prepared with the appropriate simulated sensors and actuators which in turn may need time to be appropriately initialized.
In addition, there are constraints on the switching of vehicle control between social agents 104. Vehicles have mass and inertia, and thus abrupt control change may be physically inappropriate. An incoming social agent 104 may need multiple simulation steps to appropriately initialize its internal state. The incoming social agent 104 may acquire enough pre-transition history about the state of the social vehicle and its' surrounding environment to correctly predict a future state and corresponding action. In example embodiments, simulator system 100 is configured to ensure a smooth agent to agent handover of control of a vehicle 124 in view of such constraints. In example embodiments, a smooth handover is characterized by the absence of unreasonable change of the simulated physical behavior of the social vehicle 124, and the absence of inconsistent internal control states of the incoming social agent 104.
Accordingly, in an example embodiment, simulator system 100 is configured to implement a bubble manager 108 for managing dynamic agent-vehicle assignment. Bubble manager 108 is configured to apply a “zone-based transition” methodology for managing the dynamic switching of the control of a social vehicle 124 between different social agents 104.
As used in this disclosure, “bubble” defines a region in which a specific agent-vehicle assignment holds if the social vehicle 124 is present in the region. The boundaries of a bubble are typically spatiotemporal. In some examples, a bubble may be statically defined with respect to a simulation map. In some examples, a bubble may be tied to a specific object, such as the ego vehicle 122, and move with that object through the simulation map. In some examples, the bubble can alternatively be defined by other expressible logical or functional conditions. The bubble that a social vehicle 122 is located in at a given time determines the type of social agent 104 that is to be used for its control.
In example embodiments, the types of social agents 104 are primarily specified in terms of the kinds of observations that need to be supplied from the social vehicle 124 to the social agent 104 and the kinds of actions from the social agent 104 that are expected to be executed by the social vehicle 142. In example embodiments, the bubble manager 108 is a system that is configured to manage the definition, creation, activation, updating, application (i.e. orchestration of control switch), deactivation, and destruction of bubbles during a simulation run.
In example embodiments, bubble manager 108 uses a zone-based transition method to manage the dynamic change of an agent-vehicle association and a corresponding observation and action transmission and execution. In some applications, the zone-based transition method that is described below facilitates smooth handover of control of a simulated social vehicle 124 from one social agent 104 to another social agent 104. A transition zone of a bubble is differentiated from a bubble's agent zone. The agent zone is the part of the bubble in which the intended agent-vehicle association is fully in effect. The concept of the transition zone and its use is shown in
As vehicle V enters the A=>B Transition Zone, Agent A continues to control V as before. However, Agent B will start preparing to assume control. In particular, Agent B will start receiving observations from vehicle V, execute its internal logic (e.g. apply its policy model to the received observations), and generate actions based on the received observations.
Agent B may apply a different policy model than Agent A and thus expects different observations than the observations that Agent A has been receiving from vehicle V. This could mean for example a new set of virtual sensors needs to be instantiated and appropriately initialized into the required states, which could take up to m simulation time steps. In addition to the need for initialization of the new virtual sensors, any internal states Agent B relies on to appropriately generate actions may also require multiple time steps to initialize appropriately. For example, Agent B may need to rely on the history of n time steps to accurately estimate the environment state or predict into the future and then generate an action according to the estimated state or predicted future, in which case Agent B's action will only be ready to use after n time steps. Consequently, the A=>B Transition Zone needs to be big enough to accommodate the required number of simulation steps max(m, n) that is required to appropriately prepare Agent B for the control of Vehicle V.
In example embodiments, while vehicle V is in the A=>B Transition Zone, during which Agent B starts running or operating at the same time in overlap with Agent A, the actions generated by Agent B are not used to control vehicle V. This allows Agent B time to properly initialize and time for any new sensors to be brought online. In the A=>B Transition Zone, Agent B may not be ready to control vehicle V yet. If the control is switched to B prematurely, undesirable and unnatural behavior of the vehicle V (e.g. abrupt change of direction or sudden acceleration and deceleration that is not due to environmental reality etc.) may result and thus detract from the realism of the simulation. In short, when vehicle V is in the A=>B Transition Zone, Agent A continues to control vehicle V and Agent B's action link to vehicle V is suspended or otherwise rendered ineffectual.
On the other end, as vehicle V exits B Zone to go back to A Zone, during the B=>A Transition Zone the bubble manager 108 similarly regulates: (1) observation links from vehicle V to Agents A and B, (2) action links from Agents A and B to vehicle V (with action link from Agent A to Vehicle V suspended and only Agent B controlling vehicle V), and (3) corresponding Agent A-specific initialization of sensors and computation states.
In the illustrated example, a respective transitions zone sits in between two zones (or two bubbles) to facilitate the control handover in both directions. The overall logic of such transition management using transitions zones may be summarized in a finite state machine as depicted in
In the example of
In some example embodiments, the transitions that would in the default configuration be forbidden on the basis that they skip transition zones may be permitted if the bubble manager 108 determines that the incoming agent and its required sensors and controllers require no initialization over time, is purely reactive (i.e. only react to the current observation without any regard for the recently history or possible future), or the resulted-in abrupt change is tolerable.
While both
Referring to
Unless specified otherwise, the example embodiments described below refer to logical transition zones.
The example illustrated in
The B bubble and its associated zones correspond to specific areas on the map. These areas could be specified through referencing map elements such as areas around a specific intersection, or a particular lane or road section. It could also be specified through referencing locations expressible in the coordinate system of the map.
In example embodiments, a simulator system 100 that employs bubble manager 108 may provide one or more of the following features:
Realism: The disclosed system and method may enable coherent integration and smooth handover that allows ML-based agents (trained either by imitation learning from real data or by reinforcement learning from sophisticated interaction) to be used alternately to control social vehicles, leading to more realistic interactions in the simulation.
Diversity: By allowing diverse agents to alternately control social vehicles where and when they are good at it and in spite of their differences in observation, action, internal states, history dependency, and computational dependency, simulations can be designed that have much more variability and information content (e.g. as measured by description length). Support for such diversity also opens up the possibility of crowd-sourcing agents for social vehicles.
Computing: Realistic simulation (even realism in interaction rather than sensor data) can require allot of computing resources. The disclosed system and method allows computing resources to be elastically used at giving the ego agents the most relevant experience for training, testing, evaluation, or validation.
Scalability: By adaptively devoting computing resources to only the relevant parts of the simulation, while keeping the rest of simulation at low fidelity, scaling of the simulation to larger maps and many more social vehicles may be enabled.
In the example, the B agent directs the social vehicle 124-2 to make a U turn. Transition zones (“T zones” in
In the embodiment of
Temporal bubbles could also have a global spatial scope in that a temporal bubble could cover the entire area of the simulation. For example, at 7:30 am in simulated time, all vehicles, including the ego vehicles, could shift to use the “rush-hour” versions of their corresponding agents. For another example, when the condition for raining is set, all vehicles could shift to use “rainy-weather” versions of their corresponding agents.
The use of general conditionally triggered zones with temporal boundaries (temporal on-set and off-set) can enable bubbles and transition zones be introduced where there is none. This allows dynamically creating or activating bubbles according to arbitrarily complex conditions not restricted to map regions or ego-vehicle location as in the previous two embodiments. It thus gives flexibility of adaptively changing the vehicle-agent association according to different needs. Moreover, it can also be used to globally regulate the vehicle-agent association.
As described above, the bubble manager 108 enables spatiotemporal and conditional regions (“bubbles”) to specify desired agent-vehicle assignment and to register the observation, action, computation, and initialization requirements for managing dynamic changes of the assignment.
That transition zones may be spatially specified with respect to map. That bubbles and transition zones may be spatiotemporal, may be purely time based, may be conditionally activated according to the simulation state, may travel with traffic participants, may serve as global defaults, and may be priority-managed.
In example embodiments, bubbles can be updated and applied per simulation step according to bubble specification and the simulation state. In some examples the bubble is structured in terms of transition zones and agent zones, with the transition zones being sandwiched (spatiotemporal and conditionally) between two agent zones. In example embodiments, the temporal sequence of the zones in the temporal embodiment follows the specified order: Transition Zone comes on before Incoming-Agent Zone (B Zone) and goes off after Outgoing-Agent Zone (B Zone).
The above description has focused transitioning social vehicle-social agent associations. The systems and methods described herein may also be used to transition ego agent-ego vehicle associations in some applications. For example, different ego agents 102 may be provided to control different versions of an ego vehicle that has different ego AI elements that are themselves being trained, so as to ensure realism and diversity of the ego AI element's experience in a simulation while using a reasonable amount of computing resources. Accordingly,
Although the embodiments described above have been articulated in terms of vehicle control, it could be generalized to non-vehicle traffic participants, especially pedestrians, and non-vehicle traffic actors such as traffic lights. Both pedestrians and traffic lights in a simulated environment could use complex agents to make the related interaction realistic. For example, pedestrians may behave very differently in rural areas and in urban areas, in a big crowd or alone. Likewise, the traffic light policy could change at 4 pm to get rid for the afternoon rush hour. Accordingly, the different bubble manager 108 may be applied to facilitate a transition in control between agents for any controllable object.
Further, in some examples, a transition zone could be used to manage multiple agents or controllers for the physical ego vehicle while it travels on real roads. For example, if two different sets of agent policies are used to control the ego vehicle for highway driving and city-street driving, we could use our bubble and transition zones to manage the handover to ensure a physically smooth and safe transition.
Transition zone idea could also be used in other domains either in simulation or in the real world where transition is between diverse agents with different observation types and action types.
In example embodiments, the components, modules, systems and agents included in enterprise network 110, CRM support system 120 and CRM system 200 can be implemented using one or more computer devices, servers or systems that each include a combination of a hardware processing circuit and machine-readable instructions (software and/or firmware) executable on the hardware processing circuit. A hardware processing circuit can include any or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a digital signal processor, or another hardware processing circuit.
Referring to
Operating system software 2040 executed by the processor 2004 may be stored in the persistent memory of memories 2012. A number of applications 202 executed by the processor 2004 are also stored in the persistent memory. The applications 2042 can include software instructions for implementing the systems, methods, agents and modules described above.
Although the present disclosure describes methods and processes with steps in a certain order, one or more steps of the methods and processes may be omitted or altered as appropriate. One or more steps may take place in an order other than that in which they are described, as appropriate.
Although the present disclosure is described, at least in part, in terms of methods, a person of ordinary skill in the art will understand that the present disclosure is also directed to the various components for performing at least some of the aspects and features of the described methods, be it by way of hardware components, software or any combination of the two. Accordingly, the technical solution of the present disclosure may be embodied in the form of a software product. A suitable software product may be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer readable medium, including DVDs, CD-ROMs, USB flash disk, a removable hard disk, or other storage media, for example. The software product includes instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, a server, or a network device) to execute examples of the methods disclosed herein.
The present disclosure may be embodied in other specific forms without departing from the subject matter of the claims. The described example embodiments are to be considered in all respects as being only illustrative and not restrictive. Selected features from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly described, features suitable for such combinations being understood within the scope of this disclosure.
All values and sub-ranges within disclosed ranges are also disclosed. Also, although the systems, devices and processes disclosed and shown herein may comprise a specific number of elements/components, the systems, devices and assemblies could be modified to include additional or fewer of such elements/components. For example, although any of the elements/components disclosed may be referenced as being singular, the embodiments disclosed herein could be modified to include a plurality of such elements/components. The subject matter described herein intends to cover and embrace all suitable changes in technology.
