SIMULATION-BASED TESTING FOR ROBOTIC SYSTEMS

Abstract

A directed search method is applied to a parameter space of a scenario for testing the performance of a robotic system in simulation. The directed search method is applied based multiple performance evaluation rules. A performance predictor is trained to probabilistically predict a pass or fail result for each rule at each point in the parameter space. An overall acquisition function is determined as follows: if a pass outcome is predicted at a given, the performance evaluation rule having the highest probability of an incorrect outcome prediction at determines the acquisition function; whereas, if a fail outcome is predicted at a given point for at least one rule, then the acquisition function is determined by the performance evaluation rule for which a fail outcome is predicted with the lowest probability of an incorrect outcome prediction.

Description

TECHNICAL FIELD

The present disclosure pertains to methods for evaluating the performance of trajectory planners and other robotic systems in simulated scenarios, and computer programs and systems for implementing the same. Trajectory planners are capable of autonomously planning ego trajectories for fully/semi-autonomous vehicles or other mobile robots. Example applications include ADS (Autonomous Driving System) and ADAS (Advanced Driver Assist System) performance testing.

BACKGROUND

There have been major and rapid developments in the field of autonomous vehicles. An autonomous vehicle (AV) is a vehicle which is equipped with sensors and control systems which enable it to operate without a human controlling its behavior. An autonomous vehicle is equipped with sensors which enable it to perceive its physical environment, such sensors including for example cameras, radar and lidar. Autonomous vehicles are equipped with suitably programmed computers which are capable of processing data received from the sensors and making safe and predictable decisions based on the context which has been perceived by the sensors. An autonomous vehicle may be fully autonomous (in that it is designed to operate with no human supervision or intervention, at least in certain circumstances) or semi-autonomous. Semi-autonomous systems require varying levels of human oversight and intervention, such systems including Advanced Driver Assist Systems and level three Autonomous Driving Systems. There are different facets to testing the behavior of the sensors and control systems aboard a particular autonomous vehicle, or a type of autonomous vehicle. Other mobile robots are being developed, for example for carrying freight supplies in internal and external industrial zones. Such mobile robots would have no people on board and belong to a class of mobile robot termed UAV (unmanned autonomous vehicle). Autonomous air mobile robots (drones) are also being developed.

In autonomous driving, the importance of guaranteed safety has been recognized. Guaranteed safety does not necessarily imply zero accidents, but rather means guaranteeing that some minimum level of safety is met in defined circumstances. It is generally assumed this minimum level of safety must significantly exceed that of human drivers for autonomous driving to be viable. Thus, the testing requirements for autonomous vehicles are especially challenging.

Sensor processing may be evaluated in real-world physical facilities. Similarly, the control systems for autonomous vehicles may be tested in the physical world, for example by repeatedly driving known test routes, or by driving routes with a human on-board to manage unpredictable or unknown context. Physical world testing will remain an important factor in the testing of autonomous vehicles capability to make safe and predictable decisions. However, physical world testing is expensive and time-consuming. Increasingly, there is more reliance placed on testing using simulated environments.

Simulation-based testing comes with its own challenges. A high-fidelity simulation requires significant computational resources. As the safety of autonomous systems increases, the number of required simulation instances also increases. For example, human driving is estimated to cause of the order 10⁻⁶ severe accidents per hour. It has been suggested that autonomous driving systems will need to reduce this by at least three orders of magnitude to be accepted in society, implying a minimum safety level of the order of 10⁻⁹ severe accidents per hour needs to be guaranteed. For an autonomous driving system operating at this level, a ‘naïve’ simulation strategy would therefore require of the order of 10⁹ simulated driving hours on average to encounter a single severe accident. There is therefore a need to greatly increase the efficiency of simulation-based testing.

For mobile robot testing, a test scenario includes a simulated mobile robot (the ego agent) and some physical context that the ego agent is required to navigate. The physical context of a driving scenario generally includes a road layout and may include one or more ‘challenger’ objects (static objects and/or dynamic agents, such as vehicles, pedestrians, cyclists, animals etc.). An instance of the rest scenario is realized in a simulation environment, with the robotic system under testing in control of the ego agent. The performance of the robot system may be assessed in terms of the behavior of the ego agent in the scenario instance and/or in terms of decisions taken internally within the robotic system over the course of the simulation. The robotic system has knowledge of its location at any instant of time, understands its context, and is required to make safe and predictable decisions about how to navigate its environment to reach a pre-programmed destination.

Simulation-based testing may utilize parameterized scenarios, where different instances (or ‘runs’) of the scenario may be realized in a simulation environment with different parameter combinations. It is useful to conceptualize a set of parameter values as a point in a parameter space of the scenario. A scenario is ‘run’ (instantiated) at a given point in the parameter space when an instance of the scenario is realized in a simulation environment with the corresponding set of parameter values.

One approach is based on random sampling of points in the scenario's parameter space, for example using a Monte Carlos search strategy. However, this suffers from the drawbacks noted above. ‘Directed testing’ or ‘directed exploration’ approaches are more targeted, and use the outcomes of previous simulation to guide a subsequent search of the parameter space, with the aim of finding more salient parameter combinations with fewer simulation instances. The search may commence with an initial set of points in the parameter space (e.g. uniformly spaced, or selected via Monte Carlo sampling) and the scenario is instantiated at each of those points. The performance of the robotic system is assessed in each scenario, and the results are used to guide the selection of subsequent points in the parameter space.

Directed testing approaches use some defined concept of failure, with the extent of ‘failure’ quantified via an appropriate numerical metric. For example, in a driving context, a simple failure metric might be forward distance (longitudinal distance between the ego agent and a forward vehicle ahead of the ego agent). A failure event occurs when the forward distance falls below some safety threshold, with the severity of the failure increasing as the forward distance decreases. Most directed exploration approaches target the search of the parameter space on the ‘worst’ (most severe) failure instances. A directed testing method uses the performance analysis from previous scenario instances to predict the system's performance at other points in the parameter space, and uses those predictions to guide the selection of subsequent points. In the preceding example, the directed search would attempt to guide the search to region(s) of the parameter space where the forward distance is predicted to be minimized (such that failure on the forward distance metric is maximized).

SUMMARY

A core aim herein is one of providing improved directed testing of robotic systems, in the sense of maximizing the saliency of the results for a given number of simulated runs (or, equivalently, minimizing the number of runs that are required to achieve a given level of saliency).

A directed search method is applied to a parameter space of a scenario for testing the performance of a robotic system in simulation. The directed search method is applied based multiple performance evaluation rules. A performance predictor is trained to probabilistically predict a pass or fail result for each rule at each point in the parameter space (other possible outcomes are also considered). An overall acquisition function is determined as follows: if a pass outcome is predicted at a given, the performance evaluation rule having the highest probability of an incorrect outcome prediction at x determines the acquisition function; whereas, if a fail outcome is predicted at a given point for at least one rule, then the acquisition function is determined by the performance evaluation rule for which a fail outcome is predicted with the lowest probability of an incorrect outcome prediction.

A first aspect herein is directed to computer-implemented method of testing, in simulation, performance of a robotic system in control of an ego agent of a test scenario, the method comprising: determining a first set of points in a parameter space of the test scenario, each point being a set of one or more parameter values for the test scenario; based the first set of points, generating in a simulation environment multiple first instances of the test scenario with the robotic system in control of the ego agent; assigning to each point of the first set a plurality of performance indicators based on a plurality of performance evaluation rules, thereby generating a training set of points and their assigned pluralities of performance indicators, wherein each performance indicator denotes a pass outcome or a fail outcome; using the training set to train a performance predictor for probabilistically predicting at each point x in the parameter space of the test scenario a pass or fail outcome for each performance evaluation rule; using the trained performance predictor to determine a plurality of rule acquisition functions, each rule acquisition function ƒ_i(s) denoting at each point x in the parameter space a probability of an incorrect outcome prediction for a performance evaluation rule i of the plurality of performance evaluation rules; selecting one or more second points based on an overall acquisition function defined as ƒ(x)=ƒ_j(x), wherein if a pass outcome is predicted at a given point x for every rule i then j is the performance evaluation rule having the highest probability of an incorrect outcome prediction at x, and wherein if a fail outcome is predicted at a given point x for at least one rule i, then j is the performance evaluation rule for which a fail outcome is predicted at x with the lowest probability of an incorrect outcome prediction; based on the one or more second points in the parameter space, generating in a simulation environment one or more second instances of the test scenario with the robotic system in control of the ego agent; and providing one or more outputs for evaluating performance of the robotic system in the one or more second instances based on the at least one predetermined performance evaluation rule.

In embodiments, the one or more outputs may be rendered in a graphical user interface.

The one or more outputs may comprise one or more performance scores assigned to each second point for the at least one performance evaluation rule.

The method may comprise updating the performance predictor based on the one or more second points and the one or more performance scores assigned thereto.

The method may comprise: selecting one or more third points in the parameter space based on an updated overall acquisition function defined by the updated performance predictor; based the one or more third points in the parameter space, generating in a simulation environment one or more third instances of the test scenario with the robotic system in control of the ego agent; and providing one or more second outputs for evaluating performance of the robotic system in the one or more third instances based on the at least one predetermined performance evaluation rule.

The one or more second outputs may comprise one or more performance scores assigned to the one or more third points, and the method is repeated iteratively until the performance predictor satisfies a termination condition, or a predetermined number of iterations is reached.

The performance predictor may comprise a Gaussian score prediction model for each performance prediction rule.

The score prediction model may provide a mean performance score g_i(x) and standard deviation g_σ,i(x) for each rule, i, at a given point x in the parameter space, wherein the rule acquisition function for rule i is based on g_i(x)/g_σ,i(x).

The robotic system may comprise a trajectory planner for a mobile robot.

The method may comprise using the score classification model and the score regression model to identify and mitigate an issue with the robotic system.

Further aspects herein provide a computer system configured to implement the method of any of the above aspects or embodiments, and computer program code configured to program the computer system implement the same.

BRIEF DESCRIPTION OF FIGURES

Particular embodiments will now be described, by way of example only, with reference to the following schematic figures, in which:

FIG. 1A shows a schematic function block diagram of an autonomous vehicle stack;

FIG. 1B shows a schematic overview of an autonomous vehicle testing paradigm;

FIG. 1C shows a schematic block diagram of a scenario extraction pipeline;

FIG. 2 shows a schematic block diagram of a testing pipeline;

FIG. 2A shows further details of a possible implementation of the testing pipeline;

FIG. 3A schematically illustrates certain geometric principals of a “safe distance” rule that applies to a lane follow context in autonomous vehicle testing;

FIG. 3B shows a computational graph for a safe distance robustness score;

FIG. 4A shows a schematic block diagram of a system in which a scoring function is used to generate a training set for a score predictor;

FIG. 4B shows a flowchart for a first directed testing method;

FIG. 4C shows a probabilistic score predictor;

FIG. 4D shows a flowchart for a second directed testing method with multiple performance evaluation rules;

FIG. 4E schematically illustrates the selection of a point in a scenario parameter space based on an acquisition function;

FIGS. 5A-5C illustrate how overall robustness scores may be assigned to different instances of a scenario with different overall outcomes;

FIG. 6 shows a two-dimensional parameter space divided into pass and fail regions;

FIG. 7 shows an alternative robustness scoring methodology that extends to non-numerical robustness scores;

FIG. 8 shows a two-dimensional parameter space divided into pass, fail regions and “Not a Number” (NaN) regions;

FIG. 9A shows a hierarchical score predictor comprising a probabilistic classification model and a probabilistic regression model; and

FIG. 9B shows a flow chart for a third directed testing method applied with NaN-based robustness scoring.

DETAILED DESCRIPTION

The described embodiments consider performance testing of a robotic system in the form of an autonomous vehicle stack, or some component (or components) thereof, in parameterized driving scenarios. As will be appreciated, the described techniques are applicable to other forms of robotic systems when tested in simulation based on one or more parameterized test scenarios.

As outlined above, in the field of robotics, directed exploration methods have generally targeted ‘worst-case’ failures. Such methods can be highly effective in quickly locating failure cases (or, more precisely, in reducing the number of non-salient points in the scenario space that need to be considered before a failure case is found. A non-salient point in this context generally means a set of parameter values that do not result in a failure event for the robotic system under testing). However, there are significant drawbacks to this approach. In certain contexts, the worst-case failure is not necessarily the most interesting or salient. For example, in a driving context, there may be certain variations of a scenario in which a collision or other failure event is unavoidable (generally this implies, at the very least, a set of circumstances such that no human driver could reasonably be expected to avoid a collision). A directed exploration approach targeting worst-case failure can easily guide the search to ‘uninteresting’ failure cases. For example, in a cut-in scenario, such a method could, for example, guide the search to a set of parameter values at which another agent cuts in front of the ego agent with virtually no headway, and immediately decelerates. An expensive simulation to verify that the system under testing fails in those circumstances may be entirely redundant. On the other hand, if the system under testing ‘just’ fails in circumstances where it ought to have comfortably passed, that may be far more indicative of a genuine issue in the robotic system that needs to be identified and mitigated.

In the described embodiments, an alternative directed exploration approach is considered. Rather than identifying worst-case failures, the aim is to map out one or more “performance category boundaries” in a scenario's parameter space with sufficient confidence as efficiently as possible (in terms of the number of simulated instances of the scenario that need to be realized).

With binary {PASS, FAIL} performance categories, this equates to estimating the boundary (or boundaries) between pass/fail regions of the scenario's parameter space. Points either side of such a boundary correspond to parameter combinations (variations of the scenario) where the robotic system ‘just’ passes or ‘just’ fails on some predetermined numerical performance metric (or metrics), referred to herein as rules. One form of rule returns a floating point (or other numerical) value (robustness score) at every time step in a scenario, and a pass/fail category is assigned by applying a pass/fail threshold to the robustness score. The result is a time-sequence of pass/fail results over the course of the scenario. These results can be aggregated to provide an overall pass/fail result for a given scenario instance and a given rule (e.g. with an overall fail result if and only if the rule is failed at any point in the scenario). The following examples consider robustness scores that are normalized to a fixed range and failure threshold. The range is chosen as [−1, 1] with the failure threshold at zero. This is a convent design choice that makes the scores readily interpretable, but it will be appreciated that robustness scores can be represented in other ways.

Herein, a ‘performance indicator’ can be a performance class (e.g. PASS, FAIl or “NOT APPLICABLE”—see below), or a robustness score that indicates a performance class (e.g. a numerical score above a threshold might indicate PASS, and FAIL below the threshold).

However, the described embodiments go beyond binary categorization, and introduce a third “INACTIVE” performance category. As described in further detail below, this third performance category is introduced to accommodate rules that are not guaranteed to return a floating point robustness score at every time step, but which might return a ‘Not a Number’ (NaN) at a given time step. In formal terms, a NaN means a non-numerical output of a numerical function. NaNs have been used, for example, to handle values that are undefined in the mathematical sense (such as the logarithm of a negative number). In the present context, NaN's are introduced as a deliberate design choice, to better reflect situations where a rule is simply not applicable. For example, a ‘safe distance’ rule based on longitudinal distance between an ego agent and a forward vehicle does not apply in the case that the ego agent is waiting at a T-junction to turn left or right.

Constructing the rules with NaN outputs makes the scores more informative. However, the benefits go further than this. As described in detail below, NaN outputs are incorporated into the directed search process using separate prediction models for Nan/not-NaN classification and numerical score prediction. This formulation is not only accommodating of non-numerical robustness outputs, but also mitigates issues that could prevent the directed search from converging as intended (this is explained in detail below).

The described directed testing method are incorporated in a testing pipeline to facilitate rules-based testing of mobile robot stacks simulated scenarios. The testing pipeline can also be applied to real scenarios, in combination with simulation-based testing. Agent (actor) behaviour in real or simulated scenarios is evaluated by a test oracle based on defined performance evaluation rules. Such rules may evaluate different facets of safety. For example, a safety rule set may be defined to assess the performance of the stack against a particular safety standard, regulation or safety model (such as RSS), or bespoke rule sets may be defined for testing any aspect of performance. The testing pipeline is not limited in its application to safety, and can be used to test any aspects of performance, such as comfort or progress towards some defined goal. A rule editor allows performance evaluation rules to be defined or modified and passed to the test oracle.

A “full” stack typically involves everything from processing and interpretation of low-level sensor data (perception), feeding into primary higher-level functions such as prediction and planning, as well as control logic to generate suitable control signals to implement planning-level decisions (e.g. to control braking, steering, acceleration etc.). For autonomous vehicles, level 3 stacks include some logic to implement transition demands and level 4 stacks additionally include some logic for implementing minimum risk maneuvers. The stack may also implement secondary control functions e.g. of signalling, headlights, windscreen wipers etc.

The term “stack” can also refer to individual sub-systems (sub-stacks) of the full stack, such as perception, prediction, planning or control stacks, which may be tested individually or in any desired combination. A stack can refer purely to software, i.e. one or more computer programs that can be executed on one or more general-purpose computer processors.

Whether real or simulated, a scenario requires an ego agent to navigate a real or modelled physical context. The ego agent is a real or simulated mobile robot that moves under the control of the stack under testing. The physical context includes static and/or dynamic element(s) that the stack under testing is required to respond to effectively. For example, the mobile robot may be a fully or semi-autonomous vehicle under the control of the stack (the ego vehicle). The physical context may comprise a static road layout and a given set of environmental conditions (e.g. weather, time of day, lighting conditions, humidity, pollution/particulate level etc.) that could be maintained or varied as the scenario progresses. An interactive scenario additionally includes one or more other agents (“external” agent(s), e.g. other vehicles, pedestrians, cyclists, animals etc.).

The following examples consider applications to autonomous vehicle testing. However, the principles apply equally to other forms of mobile robot or robotic system.

Scenarios may be represented or defined at different levels of abstraction. More abstracted scenarios accommodate a greater degree of variation. For example, a “cut-in scenario” or a “lane change scenario” are examples of highly abstracted scenarios, characterized by a maneuver or behaviour of interest, that accommodate many variations (e.g. different agent starting locations and speeds, road layout, environmental conditions etc.). A “scenario run” refers to a concrete occurrence of an agent(s) navigating a physical context, optionally in the presence of one or more other agents. For example, multiple runs of a cut-in or lane change scenario could be performed (in the real-world and/or in a simulator) with different agent parameters (e.g. starting location, speed etc.), different road layouts, different environmental conditions, and/or different stack configurations etc. The terms “run” and “instance” are used interchangeably in this context. Herein, a scenario has a minimum of one adjustable parameter such that salient variations of the scenario may be explored using directed testing to iteratively select different values of the scenario parameter (or parameters). A set of (one or more) parameters is referred to, for conciseness, as a point in the parameter space of the scenario.

In the following examples, the performance of the stack is assessed, at least in part, by evaluating the behaviour of the ego agent in the test oracle against a given set of performance evaluation rules, over the course of one or more runs. The rules are applied to “ground truth” of the (or each) scenario run which, in general, simply means an appropriate representation of the scenario run (including the behaviour of the ego agent) that is taken as authoritative for the purpose of testing. Ground truth is inherent to simulation; a simulator computes a sequence of scenario states, which is, by definition, a perfect, authoritative representation of the simulated scenario run. In a real-world scenario run, a “perfect” representation of the scenario run does not exist in the same sense; nevertheless, suitably informative ground truth can be obtained in numerous ways, e.g. based on manual annotation of on-board sensor data, automated/semi-automated annotation of such data (e.g. using offline/non-real time processing), and/or using external information sources (such as external sensors, maps etc.) etc.

The scenario ground truth typically includes a “trace” of the ego agent and any other (salient) agent(s) as applicable. A trace is a history of an agent's location and motion over the course of a scenario. There are many ways a trace can be represented. Trace data will typically include spatial and motion data of an agent within the environment. The term is used in relation to both real scenarios (with real-world traces) and simulated scenarios (with simulated traces). The trace typically records an actual trajectory realized by the agent in the scenario. With regards to terminology, a “trace” and a “trajectory” may contain the same or similar types of information (such as a series of spatial and motion states over time). The term trajectory is generally favoured in the context of planning (and can refer to future/predicted trajectories), whereas the term trace is generally favoured in relation to past behaviour in the context of testing/evaluation.

In a simulation context, a “scenario description” is provided to a simulator as input. For example, a scenario description may be encoded using a scenario description language (SDL), or in any other form that can be consumed by a simulator. A scenario description is typically a more abstract representation of a scenario, that can give rise to multiple simulated runs. Depending on the implementation, a scenario description may have one or more configurable parameters that can be varied to increase the degree of possible variation. The degree of abstraction and parameterization is a design choice. For example, a scenario description may encode a fixed layout, with parameterized environmental conditions (such as weather, lighting etc.). Further abstraction is possible, however, e.g. with configurable road parameter(s) (such as road curvature, lane configuration etc.). The input to the simulator comprises the scenario description together with a chosen set of parameter value(s) (as applicable). The latter may be referred to as a parameterization of the scenario. The configurable parameter(s) define a parameter space (also referred to as the scenario space), and the parameterization corresponds to a point in the parameter space. In this context, a “scenario instance” may refer to an instantiation of a scenario in a simulator based on a scenario description and (if applicable) a chosen parameterization.

For conciseness, the term scenario may also be used to refer to a scenario run, as well a scenario in the more abstracted sense. The meaning of the term scenario will be clear from the context in which it is used.

Trajectory planning is an important function in the present context, and the terms “trajectory planner”, “trajectory planning system” and “trajectory planning stack” may be used interchangeably herein to refer to a component or components that can plan trajectories for a mobile robot into the future. Trajectory planning decisions ultimately determine the actual trajectory realized by the ego agent (although, in some testing contexts, this may be influenced by other factors, such as the implementation of those decisions in the control stack, and the real or modelled dynamic response of the ego agent to the resulting control signals).

A trajectory planner may be tested in isolation, or in combination with one or more other systems (e.g. perception, prediction and/or control). Within a full stack, planning generally refers to higher-level autonomous decision-making capability (such as trajectory planning), whilst control generally refers to the lower-level generation of control signals for carrying out those autonomous decisions. However, in the context of performance testing, the term control is also used in the broader sense. For the avoidance of doubt, when a trajectory planner is said to control an ego agent in simulation, that does not necessarily imply that a control system (in the narrower sense) is tested in combination with the trajectory planner.

Example AV Stack:

To provide relevant context to the described embodiments, further details of an example form of AV stack will now be described.

FIG. 1A shows a highly schematic block diagram of an AV runtime stack 100. The run time stack 100 is shown to comprise a perception (sub-)system 102, a prediction (sub-)system 104, a planning (sub-)system (planner) 106 and a control (sub-)system (controller) 108. As noted, the term (sub-)stack may also be used to describe the aforementioned components 102-108.

In a real-world context, the perception system 102 receives sensor outputs from an on-board sensor system 110 of the AV, and uses those sensor outputs to detect external agents and measure their physical state, such as their position, velocity, acceleration etc. The on-board sensor system 110 can take different forms but generally comprises a variety of sensors such as image capture devices (cameras/optical sensors), lidar and/or radar unit(s), satellite-positioning sensor(s) (GPS etc.), motion/inertial sensor(s) (accelerometers, gyroscopes etc.) etc. The onboard sensor system 110 thus provides rich sensor data from which it is possible to extract detailed information about the surrounding environment, and the state of the AV and any external actors (vehicles, pedestrians, cyclists etc.) within that environment. The sensor outputs typically comprise sensor data of multiple sensor modalities such as stereo images from one or more stereo optical sensors, lidar, radar etc. Sensor data of multiple sensor modalities may be combined using filters, fusion components etc.

The perception system 102 typically comprises multiple perception components which co-operate to interpret the sensor outputs and thereby provide perception outputs to the prediction system 104.

In a simulation context, depending on the nature of the testing—and depending, in particular, on where the stack 100 is “sliced” for the purpose of testing (see below)—it may or may not be necessary to model the on-board sensor system 100. With higher-level slicing, simulated sensor data is not required therefore complex sensor modelling is not required.

The perception outputs from the perception system 102 are used by the prediction system 104 to predict future behaviour of external actors (agents), such as other vehicles in the vicinity of the AV.

Predictions computed by the prediction system 104 are provided to the planner 106, which uses the predictions to make autonomous driving decisions to be executed by the AV in a given driving scenario. The inputs received by the planner 106 would typically indicate a drivable area and would also capture predicted movements of any external agents (obstacles, from the AV's perspective) within the drivable area. The driveable area can be determined using perception outputs from the perception system 102 in combination with map information, such as an HD (high definition) map.

A core function of the planner 106 is the planning of trajectories for the AV (ego trajectories), taking into account predicted agent motion. This may be referred to as trajectory planning. A trajectory is planned in order to carry out a desired goal within a scenario. The goal could for example be to enter a roundabout and leave it at a desired exit; to overtake a vehicle in front; or to stay in a current lane at a target speed (lane following). The goal may, for example, be determined by an autonomous route planner (not shown).

The controller 108 executes the decisions taken by the planner 106 by providing suitable control signals to an on-board actor system 112 of the AV. In particular, the planner 106 plans trajectories for the AV and the controller 108 generates control signals to implement the planned trajectories. Typically, the planner 106 will plan into the future, such that a planned trajectory may only be partially implemented at the control level before a new trajectory is planned by the planner 106. The actor system 112 includes “primary” vehicle systems, such as braking, acceleration and steering systems, as well as secondary systems (e.g. signalling, wipers, headlights etc.).

Note, there may be a distinction between a planned trajectory at a given time instant, and the actual trajectory followed by the ego agent. Planning systems typically operate over a sequence of planning steps, updating the planned trajectory at each planning step to account for any changes in the scenario since the previous planning step (or, more precisely, any changes that deviate from the predicted changes). The planning system 106 may reason into the future, such that the planned trajectory at each planning step extends beyond the next planning step. Any individual planned trajectory may, therefore, not be fully realized (if the planning system 106 is tested in isolation, in simulation, the ego agent may simply follow the planned trajectory exactly up to the next planning step; however, as noted, in other real and simulation contexts, the planned trajectory may not be followed exactly up to the next planning step, as the behaviour of the ego agent could be influenced by other factors, such as the operation of the control system 108 and the real or modelled dynamics of the ego vehicle). In many testing contexts, the actual trajectory of the ego agent is what ultimately matters; in particular, whether the actual trajectory is safe, as well as other factors such as comfort and progress. However, the rules-based testing approach herein can also be applied to planned trajectories (even if those planned trajectories are not fully or exactly realized by the ego agent). For example, even if the actual trajectory of an agent is deemed safe according to a given set of safety rules, it might be that an instantaneous planned trajectory was unsafe; the fact that the planner 106 was considering an unsafe course of action may be revealing, even if it did not lead to unsafe agent behaviour in the scenario. Instantaneous planned trajectories constitute one form of internal state that can be usefully evaluated, in addition to actual agent behaviour in the simulation. Other forms of internal stack state can be similarly evaluated.

The example of FIG. 1A considers a relatively “modular” architecture, with separable perception, prediction, planning and control systems 102-108. The sub-stack themselves may also be modular, e.g. with separable planning modules within the planning system 106. For example, the planning system 106 may comprise multiple trajectory planning modules that can be applied in different physical contexts (e.g. simple lane driving vs. complex junctions or roundabouts). This is relevant to simulation testing for the reasons noted above, as it allows components (such as the planning system 106 or individual planning modules thereof) to be tested individually or in different combinations. For the avoidance of doubt, with modular stack architectures, the term stack can refer not only to the full stack but to any individual sub-system or module thereof.

The extent to which the various stack functions are integrated or separable can vary significantly between different stack implementations—in some stacks, certain aspects may be so tightly coupled as to be indistinguishable. For example, in other stacks, planning and control may be integrated (e.g. such stacks could plan in terms of control signals directly), whereas other stacks (such as that depicted in FIG. 1A) may be architected in a way that draws a clear distinction between the two (e.g. with planning in terms of trajectories, and with separate control optimizations to determine how best to execute a planned trajectory at the control signal level). Similarly, in some stacks, prediction and planning may be more tightly coupled. At the extreme, in so-called “end-to-end” driving, perception, prediction, planning and control may be essentially inseparable. Unless otherwise indicated, the perception, prediction planning and control terminology used herein does not imply any particular coupling or modularity of those aspects.

It will be appreciated that the term “stack” encompasses software, but can also encompass hardware. In simulation, software of the stack may be tested on a “generic” off-board computer system, before it is eventually uploaded to an on-board computer system of a physical vehicle. However, in “hardware-in-the-loop” testing, the testing may extend to underlying hardware of the vehicle itself. For example, the stack software may be run on the on-board computer system (or a replica thereof) that is coupled to the simulator for the purpose of testing. In this context, the stack under testing extends to the underlying computer hardware of the vehicle. As another example, certain functions of the stack 110 (e.g. perception functions) may be implemented in dedicated hardware. In a simulation context, hardware-in-the loop testing could involve feeding synthetic sensor data to dedicated hardware perception components.

Testing Paradigm:

FIG. 1B shows a highly schematic overview of a testing paradigm for autonomous vehicles. An ADS/ADAS stack 100, e.g. of the kind depicted in FIG. 1A, is subject to repeated testing and evaluation in simulation, by running multiple scenario instances in a simulator 202, and evaluating the performance of the stack 100 (and/or individual subs-stacks thereof) in a test oracle 252. The output of the test oracle 252 is informative to an expert 122 (team or individual), allowing them to identify issues in the stack 100 and modify the stack 100 to mitigate those issues (S124). The results also assist the expert 122 in selecting further scenarios for testing (S126), and the process continues, repeatedly modifying, testing and evaluating the performance of the stack 100 in simulation. The improved stack 100 is eventually incorporated (S125) in a real-world AV 101, equipped with a sensor system 110 and an actor system 112. The improved stack 100 typically includes program instructions (software) executed in one or more computer processors of an on-board computer system of the vehicle 101 (not shown). The software of the improved stack is uploaded to the AV 101 at step S125. Step S125 may also involve modifications to the underlying vehicle hardware. On board the AV 101, the improved stack 100 receives sensor data from the sensor system 110 and outputs control signals to the actor system 112. Real-world testing (S128) can be used in combination with simulation-based testing. For example, having reached an acceptable level of performance though the process of simulation testing and stack refinement, appropriate real-world scenarios may be selected (S130), and the performance of the AV 101 in those real scenarios may be captured and similarly evaluated in the test oracle 252.

Scenarios can be obtained for the purpose of simulation in various ways, including manual encoding. The system is also capable of extracting scenarios for the purpose of simulation from real-world runs, allowing real-world situations and variations thereof to be re-created in the simulator 202.

FIG. 1C shows a highly schematic block diagram of a scenario extraction pipeline. Data 140 of a real-world run is passed to a ‘ground-truthing’ pipeline 142 for the purpose of generating scenario ground truth. The run data 140 could comprise, for example, sensor data and/or perception outputs captured/generated on board one or more vehicles (which could be autonomous, human-driven or a combination thereof), and/or data captured from other sources such external sensors (CCTV etc.). The run data is processed within the ground truthing pipeline 142, in order to generate appropriate ground truth 144 (trace(s) and contextual data) for the real-world run. As discussed, the ground-truthing process could be based on manual annotation of the ‘raw’ run data 140, or the process could be entirely automated (e.g. using offline perception method(s)), or a combination of manual and automated ground truthing could be used. For example, 3D bounding boxes may be placed around vehicles and/or other agents captured in the run data 140, in order to determine spatial and motion states of their traces. A scenario extraction component 146 receives the scenario ground truth 144, and processes the scenario ground truth 144 to extract a more abstracted scenario description 148 that can be used for the purpose of simulation. The scenario description 148 is consumed by the simulator 202, allowing multiple simulated runs to be performed. The simulated runs are variations of the original real-world run, with the degree of possible variation determined by the extent of abstraction. Ground truth 150 is provided for each simulated run.

In the present off-board content, there is no requirement for the traces to be extracted in real-time (or, more precisely, no need for them to be extracted in a manner that would support real-time planning); rather, the traces are extracted “offline”. Examples of offline perception algorithms include non-real time and non-causal perception algorithms. Offline techniques contrast with “on-line” techniques that can feasibly be implemented within an AV stack 100 to facilitate real-time planning/decision making.

For example, it is possible to use non-real time processing, which cannot be performed on-line due to hardware or other practical constraints of an AV's onboard computer system. For example, one or more non-real time perception algorithms can be applied to the real-world run data 140 to extract the traces. A non-real time perception algorithm could be an algorithm that it would not be feasible to run in real time because of the computation or memory resources it requires.

It is also possible to use “non-causal” perception algorithms in this context. A non-causal algorithm may or may not be capable of running in real-time at the point of execution, but in any event could not be implemented in an online context, because it requires knowledge of the future. For example, a perception algorithm that detects an agent state (e.g. location, pose, speed etc.) at a particular time instant based on subsequent data could not support real-time planning within the stack 100 in an on-line context, because it requires knowledge of the future (unless it was constrained to operate with a short look ahead window). For example, filtering with a backwards pass is a non-causal algorithm that can sometimes be run in real-time, but requires knowledge of the future.

The term “perception” generally refers to techniques for perceiving structure in the real-world data 140, such as 2D or 3D bounding box detection, location detection, pose detection, motion detection etc. For example, a trace may be extracted as a time-series of bounding boxes or other spatial states in 3D space or 2D space (e.g. in a birds-eye-view frame of reference), with associated motion information (e.g. speed, acceleration, jerk etc.). In the context of image processing, such techniques are often classed as “computer vision”, but the term perception encompasses a broader range of sensor modalities.

Testing Pipeline:

Further details of the testing pipeline and the test oracle 252 will now be described. The examples that follow focus on simulation-based testing. However, as noted, the test oracle 252 can equally be applied to evaluate stack performance on real scenarios, and the relevant description below applies equally to real scenarios. The following description refers to the stack 100 of FIG. 1A by way of example. However, as noted, the testing pipeline 200 is highly flexible and can be applied to any stack or sub-stack operating at any level of autonomy.

FIG. 2 shows a schematic block diagram of the testing pipeline, denoted by reference numeral 200. The testing pipeline 200 is shown to comprise the simulator 202 and the test oracle 252. The simulator 202 runs simulated scenarios for the purpose of testing all or part of an AV run time stack 100, and the test oracle 252 evaluates the performance of the stack (or sub-stack) on the simulated scenarios. As discussed, it may be that only a sub-stack of the run-time stack (or some portion or portions thereof) is tested, but for simplicity, the following description refers to the (full) AV stack 100 throughout. However, the description applies equally to a sub-stack in place of the full stack 100. The term “slicing” is used herein to the selection of a set or subset of stack components for testing.

As described previously, the idea of simulation-based testing is to run a simulated driving scenario that an ego agent must navigate under the control of the stack 100 being tested. Typically, the scenario includes a static drivable area (e.g. a particular static road layout) that the ego agent is required to navigate, typically in the presence of one or more other dynamic agents (such as other vehicles, bicycles, pedestrians etc.). To this end, simulated inputs 203 are provided from the simulator 202 to the stack 100 under testing.

The slicing of the stack dictates the form of the simulated inputs 203. By way of example, FIG. 2 shows the prediction, planning and control systems 104, 106 and 108 within the AV stack 100 being tested. To test the full AV stack of FIG. 1A, the perception system 102 could also be applied during testing. In this case, the simulated inputs 203 would comprise synthetic sensor data that is generated using appropriate sensor model(s) and processed within the perception system 102 in the same way as real sensor data. This requires the generation of sufficiently realistic synthetic sensor inputs (such as photorealistic image data and/or equally realistic simulated lidar/radar data etc.). The resulting outputs of the perception system 102 would, in turn, feed into the higher-level prediction and planning systems 104, 106.

By contrast, so-called “planning-level” simulation would essentially bypass the perception system 102. The simulator 202 would instead provide simpler, higher-level inputs 203 directly to the prediction system 104. In some contexts, it may even be appropriate to bypass the prediction system 104 as well, in order to test the planner 106 on predictions obtained directly from the simulated scenario (i.e. “perfect” predictions).

Between these extremes, there is scope for many different levels of input slicing, e.g. testing only a subset of the perception system 102, such as “later” (higher-level) perception components, e.g. components such as filters or fusion components which operate on the outputs from lower-level perception components (such as object detectors, bounding box detectors, motion detectors etc.).

Whatever form they take, the simulated inputs 203 are used (directly or indirectly) as a basis for decision-making by the planner 108. The controller 108, in turn, implements the planner's decisions by outputting control signals 109. In a real-world context, these control signals would drive the physical actor system 112 of AV. In simulation, an ego vehicle dynamics model 204 is used to translate the resulting control signals 109 into realistic motion of the ego agent within the simulation, thereby simulating the physical response of an autonomous vehicle to the control signals 109.

Alternatively, a simpler form of simulation assumes that the ego agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 108 (to the extent it is separable from planning) and removes the need for the ego vehicle dynamic model 204. This may be sufficient for testing certain facets of planning.

To the extent that external agents exhibit autonomous behaviour/decision making within the simulator 202, some form of agent decision logic 210 is implemented to carry out those decisions and determine agent behaviour within the scenario. The agent decision logic 210 may be comparable in complexity to the ego stack 100 itself or it may have a more limited decision-making capability. The aim is to provide sufficiently realistic external agent behaviour within the simulator 202 to be able to usefully test the decision-making capabilities of the ego stack 100. In some contexts, this does not require any agent decision making logic 210 at all (open-loop simulation), and in other contexts useful testing can be provided using relatively limited agent logic 210 such as basic adaptive cruise control (ACC). One or more agent dynamics models 206 may be used to provide more realistic agent behaviour if appropriate.

A scenario is run in accordance with a scenario description 201a and (if applicable) a chosen parameterization 201b of the scenario (generally denoted by x herein). A scenario typically has both static and dynamic elements which may be “hard coded” in the scenario description 201a or configurable and thus determined by the scenario description 201a in combination with a chosen parameterization 201b. In a driving scenario, the static element(s) typically include a static road layout.

The dynamic element(s) typically include one or more external agents within the scenario, such as other vehicles, pedestrians, bicycles etc.

The extent of the dynamic information provided to the simulator 202 for each external agent can vary. For example, a scenario may be described by separable static and dynamic layers. A given static layer (e.g. defining a road layout) can be used in combination with different dynamic layers to provide different scenario instances. The dynamic layer may comprise, for each external agent, a spatial path to be followed by the agent together with one or both of motion data and behaviour data associated with the path. In simple open-loop simulation, an external actor simply follows the spatial path and motion data defined in the dynamic layer that is non-reactive i.e. does not react to the ego agent within the simulation. Such open-loop simulation can be implemented without any agent decision logic 210. However, in closed-loop simulation, the dynamic layer instead defines at least one behaviour to be followed along a static path (such as an ACC behaviour). In this case, the agent decision logic 210 implements that behaviour within the simulation in a reactive manner, i.e. reactive to the ego agent and/or other external agent(s). Motion data may still be associated with the static path but in this case is less prescriptive and may for example serve as a target along the path. For example, with an ACC behaviour, target speeds may be set along the path which the agent will seek to match, but the agent decision logic 210 might be permitted to reduce the speed of the external agent below the target at any point along the path in order to maintain a target headway from a forward vehicle.

As will be appreciated, scenarios can be described for the purpose of simulation in many ways, with any degree of configurability. For example, the number and type of agents, and their motion information may be configurable as part of the scenario parameterization 201b.

The output of the simulator 202 for a given simulation includes an ego trace 212a of the ego agent and one or more agent traces 212b of the one or more external agents (traces 212). Each trace 212a, 212b is a complete history of an agent's behaviour within a simulation having both spatial and motion components. For example, each trace 212a, 212b may take the form of a spatial path having motion data associated with points along the path such as speed, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk) etc.

Additional information is also provided to supplement and provide context to the traces 212. Such additional information is referred to as “contextual” data 214. The contextual data 214 pertains to the physical context of the scenario, and can have both static components (such as road layout) and dynamic components (such as weather conditions to the extent they vary over the course of the simulation). To an extent, the contextual data 214 may be “passthrough” in that it is directly defined by the scenario description 201a or the choice of parameterization 201b, and is thus unaffected by the outcome of the simulation. For example, the contextual data 214 may include a static road layout that comes from the scenario description 201a or the parameterization 201b directly. However, typically the contextual data 214 would include at least some elements derived within the simulator 202. This could, for example, include simulated environmental data, such as weather data, where the simulator 202 is free to change weather conditions as the simulation progresses. In that case, the weather data may be time-dependent, and that time dependency will be reflected in the contextual data 214.

The test oracle 252 receives the traces 212 and the contextual data 214, and scores those outputs in respect of a set of performance evaluation rules 254. The performance evaluation rules 254 are shown to be provided as an input to the test oracle 252.

The rules 254 are categorical in nature (e.g. pass/fail-type rules). Certain performance evaluation rules are also associated with numerical performance metrics used to “score” trajectories (e.g. indicating a degree of success or failure or some other quantity that helps explain or is otherwise relevant to the categorical results). The evaluation of the rules 254 is time-based—a given rule may have a different outcome at different points in the scenario. The scoring is also time-based: for each performance evaluation metric, the test oracle 252 tracks how the value of that metric (the score) changes over time as the simulation progresses. The test oracle 252 provides an output 256 comprising a time sequence 256a of categorical (e.g. pass/fail) results for each rule, and a score-time plot 256b for each performance metric, as described in further detail later. The results and scores 256a, 256b are informative to the expert 122 and can be used to identify and mitigate performance issues within the tested stack 100. The test oracle 252 also provides an overall (aggregate) result for the scenario (e.g. overall pass/fail). The output 256 of the test oracle 252 is stored in a test database 258, in association with information about the scenario to which the output 256 pertains. For example, the output 256 may be stored in association with the scenario description 210a (or an identifier thereof), and the chosen parameterization 201b. As well as the time-dependent results and scores, an overall score may also be assigned to the scenario and stored as part of the output 256. For example, an aggregate score for each rule (e.g. overall pass/fail) and/or an aggregate result (e.g. pass/fail) across all of the rules 254.

FIG. 2A illustrates another choice of slicing and uses reference numerals 100 and 100S to denote a full stack and sub-stack respectively. It is the sub-stack 100S that would be subject to testing within the testing pipeline 200 of FIG. 2.

A number of “later” perception components 102B form part of the sub-stack 100S to be tested and are applied, during testing, to simulated perception inputs 203. The later perception components 102B could, for example, include filtering or other fusion components that fuse perception inputs from multiple earlier perception components.

In the full stack 100, the later perception components 102B would receive actual perception inputs 213 from earlier perception components 102A. For example, the earlier perception components 102A might comprise one or more 2D or 3D bounding box detectors, in which case the simulated perception inputs provided to the late perception components could include simulated 2D or 3D bounding box detections, derived in the simulation via ray tracing. The earlier perception components 102A would generally include component(s) that operate directly on sensor data. With the slicing of FIG. 2A, the simulated perception inputs 203 would correspond in form to the actual perception inputs 213 that would normally be provided by the earlier perception components 102A. However, the earlier perception components 102A are not applied as part of the testing, but are instead used to train one or more perception error models 208 that can be used to introduce realistic error, in a statistically rigorous manner, into the simulated perception inputs 203 that are fed to the later perception components 102B of the sub-stack 100 under testing.

Such perception error models may be referred to as Perception Statistical Performance Models (PSPMs) or, synonymously, “PRISMs”. Further details of the principles of PSPMs, and suitable techniques for building and training them, may be bound in International Patent Publication Nos. WO2021037763 WO2021037760, WO2021037765, WO2021037761, and WO2021037766, each of which is incorporated herein by reference in its entirety. The idea behind PSPMs is to efficiently introduce realistic errors into the simulated perception inputs provided to the sub-stack 100S (i.e. that reflect the kind of errors that would be expected were the earlier perception components 102A to be applied in the real-world). In a simulation context, “perfect” ground truth perception inputs 203G are provided by the simulator, but these are used to derive more realistic (ablated) perception inputs 203 with realistic error introduced by the perception error models(s) 208. The perception error model(s) 208 serve as a “surrogate model” (being a surrogate for the perception system 102, or part of the perception system 102A, but operating on lower-fidelity inputs). Non-trained perception error models may also be used.

As described in the aforementioned reference, a perception error (or performance) model can be dependent on one or more variables representing physical condition(s) (“confounders”), allowing different levels of error to be introduced that reflect different possible real-world conditions. Hence, the simulator 202 can simulate different physical conditions (e.g. different weather conditions) by simply changing the value of a weather confounder(s), which will, in turn, change how perception error is introduced.

The later perception components 102b within the sub-stack 100S process the simulated perception inputs 203 in exactly the same way as they would process the real-world perception inputs 213 within the full stack 100, and their outputs, in turn, drive prediction, planning and control.

Alternatively, such models can be used to model the entire perception system 102, including the late perception components 208, in which case a model(s) is used to generate realistic perception output that are passed as inputs to the prediction system 104 directly.

Depending on the implementation, there may or may not be deterministic relationship between a given scenario parameterization 201b and the outcome of the simulation for a given configuration of the stack 100 (i.e. the same parameterization may or may not always lead to the same outcome for the same stack 100). Non-determinism can arise in various ways. For example, when simulation is based on PRISMs, a PRISM might model a distribution over possible perception outputs at each given time step of the scenario, from which a realistic perception output is sampled probabilistically. This leads to non-deterministic behaviour within the simulator 202, whereby different outcomes may be obtained for the same stack 100 and scenario parameterization because different perception outputs are sampled. Alternatively, or additionally, the simulator 202 may be inherently non-deterministic, e.g. weather, lighting or other environmental conditions may be randomized/probabilistic within the simulator 202 to a degree. As will be appreciated, this is a design choice: in other implementations, varying environmental conditions could instead be fully specified in the parameterization 201b of the scenario. With non-deterministic simulation, multiple scenario instances could be run for each parameterization. An aggregate pass/fail result could be assigned to a particular choice of parameterization 201b, e.g. as a count or percentage of pass or failure outcomes.

A test orchestration component 260 is responsible for selecting scenarios for the purpose of simulation. For example, the test orchestration component 260 may select scenario descriptions 201a and suitable parameterizations 201b automatically, which may be based on the test oracle outputs 256 from previous scenarios and/or other criteria. IN the following examples, the test orchestration component 260 implements a directed testing method, so as s

Example Performance Evaluation Rule:

One example rule considered herein is a “safe distance” rule that applies in a lane following context, and which is evaluated between an ego agent and another agent. The safe distance rule requires the ego agent to maintain a safe distance from the other threshold at all times. Lateral and longitudinal distance are both considered and, to pass the safe distance rule, it is sufficient for only one of those distances to satisfy some safety threshold (consider a lane driving scenario, with the ego agent and the other agent in adjacent lanes; when driving alongside each other, their longitudinal separation along the road may be zero or close to zero, which is safe provided a sufficient lateral separation between the agents is maintained; similarly, with the ego agent driving behind the other agent in the same lane, their latitudinal separation perpendicular to the direction of the road may be zero or close to zero assuming both agents are approximately following the midline of the lane, which is safe provided a sufficient longitudinal headway is maintained). A numerical score is computed for the safe distance rule at a given point in time based on whichever distance (latitudinal or longitudinal) is currently determinative of safety.

Robustness scores such as the score described below with reference to FIGS. 3A and 3B, take quantities such as absolute or relative positions, velocities, or other quantities of agents' relative motion. The robustness score is typically defined based on a threshold to one or more of these quantities based on the given rule (for example, the threshold might define a minimum lateral distance to a nearest agent that is deemed acceptable in terms of safety). The robustness score is then defined by whether the given quantity is above or below that threshold, and the degree to which the quantity exceeds or falls below the threshold. The robustness score therefore provides a measure of whether the agent is performing how it should or how it is expected to perform relative to other agents and to its environment (including, e.g. any speed limits, etc. defined within the drivable area for the given scenario).

The safe distance rule is chosen to illustrate certain principles underpinning the described methodology because it is simple and intuitive. It will, however, be appreciated that the described techniques can be applied to any rule that is designed to quantify some aspect (or aspects) of driving performance, such as safety, comfort and/or progress towards some defined goal, by way of a numerical “robustness score”. A time-varying robustness score over the duration of a scenario run is denoted (t) and an overall robustness score for a run is denoted y. For example, a robustness scoring framework may be constructed for driving rules that are based on signal-temporal logic.

FIG. 3A schematically illustrates the geometric principles of the safe distance rule, evaluated between an ego agent E and another agent C (the challenger).

FIGS. 3A and 3B are described in conjunction with each other.

Lateral distance is measured along a road reference line (which could be a straight line or a curve), and longitudinal separation is measured in the direction perpendicular to the road reference line. Lateral and longitudinal separation (distance between the ego agent E and the challenger C) are denoted by d_lat and d_lon respectively. Latitudinal and longitudinal distance thresholds (safety distances) are denoted by d_slat and d_slon.

The safety distances d_slat, d_slon are typically not fixed, but typically vary as functions of the agents' relative speed (and/or other factors, such as weather, road curvature, road surface, lighting etc.). Expressing the separations and safety distances as functions of time, t, latitudinal and longitudinal “headroom” distances are defined as:

$D_{\mathrm{lat}}\left(t\right)=d_{\mathrm{lat}}\left(t\right)-d_{\mathrm{slat}}\left(t\right)D_{\mathrm{lon}}\left(t\right)=d_{\mathrm{lon}}\left(t\right)-d_{\mathrm{slon}}\left(t\right).$

FIG. 3A(1) shows an ego agent E at a safe distance from a challenger C, by virtue of the fact that the agents' lateral separation d_lat is greater than the current lateral safety distance d_slat for that pair of agents (positive D_lat).

FIG. 3A(2) shows an ego agent E at a safe distance from a challenger C, by virtue of the fact that the agents' longitudinal separation d_lon is greater than the current longitudinal safety distance d_slon for that pair of agents (negative D_lon).

FIG. 3A(E) shows an ego agent E at an unsafe distance from a challenger C. The safe distance rule is failed because both D_lat and D_lon are negative.

FIG. 3B shows the safe distance rule implemented as a computational graph applied to a set of scenario ground truth 310 (or other scenario data).

The latitudinal separation, latitudinal safety distance, longitudinal separation and longitudinal safety distance are extracted from the scenario ground truth 310 by, respectively, first, second, third and fourths extractor nodes 302, 304, 312, 314 of the computational graph 300 as time-varying signals. The latitudinal and longitudinal headroom distances are computed by first and second computational (assessor) nodes 306, 316, and converted to robustness scores as follows. The following examples consider normalized robustness scores over some fixed range, such as [−1,1], with 0 as the pass threshold.

The headroom distances quantify the extent to which the relevant safety distance is or is not breached: a positive latitudinal/longitudinal headroom distance implies that the latitudinal/longitudinal separation between the ego E and the challenger C is greater than the current latitudinal/longitudinal safety distance, and a negative headroom distance implies the opposite. Following the principles set out above, robustness scores for lateral and longitudinal distance may, for example, be defined as follows:

$\mathrm{If}{D}_{\mathrm{lat}}\left(t\right)>0\mathrm{then}{s}_{\mathrm{lat}}\left(t\right)=\min \left[1,D_{\mathrm{lat}}\left(t\right)/A_{\mathrm{lat}}\right]\mathrm{If}{D}_{\mathrm{lat}}\left(t\right)\le 0\mathrm{the}{s}_{\mathrm{lat}}\left(t\right)=\max \left[-1,D_{\mathrm{lat}}\left(t\right)/B_{\mathrm{lat}}\right]\mathrm{If}{D}_{\mathrm{lon}}\left(t\right)>0\mathrm{then}{s}_{\mathrm{lon}}\left(t\right)=\min \left[1,D_{\mathrm{lon}}\left(t\right)/A_{\mathrm{lon}}\right]\mathrm{If}{D}_{\mathrm{lon}}\left(t\right)\le 0\mathrm{then}{s}_{\mathrm{lon}}\left(t\right)=\max \left[-1,D_{\mathrm{lon}}\left(t\right)/B_{\mathrm{lon}}\right]$

Here, A and B denote some predefined normalization distances (which may be the same or different for the lateral and longitudinal scores). For example, it can be seen that the longitudinal robustness score s_lon(t) varies between 1 and −1 as D_lon(t) varies between A_lon, and −B_lon. For D_lon(t)>A, the longitudinal robustness score is fixed at 1, and for s_lon(t)<B_lon the robustness score is fixed at −1. The longitudinal robustness score s_lon(t) varies continuously over all possible values of longitudinal headroom. The same considerations apply to the lateral robustness score. As will be appreciated, this is merely one example, and a robustness score s(t) can be defined in various ways based on headroom distance.

Score normalization is convenient, because it makes the rules more interpretable, and facilitates comparison of the scores between different rules. However, it is not essential for scores to be normalized in this way. A score could be defined over any range with any failure threshold (not necessarily at zero).

The robustness score s(t) for the safe distance rule as a whole is computed by a third assessor node 308 as:

$s\left(t\right)=\min \left[s_{\mathrm{lat}}\left(t\right),s_{\mathrm{lon}}\left(t\right)\right].$

The rule is passed when s(t)>0 and failed when s(t)≤0. The rule is ‘just’ failed when s=0 (implying that one of the longitudinal and lateral separations is equal to its safety distance), representing the boundary between PASS and FAIL outcomes (performance categories). Alternatively, s=0 could be defined at the point at which the ego E just passes; this is an immaterial design choice and, for that reason, the terms “pass threshold” and “failure threshold” are used interchangeably herein to refer to the subset of the parameter space where the robustness score y=0.

A pass/fail result (or, or more generally, a performance category) may be assigned to each time step of a scenario run based on the robustness score s(t) at that time, which is useful to an expert interpreting the results. However, the example directed search methods below are based on an overall robustness score assigned to a scenario run by a fourth assessor node 309 as:

$y=\underset{t}{\min }s\left(t\right),$

with an overall pass when y>0 and an overall fail when y≤0.

One possible directed search methodology utilizing the overall score y will now be described. The directed search uses a Gaussian process model to build a score predictor, which in turn is used to guide the selection of points (parameter sets) in the parameter space of a scenario.

To facilitate the description, an x superscript is introduced into the above notation to denote a quantity obtained with a given parameter set, e.g. y^(x) denotes the overall robustness score obtained on a run characterized by parameter set x (a given point in the parameter space).

FIG. 4A shows a score predictor 402 which is trained to learn a scoring function g(x):=y^(x).

The Gaussian process model described herein assumes that the scoring function g(x) varies continuously as a function of the scenario parameters x. That is, it is assumed that a small change in x causes only a small change in the overall score Y that is obtained. It will be appreciated that the techniques can be applied with other forms of machine learning model, and the description applies equally to other such models.

FIG. 4A shows the scoring function g(x) to be composed of the simulator 202, the stack 100 (or some part of parts thereof) coupled to the simulator 202, the test oracle 252 (running on the output of the simulator 252, and a max function that computes the overall score for a run on a given parameter set x as

$y^{\left(x\right)}=\underset{t}{\min }{s}^{\left(x\right)}\left(t\right)$

The score predictor 402 is a machine learning component based on a trainable regression model. A training set 404 is comprised of training example (x, y) pairs, where x is a parameter set on which a simulated run has been performed and y is the resulting score (for conciseness, the x superscript is omitted in the following description when the meaning is clear).

The score predictor 402 consumes a parameter set x as input, and outputs a predicted score 406. In fact, the output is a probability distribution over possible scores in the following examples, which provides not only a predicted score 406 but also a confidence level associated with the prediction. Ultimately, the aim is to train the score predictor 402 so that it can accurately predict the overall performance of a given stack 100 (or some part of parts thereof) in a given scenario with a given parameter set x, without having to actually run and score a simulated instance of the scenario with those parameters x. To generate a training example for a given scenario, an instance of that scenario is run on a given parameter set x in the simulator 202, the test oracle 252 is applied to the simulator output, and the overall score Y is computed from the oracle output. This requires significant computational resources, both to tune the simulation, and then score the simulator output against the rule or rules in question. It is therefore highly desirable to build the training set 404 in a targeted manner, with as few examples as needed to give sufficient confidence given some analysis objective. In the following examples, that objective is to accurately map out the ‘boundaries’ in the scenario's parameter space between PASS and FAIL regions. Here, the methodology is less concerned with accurately predicting the score y per se; rather, the aim is to accurately predict the sign of the score with confidence (or, more generally, to predict whether or not the score satisfies the pass threshold).

The directed search methodology can be applied with multiple rules simultaneously based on a set of scores

$\left\{y_i❘i=1,\dots ,N\right\},$

where the i subscript denotes the ith rule and N is the number of rules on which the directed search is applied. In this case, a score predictor is constructed and trained for each rule i, as described in further detail below.

FIG. 4 shows a high-level flow chart for a directed search method applied to a scenario.

At step 412, an initial set of points is selected in the parameter space of the scenario. For example, these could be selected randomly (e.g. based on Monte Carlo sampling, or a ‘grid’ of uniformly spaced points in the parameter space may be chosen initially. The scenario is instantiated at each of these initial points, and the simulator output is analyzed in the test oracle 252 to obtain an overall score for each initial point (step 414).

The training set 404 is constructed initially based on the initially selected points and their computed scores.

At step 416, the score predictor 402 is trained based on the training set 406 to compute a predicted score for any given input x in the parameter space. As noted above, the output of the score predictor is actually a probability distribution 407 over possible scores Y given x, denoted by denoted by p(y|x) (score distribution).

The method could terminate at this point (decision block 417), otherwise at step 418, the score distribution 407 is used to determine an “acquisition function”. Details of the acquisition function are described below. For now, suffice it to say that the acquisition function encodes the analysis objective (e.g. the objective to map out PASS/FAIL boundaries with confidence) in precise, mathematical terms. The acquisition function is then used to select (step 420) a next point x in the parameter space in fulfilment of that objective.

At step 422, the scenario is instantiated at the selected next point x, the simulator output is scored, and that point x is added to the training set 402 with its associated score y (step 424). Thereafter, the method is repeated from step 416 onwards. Step 416 is an update/re-training step in subsequent iterations, at which the score predictor 402 is updated or retrained based on the updated training set 402.

The current example considers the selection of a single point based on the acquisition function. A single additional data point is obtained (the next parameter set and its actual score) and the scoring function 402 is updated based on this single additional data point. In other implementations, multiple points may be selected at each update step instead. Multiple points may, for example, be selected in a given iteration based on an acquisition function using a batched greedy optimisation as described at https://emukit.readthedocs.io/en/latest/api/emukit.core.loop.html#emukit.core.loop.html#emukit.core.loop.candidate_point_calculators.GreedyBatchPointCalculator, and also in sec 19.4.3 of https://github.co/probml/pml-book/releases/latest/download/book1.pdf, each incorporated herein by reference in its entirety.

Returning to decision block 417, there are various termination criteria that could be applied. For example, the method might terminate (426) following step 416 if it is determined from the score distribution 407 that the analysis objective has been satisfied (e.g. the PASS/FAIL boundary has been mapped with sufficient confidence, e.g. as determined when a minimum misclassification probability has been achieved throughout the parameter space, or some subset of the parameter space that is of interest). Alternatively or additionally, the method may terminate after some defined number of iterations.

The predicted PASS/FAIL boundary refers to the subset of the parameter space where g(x) is zero or close to zero.

FIG. 4C shows further details of the output of the score predictor 402. In this example, the score distribution 407 has a Gaussian form, defined by a mean g_μ(x) and a standard deviation g_σ(x) that vary as functions of x that are outputted by the score predictor 402. Here, the computed mean g_μ(x) is interpreted as the predicted score 406, and the standard deviation g_σ(x) denotes a confidence in the predicted score 406.

For conciseness, the μ is dropped in the following description, and the mean score is denoted by g(x).

The acquisition function, denoted by reference numeral 430, is defined based on “misclassification probability”. The misclassification probability for a predicted score is the probability that the sign of the score is wrong (or, more generally, the probability that the predicted score is on the wrong side of the pass threshold compared with the true score that would be obtained by instantiating the scenario at x). This is not the same as the confidence in the predicted score 406 itself: the misclassification probability does depend on the standard deviation g_σ(x), but this is not the only factor; it also depends on how close x is to a PASS/FAIL boundary. A relatively high standard deviation (low confidence) at x may still imply a low misclassification probability if the predicted score g(x) is close to 1 or −1; on the other hand, a relatively low standard deviation may still imply a high misclassification probability if the predicted score g(x) is close to zero.

Taking the above factors into account, the acquisition function 430 is defined as:

$f\left(x\right)=❘g\left(x\right)❘/g_{\sigma }\left(x\right).$

That is, the acquisition function 430 is defined as the modulus of the predicted score divided by its computed standard deviation.

The next point to is selected at step 420 above as:

$x=\arg {\min }_{x}f\left(x\right)$

That is, the next point x is selected as that which minimizes the acquisition function ƒ(x).

As can be seen, as the predicted score in the numerator can decrease the value of the acquisition function as the predicted score moves closer to zero (this is true for both positive and negative scores), but an increase in the predicted standard deviation can also decrease the value of ƒ(x).

In a Gaussian model, the misclassification probability is given by

$p\left(x\right)=\Phi \left(-f\left(x\right)\right).$

Here, Φ denotes the standard normal distribution (Gaussian with mean and variance of one). The misclassification probability does not necessarily need to be computed; the method can be applied with the acquisition function ƒ(x) directly. As can be seen, the misclassification probability increases as ƒ(x) decreases, such that the argmin of ƒ(x) (or, equivalently, the argmax of −ƒ(x)) corresponds to the point of greatest misclassification probability.

Points which have already been explored will have g_σ≈0 hence ƒ(x) should be small and these points will not be chosen. In practice there can be numerical instabilities when g(x) is also close to zero. This can be mitigated in various ways, for example, by sampling from the acquisition function rather than the argmin of the acquisition. Another technique excludes points based on a desired accuracy level. For example, a credible interval may be determined for the failure probability (an interval in which the failure probability falls with some given probability, such as 0.95), and points x may be excluded if the credible interval falls within some predetermined range. When calculating the misclassification probability, a small constant may be added to the failure threshold when the mean associated with the credible interval is close to the threshold. This will mitigate the effect of g(x) and g_σ(x) both being close to zero simultaneously.

FIG. 4D shows an extension of the method to a rule set comprising multiple rules. As noted, in this case, a score predictor 402-i is constructed for each rule i under consideration. In the multiple rules case, the steps of FIG. 4B are modified as shown in FIG. 4D. An component acquisition function ƒ_i(x) is defined as above for each rule i.

$f_i\left(x\right)=❘g_i\left(x\right)❘/g_{\sigma ,i}\left(x\right),$

where |g_i(x) and g_σ,i(x) denote the predicted mean and standard deviation variance for rule i.

Each update is applied to a single rule only. With multiple roles, an overall “PASS” is obtained if and only if all rules are passed. On the other hand, an overall “FAIL” result is obtained if any of the rules is failed. The rule selection mechanism takes this into account, in order to accommodate multiple rules in the directed search process in the most efficient way possible.

At step 442, the method selects which of the rules to update as follows (index j denotes the rule selected at step 422). An overall acquisition function ƒ(x) is determined (step 422) in terms of the component acquisitions functions:

$\mathrm{If}{g}_i\left(x\right)>0\forall i\mathrm{then}f\left(x\right)=f_j\left(x\right)\mathrm{where}j=\arg {\min }_i{f}_i\left(x\right),\mathrm{If}{g}_i\left(x\right)\le 0\exists i\mathrm{then}f\left(x\right)=f_j\left(x\right)\mathrm{where}j=\arg {\max }_{i\in \hat{p}}{f}_i\left(x\right),\mathrm{where}\hat{p}=\left\{i:g_i\left(x\right)\le 0\right\}.$

The rationale is as follows. If all rules are currently predicted to be passed for a given parameter set x, then a misclassification occurs if any one of those rules is, in fact, failed at that point x in the parameter space. Therefore, in the first case above (all rules predicted to be passed at x), the rule with the highest misclassification probability is selected. This is the rule that is most likely to change the overall outcome when a simulation is actually performed, and therefore the most “promising” in terms of discovering a parameter combination x on which an overall FAIL result is obtained (finding a failure point is the first step towards mapping out the PASS/FAIL boundary, assuming the parameter space contains at least one failure region).

On the other hand, if at least one rule is predicted to be failed at a given x with relatively high confidence, there is little point in incurring the significant expense of evaluating other rules at that point (requiring expensive simulations and analysis): the overall “FAIL” outcome is unlikely to change (a confident predicted failure on rule j at point x implies a confident predicted overall fail). Somewhat counterintuitively, it is therefore the rule that is predicted to be failed with the lowest misclassification probability that is taken to be indicative of the overall probability of misclassification (because an overall misclassification result would require the most confident failure prediction to be wrong). The set {circumflex over (p)} is the subset of rule(s) for which a failure outcome is predicted at x, and the overall acquisition function is defined at that point by the rule for which a failure outcome is predicted with the lowest misclassification probability.

At step 444, the next point in the parameter space is selected in the same manner as above:

$x=\arg {\min }_{x}f\left(x\right)=\arg {\min }_x{f}_j\left(x\right),$

with j defined as above. Thus, in addition to selecting the next point x, one of the rules, j is also selected (as the rule that determines the acquisition function ƒ(x) at the selected point).

At step 446, the scenario is instantiated at the next point x. However, the simulator output need only be scored on rule j, as it is only the score predictor 402-j for rule j that is updated. This requires a simulation, but does not require a multi-rule scoring analysis (which is expensive in and of itself).

In principle, a situation can arise where the next point x has been previously explored, but only evaluated for some different rule k≠j. In this case, the simulation at x does not need to be repeated, and the existing simulator output can be accessed and scored at this point for rule j. In practice, this is almost impossible because choosing the same float when sampling from a continuous distribution is extremely unlikely.

Alternatively, every rule may be scored on every run, and the Gaussian process models for all rules may be updated after the scoring of each run.

FIG. 4E shows the negative of an acquisition function −ƒ(x) visually plotted in a single x dimension (recall that the misclassification probability is a function of −ƒ(x); a maximum in −ƒ(x) implies maximum misclassification probability, and corresponds to a minimum in ƒ(x)). The point x of maximum misclassification probability (overall misclassification probability in the case of multiple rules) is selected for the next simulation, leading to reduced uncertainty at that point when the scoring model is updated based on the robustness score that is obtained.

As briefly discussed, when a Gaussian process model is used, the score prediction assumes that the scoring function g(x) is continuous over the parameter space. If the rules are constructed in a way that materially breaches this assumption, the resulting discontinuities in the parameter space can prevent the Gaussian process search from converging. These considerations apply to other forms of model, and are not unique to Gaussian processes.

Without careful design, discontinuities in g(x) are more likely to arise in more complex scenarios where certain rules might only be naturally applicable for some part (or parts) of a run.

For example, consider an ego vehicle waiting on a minor road at a T-junction to join a traffic stream on a major road. Before the ego vehicle joins the traffic stream, the “safe distance” rule is not naturally applicable between the ego vehicle and other vehicles in the traffic stream, as that is not a lane following context; the rule only becomes pertinent once the ego vehicle enters the road and joins the traffic stream.

A simple way to address this would be to define a set of circumstances in which a given rule applies, and to set the rule robustness to a default value of 1 whenever the rule is inapplicable. For example, the safe distance rule may only be “activated” when the go agent crosses the junction line between the major and minor roads. Prior to that, the rule is inactive and set to a default value of 1. This is superficially appealing; generally the system designer/tester is only interested in failure cases. A “maximum” pass result of 1 is, in that context, the least interesting result, and it would therefore appear natural to extend a score of one to circumstances when the rule is inapplicable. However, in the context of the directed exploration method described above, this can cause significant convergence issues, as will now be illustrated with reference to FIGS. 5A to 6.

FIG. 5A shows first and second instances of a scenario 502, 504 with very similar parameters: x₁ and x′₁=x₁+δx₁ respectively. The “safe distance” rule discussed above is considered in this context.

The scenario commences with an ego vehicle, E, waiting at a junction on a minor road to join a traffic stream on a major road. In the first instance 520, the parameters x₁ are such that the ego vehicle E is never able to find an opportunity to join the major road, and simply waits at the junction for the entirety of the run (this does not necessarily mean that it would have been impossible for any ego vehicle to join the major road safely; it could also be that the ego vehicle is exercising ‘excessive’ caution).

By contrast, in the second instance 504, the ego vehicle E determines that it is able to safely join the traffic stream. This has arisen because the small difference δx₁ between the two parameter sets causes, for example, a slight reduction in the speed of the vehicles on the major road and/or a slight widening of a gap in the stream of traffic.

In the first instance 502, the robustness score for the safe distance rule s^(x1)(t) is artificially set to zero whilst the ego vehicle is waiting at the junction, and thus remains at one throughout the scenario. In the second instance 504, the robustness score s^(x′1)(t) suddenly drops as the ego vehicle E joins the major road, as from that point on the score is calculated as set out above. In this example, the robustness score s^(x′1)(t) reaches its minimum value of 0.2 at t=T₁.

Thus, the first parameter set is assigned an overall score of g(x₁)=1 whilst the second parameter set is assigned an overall score of g(x′₁)=g(x₁+δx₁)=0.2. Both outcomes are a “PASS” overall on the safe distance rule (as the score never falls below zero). However, a discontinuity has arisen in the scoring function 9) in that a small change in the variable has caused a large and sudden change in the score. The pass in the first instance 502 is ‘artificial’ in the sense that the safe distance rule was never actually applicable.

FIG. 5B illustrates a similar situation, where for a given parameter set x₂, the ego vehicle, once again, never joins the major road, resulting in an overall score of one (third instance 506). In this case, a small change in x₂→x′₂=x₂+δx₂ causes the ego vehicle E to pull out unsafely, resulting in an overall robustness score of −0.7 (the minimum value at time $T_2$), and an overall “FAIL” (fourth instance 508). Here, a discontinuity in g(x) has arisen across the PASS/FAIL boundary in the parameter space.

FIG. 5C, by contrast, illustrates fifth and sixth instances 510, 512 defined by parameters sets x₃ and x′₃=x₃+δx₃ where δx₃ is again small. In the fifth instance 510, the ego vehicle E joins the stream of traffic in a way that ‘just’ passes the safe distance rule, in the sense that the robustness score s^(x3)(t) falls very close to zero, to a minimum value of 0.04 at time T₃. In the sixth instance, the small change δx₃ does not materially alter the ego vehicle's behavior; the ego vehicle E still joins the traffic flow at a similar time, but now does so in a way that violates the safe distance rule (e.g. because the speed of the traffic flow has increased slightly and/or the gap in the traffic has reduced slightly). The robustness score s^(x′3)(t) now falls slightly below zero to a minimum value of −0.02 at time T′₃, meaning the ego vehicle E has ‘just’ failed on the safe distance rule. In contrast to FIGS. 5A and 5B, the change in the scoring function g(x) is continuous across the ‘true’ PASS/FAIL boundary, in that the small change in the parameters δx₃ has caused only a small change in g(x) from 0.04 to −0.02.

FIG. 6 summarizes the scoring outcomes of FIGS. 5A to 5C. A two-dimensional parameter space is shown, but the parameter space can have any number of dimensions in general. A first line segment 602 represents a discontinuity in the scoring function g(x) for the safe distance rule; in a first region 606 of the parameter space to the left of this line 602, the conditions of the scenario are such that ego vehicle E never joins the traffic stream; in a second region 608 to the right, the ego vehicle does join the traffic stream at some point. Points x₁ and x′₁=x₁+δx₁+ (FIG. 5A) are shown either side of this line 602, as are points x₂ and x′₂=x₂+δx₂ (FIG. 5B).

By contrast, a second line segment 604 denotes the true PASS/FAIL boundary. In a third ‘fail’ region 608A to the left of this line 604, the conditions of the scenario are such that the ego vehicle E joins the major road but fails the safe distance rule; in a fourth ‘pass’ region 608B to the right, the ego vehicle E joins the major road and passes. The third and fourth regions 608A, 608B are subsets of the second region 608. The scoring function g(x) changes continuously across the true PASS/FAIL boundary 604, with x₃ (FIG. 5C) shown just to the right of the boundary 604 and x′₃=x₃+δx₃ shown just to the left.

The Gaussian process methodology described above is based on an assumption that the scoring function g(x) is continuous. The discontinuity 602 that has been created by artificially setting a rule to one when it is not inherently applicable violates this assumption, and can prevent the directed search from converging. This applies to any rules, and to other forms of directed search that assume a continuous scoring function.

To address the aforementioned convergence problem, rules are instead assigned a special, non-numerical value (‘not a number’ or ‘NaN’) in conditions where they are not applicable. The ‘min’ function is now defined so that g(x)=nan if the robustness score remains NaN-valued throughout the entire duration of the scenario; if the robustness score is numerical at any point, the overall score is equal to the lowest numerical robustness value. For the safe distance rule example, the robustness score is set to be NaN valued at any time at which the ego vehicle E is waiting at the junction. Once the ego vehicle joins the major road, e.g. by crossing a defined junction line, the robustness score becomes numerical, and is computed as set out above.

Another benefit of NaN results is that they are more informative to an expert, expressly distinguishing, in the value of the robustness score, between circumstances in which the rule applies and is passed, and circumstances where the rule does not apply.

FIG. 7 shows the first and second instances 502, 504 of FIG. 5A, but now scored with robustness scores that can be NaN-valued. For the first instance 502, the robustness score s^(x1)(t) remains NaN-valued at all times, hence g(x₁)=nan. For the second instance, g(x′₁)=0.2 as before.

FIG. 8 shows the parameter space of the scenario as per FIG. 6; the difference is that the scoring function g(x) is now NaN-valued in the first region 606. The first line segment 602 now denotes the boundary between the first region 606 in which the scoring function g(x) is NaN-valued and the second region 608 in which it has a numerical value (e.g. float-valued), referred to as the ‘NaN/float’ boundary. The NaN/float 602 is divided into two sections: a first section 602A between the NaN region 606 and the FAIL region 608A and a second section 602B between the NaN region 606 and the PASS region 602B.

The directed search method above cannot be directly applied with a scoring function g(x) that can be NaN-valued. However, the scoring function g(x) now distinguishes explicitly between a ‘PASS’ outcome and a ‘NOT APPLICABLE’ (NaN) outcome, it becomes possible to model the discontinuity 602 explicitly using a hierarchical model.

With a Bayesian hierarchical approach, two models are learned for each rule, as shown in FIG. 9A.

FIG. 9A shows hierarchical score predictor formed of a score regression model 902 and a score classification model 904. The score classification model 904 is trained to predict, for a given x, a first probability distribution 903 of the form:

$p\left(y_i=\mathrm{nan}❘x\right),$

• • denoting the probability that the scoring function is NaN-valued given input x. The probability that the scoring function is numerical is given by:

$p\left(y_i\ne \mathrm{nan}❘x\right)=1-\left(y_i=\mathrm{nan}❘x\right).$

The score regression model 902 is configured in the same way as the score predictor 404, but is now computing a second probability distribution 905 of the form:

$p\left(y_i❘x,y_i\ne \mathrm{nan}\right),$

• • denoting the conditional probability of the score value y_i given that y_i is not NaN-valued.

Any probabilistic classification and regression models can be used in this context. The training of the models 902, 904 is described in detail below.

FIG. 9B shows a modified directed search method using the hierarchical models of FIG. 9A.

As in the method of FIG. 4A, an initial set of points in the parameter space is selected (step 912) and simulations are performed for each of the initial parameter sets (step 914). As before, the result is a training set 904 of (x, y_i) example pairs, with the difference that $y_i$ can now be NaN valued. The full training set 904 is used to train the classification model 904 as a probabilistic binary classifier on the {NAN, NUMERICAL}. For the purpose of training the classification model 904, the actual numerical score values are not germain; all parameter sets in the training set 904 are signed to the NUMERICAL (not-NaN) class.

The regression model 902 is trained only on the subset 907 of training examples that are numerically valued.

Thereafter, the method proceeds as before, but with a modified acquisition function that is described below. At step 916, the modified acquisition function is determined based on the first and second distributions 903, 905. At step 918, the next point (or points) to explore is selected based on the modified acquisition function and, at step 920, the scenario is instantiated and scored at that (or those) points. The latest result(s) is added to the training set 906, and the method repeats. The classification model 904 is always updated at this point (step 914B). The regression model 902 is only updated (step 914A) if a non-NaN-valued example has been added to the training set 906 (as it is only trained on the subset of numerically-valued examples).

If, following the update of the models at steps 912a, 915b, a termination condition is satisfied (decision block 921), the method terminates (step 922).

When the method eventually terminates at step 922, the final regression and classification models 902, 904 serve to map out not only the PASS/FAIL boundary (FIG. 8, 604), but also (to a certain extent) the NaN/not-NaN boundary (FIG. 8, 602). This provides an accurate approximation of the behavior of the stack 100 under testing over the entire parameter space, in terms of {PASS, FAIL, NaN} outcomes for each rule, with as few simulations as possible. This, in turn, allows regions of the parameter space to be identified where it can be predicted, with confidence, that the stack 100 will not behave as expected, indicating some issue within the stack 100 that can now be identified and mitigated. This could be a region in which the stack 100 is predicted to fail when it ought to be capable of passing. This could also be an unexpected NaN region indicating, for example, that the ego vehicle is predicted to behave too cautiously (e.g. never joining the oncoming traffic flow in the junction examples above).

The modified acquisition function is determined at step 916 as follows. The i subscript is omitted for conciseness; with multiple rules, a modified acquisition function is determined for each rule as set out below.

The scoring function g may now be expressed as a mapping:

$g:𝒳\to ℝ\bigcup \left\{\mathrm{nan}\right\}$

• • where =ⁿ denotes the parameter space of a scenario and denotes the set of real numbers. In other words, the overall score y=g(x) can be numerical or NaN-valued.

Here, n≥1 is the dimensionality of the parameter space, where each parameter set x∈ (or, strictly speaking, each parameter tuple) has one or more numerically-valued elements.

Let F denote a failure event and N denote a NaN event. Using logic notation, the approach described herein seeks to classify cases where F∩N vs. cases where F↔N. In other words, a first category cases with a failure (and therefore a non-NaN) outcome vs. a second category of cases where the outcome is either a non-failure event (pass) or a Nan event.

As set out above, the first model 902 is a Gaussian Process (GP) regression model that provides p(y|x, y≠nan). This is trained by performing GP regression for the dataset 907, which may be expressed in mathematical notation as {(x, y)|y≠nan}.

The model for F can, in turn, be expressed as:

$p\left(F\bigcap \stackrel{\_}{N}❘x\right)={\int }_{y<0}p\left(y❘x,y\ne \mathrm{nan}\right)\left(1-p\left(y=\mathrm{nan}❘x\right)\right)\mathrm{dy}.$

This assumes the pass/fail threshold is defined by g(x)=0. As will be appreciated, the definition can be extended to any threshold by integrating over the range of numerical y values that are classified as failure.

Because p(y|x, y≠nan) is parameterised as a Gaussian distribution with mean g_μ(x) and standard deviation g_σ(x), this can be written as

$p\left(F\bigcap \stackrel{\_}{N}❘x\right)=\Phi \left(-\frac{g_{\mu }\left(x\right)}{g_{\sigma }\left(x\right)}\right)\left(1-p\left(y=\mathrm{nan}❘x\right)\right),$

• • where φ is the standard normal CDF. If p(F∩N|x)>05, x is classified as ‘interesting’ (F∩N or ‘not-NaN and FAIL’), implying an overall prediction is that x would result in a numerical score below the threshold; otherwise x is classified as ‘uninteresting’ (F∪N), implying an overall prediction that x would lead to either a numerical pass score or a NaN score. Further explanation of the ‘interesting/uninteresting’ terminology is provided below.

Therefore, the probability of misclassification acquisition function can be calculated as follows:

• • if classified as F∩N then misclassification probability is p(F∪N)=1−p(F∩N|x)
  • if classified as F∪N the misclassification probability is p(F∩N|x).

This, in turn, allows the modified acquisition function to be defined in terms of the first and second distributions 903, 905 as:

$\min \left(p\left(F❘x,\stackrel{\_}{N}\right)p\left(\stackrel{\_}{N}❘x\right),1-p\left(F❘x,\stackrel{\_}{N}\right)p\left(\stackrel{\_}{N}❘x\right)\right),$

• • where

$p\left(F❘x,\stackrel{\_}{N}\right)=\Phi \left(\frac{-g_{\mu }\left(x\right)}{g_{\sigma }\left(x\right)}\right),$

• • with the mean g_μ(x) and standard deviation g_σ(x) of the first distribution 903 as provided by the regression model 902, and

$p\left(\stackrel{\_}{N}❘x\right)=1-p\left(y=\mathrm{nan}❘x\right)$

as provided by the classification model 904.

The definition of ‘misclassification’ in this context is somewhat nuanced. Misclassification is defined with respect to the ‘interesting’ category (not-NaN and FAIL) vs. all other possible outcomes (the ‘uninteresting’ category): the aim of the directed search is to determine the PASS/FAIL boundary 604 and the FAIL/NaN boundary 602B with sufficient confidence; the search is not concerned with the PASS/NaN boundary 602A. Accordingly, a misclassification occurs if x is classified as ‘interesting’ (not-NaN and FAIL) when the actual outcome is anything else (uninteresting). The acquisition function does not distinguish between different ‘uninteresting’ outcomes; hence, the definition of misclassification probability in the first bullet point above. A misclassification also occurs if the actual outcome for x is ‘interesting’ (not-NaN and FAIL), but it has been classified as ‘uninteresting’ (implying a predicted outcome of anything other than ‘not-NaN and FAIL’); hence the definition of misclassification probability in the second bullet point.

On the other hand, if the most probable outcome for x is ‘not-NaN and PASS’, but the actual outcome is NaN, this is not considered a misclassification (and vice versa). To give an example, suppose x has been classified as PASS with p(F|x, N)=0.1 and p(N|x)=0.51. This is a state of near-maximum entropy between the PASS and NaN classes, with a 0.49 probability that the parameter set x would actually lead to a NaN outcome in the simulator rather than the predicted PASS (so almost 50/50 either way). However, the probability of x leading to a FAIL outcome in the simulator is only 0.51*0.1=0.051 (just over 5%). The search is only concerned with the latter metric, and the high degree of uncertainty as between the PASS and NaN classes does not contribute to the modified acquisition function. This is a logical choice from the perspective of a system designer or engineer; if they can be highly confident that a given parameter choice x will lead to PASS or ‘not applicable’ outcome on a given rule, there is little to be gained in exploring that rule further at x (of course, that is not to say that scenario instances with a ‘not applicable’ outcome on a given rule are of no interest at all; for example, if an ego vehicle is being too hesitant or conservative waiting at a junction, leading to a NaN result on some rule that only applies when the ego exits the junction, the underlying ego behavior may well be of interest to the expert; but rather than trying to explore this in terms of PASS/NaN outcomes on the rule that is inapplicable at the junction, the correct approach would be to design a rule that targets the desired behavior directly in terms of PASS/FAIL outcomes and perform a directed search with respect to that rule, which could, for example, be some progress metric that applies whilst the ego vehicle waits at the junction and results in a FAIL outcome if the ego vehicle is determined to have waited for ‘too long’ before joining the major road in a given set of circumstances).

Note that the ‘interesting/uninteresting’ terminology refers to the predicted outcome in a testing context, and reflects the special status of the ‘not-NaN and FAIL’ outcome. A point x with p(F∩N|x)=0.149 belongs to the uninteresting category, because the overall prediction is that the result will be uninteresting (i.e. something other than ‘not-Nan and FAIL’); nevertheless, there is a 49% probability that the outcome will actually be interesting, and such a point could well be selected as the next point to explore if 49% is the highest overall misclassification probability.

Applying the above definition of the modified acquisition function, the next point x is determined as that most likely to have been misclassified with respect to the FAIL category vs. all other categories:

$\arg {\max }_x\min \left(p\left(F❘x,\stackrel{\_}{N}\right)p\left(\stackrel{\_}{N}❘x\right),1-p\left(F❘x,\stackrel{\_}{N}\right)p\left(\stackrel{\_}{N}❘x\right)\right).$

This considers the selection of a single next point in each iteration for simplicity. Alternatively, multiple points may be selected for the next iteration, e.g. using the batched greedy optimisation described above.

It is useful to note that, when p(y=nan|x)=0, the modified acquisition function reduces, as expected, to the simpler, non-hierarchical form described above:

$\min \left(\Phi \left(-\frac{g_{\mu }\left(x\right)}{g_{\sigma }\left(x\right)}\right),1-\Phi \left(-\frac{g_{\mu }\left(x\right)}{g_{\sigma }\left(x\right)}\right)\right)=\Phi \left(-\frac{❘g_{\mu }\left(x\right)❘}{g_{\sigma }\left(x\right)}\right)$

Multiple rules can be combined using the method of FIG. 4D, but replacing the simpler form of the acquisition function with the modified acquisition function for each rule.

References herein to components, functions, modules and the like, denote functional components of a computer system which may be implemented at the hardware level in various ways. A computer system comprises execution hardware which may be configured to execute the method/algorithmic steps disclosed herein and/or to implement a model trained using the present techniques. The term execution hardware encompasses any form/combination of hardware configured to execute the relevant method/algorithmic steps. The execution hardware may take the form of one or more processors, which may be programmable or non-programmable, or a combination of programmable and non-programmable hardware may be used. Examples of suitable programmable processors include general purpose processors based on an instruction set architecture, such as CPUs, GPUs/accelerator processors etc. Such general-purpose processors typically execute computer readable instructions held in memory coupled to or internal to the processor and carry out the relevant steps in accordance with those instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs) having a circuit configuration programmable through circuit description code. Examples of non-programmable processors include application specific integrated circuits (ASICs). Code, instructions etc. may be stored as appropriate on transitory or non-transitory media (examples of the latter including solid state, magnetic and optical storage device(s) and the like). The subsystems 102-108 of the runtime stack FIG. 1A may be implemented in programmable or dedicated processor(s), or a combination of both, on-board a vehicle or in an off-board computer system in the context of testing and the like. The various components of FIG. 2, such as the simulator 202 and the test oracle 252 may be similarly implemented in programmable and/or dedicated hardware.

Claims

1. A computer-implemented method of testing, in simulation, performance of a robotic system in control of an ego agent of a test scenario, the method comprising: determining a first set of points in a parameter space of the test scenario, each point being a set of one or more parameter values for the test scenario;based the first set of points, generating in a simulation environment multiple first instances of the test scenario with the robotic system in control of the ego agent;assigning to each point of the first set a plurality of performance indicators based on a plurality of performance evaluation rules, thereby generating a training set of points and their assigned pluralities of performance indicators, wherein each performance indicator denotes a pass outcome or a fail outcome;using the training set to train a performance predictor for probabilistically predicting at each point x in the parameter space of the test scenario a pass or fail outcome for each performance evaluation rule;using the trained performance predictor to determine a plurality of rule acquisition functions, each rule acquisition function ƒi(x) denoting at each point x in the parameter space a probability of an incorrect outcome prediction for a performance evaluation rule i of the plurality of performance evaluation rules;selecting one or more second points based on an overall acquisition function defined as ƒ(x)=ƒj(x), wherein if a pass outcome is predicted at a given point x for every rule i then j is the performance evaluation rule having highest probability of an incorrect outcome prediction at x, and wherein if a fail outcome is predicted at a given point x for at least one rule i, then j is the performance evaluation rule for which a fail outcome is predicted at x with lowest probability of an incorrect outcome prediction;based on the one or more second points in the parameter space, generating in a simulation environment one or more second instances of the test scenario with the robotic system in control of the ego agent; andproviding one or more outputs for evaluating performance of the robotic system in the one or more second instances based on the at least one predetermined performance evaluation rule.

2. The method of claim 1, wherein the one or more outputs are rendered in a graphical user interface.

3. The method of claim 1, wherein the one or more outputs comprise one or more performance scores assigned to each second point for the at least one performance evaluation rule.

4. The method of claim 3, comprising updating the performance predictor based on the one or more second points and the one or more performance scores assigned thereto.

5. The method of claim 4, comprising: selecting one or more third points in the parameter space based on an updated overall acquisition function defined by the updated performance predictor;based the one or more third points in the parameter space, generating in a simulation environment one or more third instances of the test scenario with the robotic system in control of the ego agent; andproviding one or more second outputs for evaluating performance of the robotic system in the one or more third instances based on the at least one predetermined performance evaluation rule.

6. The method of claim 5, wherein the one or more second outputs comprise one or more performance scores assigned to the one or more third points, and the method is repeated iteratively until the performance predictor satisfies a termination condition, or a predetermined number of iterations is reached.

7. The method of claim 1, wherein the performance predictor comprises a Gaussian score prediction model for each performance prediction rule.

8. The method of claim 7, wherein the score prediction model provides a mean performance score gi(x) and standard deviation gσ,i(x) for each rule, i, at a given point x in the parameter space, wherein the rule acquisition function for rule i is based on

9. The method of claim 1, wherein the robotic system comprises a trajectory planner for a mobile robot.

10. The method of claim 1, comprising using a score classification model and a score regression model to identify and mitigate an issue with the robotic system.

11. A computer system for testing, in simulation, performance of a robotic system in control of an ego agent of a test scenario, the computer system comprising: at least one memory storing computer-readable instructions; andat least one processor coupled to the at least one memory and configured to execute the computer-readable instructions, which upon execution cause the at least one processor to carry out operations of: determining a first set of points in a parameter space of the test scenario, each point being a set of one or more parameter values for the test scenario;based the first set of points, generating in a simulation environment multiple first instances of the test scenario with the robotic system in control of the ego agent;assigning to each point of the first set a plurality of performance indicators based on a plurality of performance evaluation rules, thereby generating a training set of points and their assigned pluralities of performance indicators, wherein each performance indicator denotes a pass outcome or a fail outcome;using the training set to train a performance predictor for probabilistically predicting at each point in the parameter space of the test scenario a pass or fail outcome for each performance evaluation rule;using the trained performance predictor to determine a plurality of rule acquisition functions, each rule acquisition function denoting at each point in the parameter space a probability of an incorrect outcome prediction for a performance evaluation rule of the plurality of performance evaluation rules;selecting one or more second points based on an overall acquisition function, wherein if a pass outcome is predicted at a given point for every rule then the overall acquisition function is at each point the rule acquisition function for the performance evaluation rule having highest probability of an incorrect outcome prediction at that point, and wherein if a fail outcome is predicted at a given point for at least one rule i, then the overall acquisition function is the rule acquisition function for the performance evaluation rule for which a fail outcome is predicted at that point with lowest probability of an incorrect outcome prediction;based on the one or more second points in the parameter space, generating in a simulation environment one or more second instances of the test scenario with the robotic system in control of the ego agent; andproviding one or more outputs for evaluating performance of the robotic system in the one or more second instances based on the at least one predetermined performance evaluation rule.

12. A non-transitory computer readable medium having encoded thereon computer program instructions, the computer program instructions configured so as, when executed on one or more hardware processors, to implement operations for testing, in simulation, performance of a robotic system in control of an ego agent of a test scenario, the operations comprising: determining a first set of points in a parameter space of the test scenario, each point being a set of one or more parameter values for the test scenario;based the first set of points, generating in a simulation environment multiple first instances of the test scenario with the robotic system in control of the ego agent;assigning to each point of the first set a plurality of performance indicators based on a plurality of performance evaluation rules, thereby generating a training set of points and their assigned pluralities of performance indicators, wherein each performance indicator denotes a pass outcome or a fail outcome;using the training set to train a performance predictor for probabilistically predicting at each point in the parameter space of the test scenario a pass or fail outcome for each performance evaluation rule;using the trained performance predictor to determine a plurality of rule acquisition functions, each rule acquisition function denoting at each point in the parameter space a probability of an incorrect outcome prediction for a performance evaluation rule of the plurality of performance evaluation rules;selecting one or more second points based on an overall acquisition function, wherein if a pass outcome is predicted at a given point for every rule then the overall acquisition function is at each point the rule acquisition function for the performance evaluation rule having highest probability of an incorrect outcome prediction at that point, and wherein if a fail outcome is predicted at a given point for at least one rule, then the overall acquisition function is the rule acquisition function for the performance evaluation rule for which a fail outcome is predicted at that point with lowest probability of an incorrect outcome prediction;based on the one or more second points in the parameter space, generating in a simulation environment one or more second instances of the test scenario with the robotic system in control of the ego agent; andproviding one or more outputs for evaluating performance of the robotic system in the one or more second instances based on the at least one predetermined performance evaluation rule.

13. The computer system of claim 11, wherein the one or more outputs are rendered in a graphical user interface.

14. The computer system of claim 11, wherein the one or more outputs comprise one or more performance scores assigned to each second point for the at least one performance evaluation rule.

15. The computer system of claim 14, comprising updating the performance predictor based on the one or more second points and the one or more performance scores assigned thereto.

16. The computer system of claim 15, comprising: selecting one or more third points in the parameter space based on an updated overall acquisition function defined by the updated performance predictor;based the one or more third points in the parameter space, generating in a simulation environment one or more third instances of the test scenario with the robotic system in control of the ego agent; andproviding one or more second outputs for evaluating performance of the robotic system in the one or more third instances based on the at least one predetermined performance evaluation rule.

17. The computer system of claim 16, wherein the one or more second outputs comprise one or more performance scores assigned to the one or more third points, and the operations are repeated iteratively until the performance predictor satisfies a termination condition or a predetermined number of iterations is reached.

18. The computer system of claim 11, wherein the performance predictor comprises a Gaussian score prediction model for each performance prediction rule.

19. The computer system of claim 18, wherein the Gaussian score prediction model provides a mean performance score and a standard deviation for each rule at a given point in the parameter space, wherein the rule acquisition function for the rule is based on the mean performance score divided by the standard deviation.

20. The computer system of claim 11, wherein the robotic system comprises a trajectory planner for a mobile robot.