AUTOMATED IDENTIFICATION OF TRAINING DATA CANDIDATES FOR PERCEPTION SYSTEMS

Abstract

Methods are described for automatically identifying perception weaknesses for training data to be used in improving the performance of perception systems. Deficiencies in simulated data are also identified. The methods, which can be incorporated into a system or into instructions placed on storage media, include comparing perception system results between baseline results and results with augmented inputs and identifying perception weaknesses responsive to that comparison. The perception system is retrained using the relabeled data and is improved thereby.

Description

FIELD OF THE INVENTION

This invention relates to novel approaches to identify which data will be useful for training a system that learns by example (e.g., one that uses a form of machine learning). More particularly, the invention relates to methods and systems for identifying high-value training data to be used in retraining a learning-based system such as one implemented using machine learning. Vision-based perception is given as an example embodiment, but other systems will be amenable to this approach as will be apparent to those of skill in the art.

BACKGROUND OF THE INVENTION

Autonomous systems, such as self-driving cars, use a perception system to transform sensor data (e.g., video camera data) into a model of the world around the system (e.g., a list of objects and positions). This is a more general example of systems that behave based on training data (e.g., implemented with a form of machine learning such as deep learning based on convolutional neural networks).

An important component of performance for such systems is whether the training data covers all important aspects of the behavior to be learned with sufficient breadth and with a sufficient number of examples of each object type to be learned. Each object type typically has multiple sub-types which must also be included in the training data for proper coverage. As a simple example, an object type might be a pedestrian, and sub-types might be whether a pedestrian is wearing a red coat, black coat, or blue coat. “Sub-types” in this context do not necessarily mean classification information that is required by the end application, but rather different presentations of objects that each require sufficient training data to be able to generalize to all relevant objects of that type. In practice, under-sampled training data can result in biases in the output of a machine learning process that might reduce the probability of correct classification of an object to an unacceptably low level because some characteristic is different enough from other instances of that object type to expose a potential defect in learning results.

As an example, consider a perception system that has been trained thoroughly on pedestrians with red coats, black coats, and blue coats. A pedestrian in a yellow rain coat might be detected with lower probability as a pedestrian, or potentially not recognized as a pedestrian at all, if the perception system has learned via its training data that the color yellow is associated with objects other than pedestrian (for example, traffic warning signs and traffic barriers). This situation can be said to result from a systemic bias in training data that under-represents pedestrians in yellow coats.

One approach to identifying such systemic biases is based on comparing object detection and/or classification results between a baseline sensor data stream and a sensor data stream that has been augmented (e.g., with Gaussian noise). Such an approach is described in International Patent Application PCT/US19/59619 Systems and Methods for Evaluating Perception System Quality filed 4 Nov. 2019 which is hereby incorporated by reference in its entirety. The approaches described in that patent application are generically called “weakness detections” for a perception system. For example, a comparison of a baseline video data stream with a video data stream having Gaussian noise injected would be an example of a “perception weakness detector.” (Note that such detectors can work on other sensor modalities, such as lidar and radar.) An instance of a sensor data sample detecting a weakness is known as a “detection.”

One way to employ a perception weakness detector is to present examples of weaknesses to a human analyst. The human analyst can look at numerous detections, discern a pattern, and determine that a corresponding weakness is present in the perception system. Once the weakness has been determined, perception designers can search for more examples of similar data samples and use them to retrain the system, potentially improving performance. If sufficiently numerous additional data samples do not improve the perception system through retraining alone, the perception designers might need to reengineer the system in some way to further improve performance on the identified weakness. The use of manual analysis to determine perception weaknesses can be effective, but can also be time consuming of highly skilled specialists. It is an objective of this invention to reduce the manual analysis effort required to improve perception system performance. In embodiments, improvement of the perception system performance can be demonstrated by a reduction in the number of perception weakness or defects that are identified by the system.

As mentioned, once a perception weakness has been identified, additional training data must be identified. This can involve significant manual effort, such as scanning through archived video for pedestrians wearing yellow coats in line with previously given examples. It is a further objective of this invention to reduce the time and effort required to identify such training data.

After a weakness in a perception system has been identified, training data must be labeled before it can be fed into the machine learning retraining process. Labeling can be extremely expensive due to a significant amount of human labor involvement. For example, it has been reported that a single hour of driving data can take up to eight hundred (800) hours of man-power to label. Another objective of this invention is to make labelling costs more effective by increasing the proportion of data being labelled that will actually help improve perception performance via use as retraining data. As used herein, “training data” refers to both training and validation data as used, for example, in a supervised learning machine learning based approach. The “training data” identified by the disclosed embodiments can be partitioned into all training data, all validation data, or a combination of both as deemed advantageous by perception system designers and the perception validation activity.

These and other features and advantages will be apparent from a reading of the following detailed description and a review of the appended drawings. It is to be understood that the foregoing summary, the following detailed description and the appended drawings are explanatory only and are not restrictive of various aspects as claimed.

SUMMARY OF THE INVENTION

In a first aspect, a computer-implemented method for improving the labeling performance of a machine-learned (ML) perception system on an input dataset is provided that comprises processing the input dataset through the perception system After the input data system is processed, the method further comprises identifying one or more perception weaknesses, isolating at least one scene from the input dataset that includes the one or more perception weaknesses, relabeling the one or more perception weaknesses in the at least one scene to obtain at least one relabeled scene, and retraining the perception system with the at least one relabeled scene. In this manner, the labeling performance of the ML perception system is improved.

In a second aspect, a computer-implemented method evaluating the quality of simulated data used to train a ML perception system is disclosed. In certain embodiments, the method comprises processing an input dataset of simulated data through a perception system to obtain one or more candidate defects, extracting a simulated object rendering related to the one or more candidate defects, comparing the simulated object rendering to other examples of the object, and flagging a simulated object having low confidence matching to the other examples.

In a third aspect, a computer-implemented method of determining a triggering condition for perception weaknesses identified during perception engine testing with simulated data is disclosed. In certain embodiments, the method comprises processing an input dataset comprising simulated data through a perception system to get a first set of results, processing an input dataset comprising augmented simulated data through the perception system to get a second set of results, comparing the first set of results with the second set of results using a defect detection engine to identify perception weaknesses caused by simulated objects, the simulated objects defined by a plurality of parameters, and altering at least one parameter to create an altered simulated object. In certain embodiments, the third aspect further involves repeating the processing and comparing steps with the altered simulated object to determine whether the altered simulated object affected the perception weaknesses.

In certain embodiments, these computer-enabled methods can be enabled and performed via a computer system having at least a memory or other data storage facility, one or more processors, and a non-transitory computer-readable medium to store the instructions so that when enabled by the one or more processors, they perform one or more of the aforementioned aspects.

BRIEF DESCRIPTION OF THE DRAWINGS

For a fuller understanding of the nature and advantages of the present invention, reference should be made to the following detailed description, taken in connection with the accompanying drawings, in which:

FIG. 1 illustrates the inputs for identification of defects in data for a system being tested.

FIG. 2 is a block diagram demonstrating the workflow of a method of improving the labeling performance of a machine-learned system.

FIG. 3 is a block diagram demonstrating the workflow of a method of evaluating the quality of simulated data used to train a machine-learned system.

DETAILED DESCRIPTION OF THE INVENTION

The detailed description provided below in connection with the appended drawings is intended as a description of examples and is not intended to represent the only forms in which the present examples can be constructed or utilized. The description sets forth functions of the examples and sequences of steps for constructing and operating the examples. However, the same or equivalent functions and sequences can be accomplished by different examples.

References to “one embodiment,” “an embodiment,” “an example embodiment,” “certain embodiments,” “one implementation,” “an implementation,” “one example,” “an example” and the like, indicate that the described embodiment, implementation or example can include a particular feature, structure or characteristic, but every embodiment, implementation or example can not necessarily include the particular feature, structure or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment, implementation or example. Further, when a particular feature, structure or characteristic is described in connection with an embodiment, implementation or example, it is to be appreciated that such feature, structure or characteristic can be implemented in connection with other embodiments, implementations or examples whether or not explicitly described.

FIG. 1 shows some of the data sets that are used by defect detection engine 10, which is used to identify the presence of a perception weakness (hereinafter also called a “defect”) within a given set of input data, the baseline dataset 20. The defect detection engine 10 corresponds to a suitable embodiment described in International Application PCT/US2019/59619, entitled “Systems and Methods for Evaluating Perception System Quality”, the content of which is incorporated herein in its entirety, to identify weak perception performance.

In certain embodiments, the defect detection engine 10 is parameterized by the following attributes: baseline dataset 20, augmentations 30, detectors 40, and, optionally, comparison dataset 50. The baseline dataset 20 is the set of data on which the defect detection engine 10 operates in order to identify defects. In some embodiments, the engine 10 may need to alter the baseline dataset 20 in some form in order to facilitate defect detection; augmentations 30 are specifications indicating how the data is to be altered in order to accomplish this end. In alternative embodiments, the engine may be provided with a comparison dataset 50 to be used in concert with the baseline dataset 20 in order to perform the defect detection process.

The engine 10 uses one or more computational components in order to perform the desired detection process(es). Detectors 40 are specifications indicating which components to use in order to accomplish this end. The engine 10 executes defect detection process(es) against the baseline dataset 20 and, in some embodiments, a comparison dataset 50, of a specific “System Under Test” (SUT) 60 in order to evaluate its performance against the test criteria for which given detectors 40 were created. In certain embodiments, the SUT 60 is a perception system, such as a deep convolutional neural network. Perception weaknesses, or “defects,” are deposited in a defect database 70 for further processing or label review in accordance with certain embodiments of the present invention.

An embodiment of the inventive workflow system 200 is set forth in FIG. 2. The embodiment assumes that work executed by the workflow system 200 is represented by a sequence of job specifications 205. As used herein, job specifications 205 are descriptions of the data inputs, configuration metadata, processing steps and output of each unit of work processed by the system 200. These job specifications 205 are processed by an orchestrator 210, which organizes and executes components of the workflow as necessary to satisfy each submitted job specification 205. Components of the workflow system 200 can assume that all parameters necessary to execute their stage of the overall workflow are provided to them by the orchestrator 210.

Labeled data is required to train machine-learned models. Labeled data can only be produced at sufficient quality and diversity by human beings involved in some manner doing labeling, and/or quality checks of any automated labeling support. Hiring, training and managing teams of human labelers is expensive and time consuming. In order to maximize the ROI associated with teams of human labelers, the quality, unit cost and value of labels produced by these teams must be optimized. Label quality is optimized by having well-trained labelers and implementing solid quality control processes on the labels they produce. Label unit cost is optimized by maximizing the quantity of labels that can be produced per labeler while maintaining quality, and minimizing the effort required to determine which items are best to label amongst all items that could be labeled. Label value is optimized by ensuring the labels the labelers are being tasked to produce have the highest possible improvement of the models trained using those labels.

The utilization and efficiency of labeling resources that are used to both train and evaluate a given perception system are increased by automatically identifying high-training value defects to be used in training. Certain embodiments exploit the property that scenes with objects identified as perception defects represent examples for which the machine-learned models being evaluated have not been well trained to process, and thus are likely to have a positive impact on model behavior if added to training/evaluation datasets. In effect, labels produced from scenes identified by a defect detection engine, such as 10, as uncovering sensitivities within the perception system, have higher expected value to the development process than labels produced from scenes selected through other means such as random sampling or subjective human decision-making. Further, the defect detection engine is an automated tool that does not require human interaction in order to identify scenes that uncover sensitivities within the perception system, thus lowering the time and effort needed to determine what items to label and consequently driving down the unit cost of producing labels. In other words, automated identification of defects is more scalable than human-assisted identification of defects. Note that in certain embodiments, human analysts are not used to hypothesize what the root cause of a perception defect might be; rather, labeling activities are automatically invoked in response to automated defect identification.

Certain embodiments provide a method for automatically identifying data items which, when labeled by human labeling teams, are expected to have a positive effect on subsequent model training operations which consume those labels. As illustrated in FIG. 2, autonomous vehicles (e.g. test vehicles) under the control of their autonomy stacks are tasked with driving out in the world (track or road). Road data 210 that includes both sensor and behavioral data concerning what the vehicle encountered and how it behaved are captured during those drives. Eventually, the road data 210 is uploaded to a storage device 215 and stored for later access by a workflow system 200. In certain embodiments that will be discussed later, this same workflow can equally be applied to data captured from simulated drives involving purely software simulations and/or hybrid Hardware in the Loop simulations.

The following steps outline the stages of an exemplary embodiment illustrated by FIG. 2. Job specification 205 is submitted to orchestrator 207, requesting a labeling run on some portion of available drive data. Orchestrator 207, which, in some embodiments, may be an algorithm expressed by software that controls the sequence of steps that need to be performed in order to accomplish a particular task, instructs defect detection engine 220 to execute a labeling run using a set of parameters sourced from the job specification 205. By way of instructive example, the specified input data can be selected from historical data found in sensor data storage 215.

In embodiments, the defect detection engine 220 is then instructed to perform one or more specified augmentations on the specified input data and then performs the specified detections using baseline and augmented data as inputs. Next, as a result of the evaluation, candidate defects are identified and stored in a defect database 225. Optionally, operational data (e.g., progress of data analysis) are provided to an operations dashboard and the orchestrator 207 is notified that the analysis is complete.

Next, orchestrator 207 instructs defect detection filter 235 that a new collection of candidate defects is available for filtering and transmission to a labeling facility 240. The filter 235 then extracts those defects from the current job that match the desired filtering criteria, produces a set of instructions to the labeling facility 240 indicating the scenes to be labeled, and transmits those instructions to the labeling facility 240. As used herein, the term “scene” refers to one complete sample in the environment that a particular sensor is designed for. By way of non-limiting example, a scene may be a single video frame or the data associated with a single Lidar sweep.

In certain embodiments, the filter configurator 245 optionally provides a way to filter some defects in order to concentrate on specific defects of engineering interest (example: perception weaknesses associated with pedestrians more than 250 pixels tall). An associated viewer displays examples of defects that satisfy the filter criteria for optional tuning by a user of the system. However, there is no requirement for the user to view any filtering results, nor is there a requirement for a filter to do any subsetting of the defect database. Thus, this part of the workflow can be performed with minimal or no human supervision.

The labeling facility 240 labels the scenes and artifacts requested, adds the results to the available set of labeled data at the training and validation database 250, and notifies orchestrator 207 that labeling has completed. Orchestrator 207 then directs the system to recommence perception system retraining 260. Once the machine learning models associated with the newly labeled data are retrained, and a newly trained perception engine 265 is produced responsive to training data identified by the preceding steps. Since this data provides labeled examples of data samples that the perception engine 265 was measured to perform weakly upon, there is an expectation that training on such samples will improve perception performance.

In certain embodiments, the steps will be executed on a repeated basis, as a way of continuously identifying new perception weaknesses and automatically incorporating re-labeled scenes into data sets used to train associated ML models. In embodiments, a performance dashboard displays current and historical perception engine performance data, such as reduction in defects over successive runs of the described workflow.

In certain embodiments of the present invention, an evaluator 270 and a re-engineer process 275 handle situations in which brute force retraining reveals systemic issues that are not amenable to such retraining. ML models evolve along two primary axes: learning and re-engineering and are able to gain capabilities and performance over time through continual expansion and curation of the datasets on which they are trained; as the set of training data gets larger and increases in quality, well constructed models will grow in capability and performance. However, ML models are also subject to local maxima, meaning that at various points in their development further training will not enable new capabilities or enhance performance. When such plateaus are reached, models must be re-engineered. Examples of re-engineering include architectural changes to the model and hyperparameter changes as are known to those skilled in the art (e.g., increasing number of layers in a deep network, topology, and training approach).

Certain embodiments of the present invention leverage signals inherent in results captured from a sequence of defect detection engine runs against an ML model trained on a growing collection of training data (such as that produced via FIG. 2). Evaluating the results of a succession of labeling runs is expected to reveal either progress towards an intended goal (indicating there is utility in continuing to re-train with additional data) or lack thereof (indicating a local maxima has been reached such that adding more labeled data will not make adequate progress, and the model needs to be re-engineered).

As mentioned, certain embodiments include monitoring the progress of an ML model across a sequence of training runs, to identify when additional training is no longer effective and model re-engineering is required. In particular, the orchestrator 207 instructs evaluator 270 to compare a newly created set of defects with the sets of defects generated from prior labeling runs against the same source dataset. If it is determined that meaningful progress was made by adding new labels, evaluator 270 instructs orchestrator 207 to continue the labeling. If it is determined that no meaningful progress was made by adding new labels, evaluator 270 instructs orchestrator 207 to initiate re-engineering with regard to the identified set of defects that have proven resistant to model retraining. Meaningful progress in this context is measured in terms of improvement in the Precision-Recall (PR) curve for the ML perception system. As used for this curve, “precision” what is the false positive percentage and recall is the false negative percentage. In embodiments, if the curve has improved in an amount greater than 0.5% percent, it is still considered meaningful progress such that re-engineering is not necessary. One non-limiting example of “meaningful improvement” is a run over run comparision of the perception system results wherein, after re-labeling, there was a 0.5% improvement in precision with no degradation in recall. Nevertheless, those of skill in the art will recognize that meaningful improvement can be characterized in different ways.

In embodiments, perception system re-engineering 275 may involve adjusting model hyperparameters, adding/changing layers, applying regularization techniques and other techniques. When complete, perception system re-engineering 275 submits new job specifications to Orchestrator 207 to restart labeling and other workflows using the newly adjusted perception engine 265.

In certain embodiments, the system identified in FIG. 2 may also be used to search for systemic training errors, according to the following steps:

A job specification 205 is submitted to the orchestrator 207, requesting a defect detection run on some portion of stored real-world drive data, whereupon orchestrator 207 instructs the defect detection engine 220 to execute an evaluation run, per the following procedure. Parameters used in this example are for purposes of illustration only and would be sourced from the job specification 205:

Data that has already been collected and which has available ground truth is selected from sensor data storage 215. Preferably, this data corresponds to data samples that are also in the training & validation database 250 and therefore have labels available. No augmentations are performed in the data for this embodiment. Specified detections are performed using baseline and ground truth data and candidate defects are stored in the defect database 225. Next, evaluator 270 compares the newly created set of defects against the sets of defects generated from prior evaluation runs against ground truth. If no meaningful progress, as previously described, has been made with the model since the last evaluation against ground truth, orchestrator 207 initiates the core re-engineering workflow described above. In certain embodiments, this comparison can involve selection of specific object types and characteristics to evaluate by reversing the order of execution of the evaluator 270 and the filter 235, and submitting the filtered-then-evaluated results for Perception System Re-Engineering 275.

In still further embodiments, the systems and methods set forth in FIG. 2 may also be used to search for classification errors, according to the following: a job specification 205 is submitted to orchestrator 207, requesting an evaluation run on some portion of stored real-world previously collected data. Orchestrator 207 instructs the defect detection engine 220 to execute a defect detection run. As before, parameters used in this example are for purposes of illustration only and would be sourced from the job specification 205. First, data with available ground truth is selected from the historical data sensor storage 215 and certain augmentations are performed on a copy of the data to enhance weakness detection as taught in PCT/US2019/59619. Specified detections are then identified using augmented and ground truth data and candidate defects are stored in the defect database 225. Next, the newly created set of defects are compared against the sets of defects generated from prior evaluation runs against ground truth. If no meaningful progress has been made with the model since the last evaluation against ground truth, orchestrator 207 initiates the core re-engineering workflow outlined above. As previously described, filter 235 and evaluator 270 can optionally be executed in reverse order. It is expected that this embodiment will be executed on a repeated basis, as a way of constantly evaluating the current state of the model and whether further training or re-engineering is required.

In still further embodiments, the quality of simulations used to train ML models can be improved, which will increase the trust placed in the simulation tooling as a result. Autonomous systems are tested via simulations as a way of expanding scenario coverage and exploring the existing testing space. When evaluating ML models against simulated data however, there is always a question of whether the behavior of a given ML model is due to the ML model itself or the quality of the simulation it was trained upon. Providing quantitative mechanisms for evaluating the quality of simulated objects enables both producers and users of simulation tooling to build trust in the results of running ML models against simulated worlds.

The embodiment illustrated in FIG. 3 leverages a latent signal embodied in the results captured from a sequence of defect detection runs against a given ML model trained on simulated data. When the defect detection engine identifies a particular feature or object in a simulated scene as producing a weakness within a given ML model, the scene (one or more data samples) can be automatically run through an evaluation process against existing ground truth information to assess whether the simulated scene data itself is a source of problems. Put another way, this embodiment outlines a method for capturing sensitivities of particular ML models to simulated features, and automatically invoking evaluations on those features to determine the quality of those features against a given set of quantitative measures.

Simulated vehicles under the control of their autonomy stacks are being tasked with driving through simulated worlds; sensor and behavioral data concerning what the vehicle encountered and how it behaved are being captured during those drives, and eventually moved into a repository accessible to components in the workflow system.

FIG. 3 illustrates certain embodiments of the present invention tailored to evaluate the quality of simulated objects. In this workflow 300, job specification 305 is submitted to orchestrator 310, requesting a labeling run on some portion of stored simulated drive data 315. This differs from the earlier embodiment where actual road data was used. Orchestrator 310 instructs the defect detection engine 320 to execute an evaluation run, per the following procedure. As before, parameters used in this example are for purposes of illustration only and would be sourced from the job specification 305.

Data with available ground truth is selected from sensor data storage 315, but augmentations on the data are not performed. Specified detections are performed using baseline and ground truth data by defect detection engine 320 and candidate defects are stored in the defect database 325. In certain embodiments, scenes related to the candidate defects are stored in the database 325 as well.

Next, orchestrator 310 initiates the evaluate simulated objects engine 330 to determine the rendering quality of objects in a scene identified as candidate defects. In certain embodiments, the workflow 300 will extract the rendering within each bounding box identifying a candidate defect, determine the type(s) of the simulated objects it contains, and execute an automated comparison function of those simulated objects against other examples of those objects stored in an existing catalog of labeled actual sensor footage imagery. In other embodiments, a learned model could be trained on a library of existing objects that have been labeled. Then, if there is a poor match, a new simulation can be created to see if the new object has higher confidence against what the model says it is. If so, it's better. Note that since a simulator has generated the sensor data, ground truth is known without need for human-assisted labeling.

If subsequent Quality Assessment 335 determines a rendered entity is of unacceptable quality, the corresponding candidate defect is flagged in the defect database 325 as requiring an update to the simulation world model. The Orchestrator 310 is notified that the Quality Assessment 335 is complete. If, however, any candidate defects remain that do not require an update to the simulation world model, the Orchestrator 310 initiates the remainder of a standard labeling workflow, proceeding next to the Filter 340 step, labeling 345, and deposit of labeled data in training database 360 with subsequent re-training 370 of the perception engine 380 as previously explained. If candidate defects have been flagged as requiring updates to the simulation world model, the orchestrator 310 provides the set of candidate defects and associated metadata to developers/maintainers of the simulation information base pursuant to the update world models step 350. Developers and maintainers of the simulation information base examine the candidate defect information and metadata, identify necessary improvements and update the affected world models. Once adjustments to the affected world models are complete, new simulator 355 runs are initiated to generate new simulation entries in sensor data storage 315 for use in future evaluation runs. It is expected that this embodiment will be executed on a repeated basis, as a way of constantly evaluating the quality of simulated features provided by the simulation tooling.

In still other embodiments, an automated process is provided for determining the triggering condition for defects discovered during model testing with simulated data. When developing an autonomous system it often becomes necessary to determine the root case, or “triggering condition,” for a witnessed behavior of the system. This process is generally a manual process, requiring highly trained and experienced human evaluators to examine individual scenes in a dataset, assert potential causes for the exhibited behavior, and run experiments to confirm or refute those assertions. The cost of executing this manual process is quite high, thus identifying ways to substantially or fully automate this process brings considerable value.

Certain embodiments of the disclosed system and method leverage the latent signal embodied in the results captured from a sequence of defect detection runs against a given ML model trained on simulated data. When the defect detection engine identifies a particular feature in a simulated scene as producing a sensitivity within a given ML model, it can be considered to have automatically identified an interesting condition for further evaluation. Further, having identified the defect from a scene originated in a simulated world, it has also identified a particular scenario and world model embodying the sensitivity. The scenario configuration and world models are parameterized, and thus can be used to generate a set of new simulations using the original scenario and/or world model with one or more altered parameters. Running the defect detection engine on the output from simulations created using these altered scenarios and/or world models will produce new sets of defects, which can be compared with the defects generated from the simulation run that triggered the original defect. If altering a particular parameter of the scenario or world model is shown to have a material effect on the set of defects generated by the defect detection engine over a sequence of runs, for some class of parameters the triggering condition for the defect can be inferred from the nature of the altered parameter.

This embodiment outlines a method for identifying candidate defects for root cause analysis of triggering conditions, and a mechanism for leveraging the defect detection engine to evaluate simulated driving data to automatically infer triggering conditions for those defects.

In certain embodiments, simulated vehicles under the control of their autonomy stacks are being tasked with driving through simulated worlds; sensor and behavioral data concerning what the vehicle encountered and how it behaved are being captured during those drives, and eventually moved into a repository accessible to components in the workflow system.

Still referring to FIG. 3, a modified version of the trigger identification process 300 will now be described. A job specification 305 is submitted to the orchestrator 310, which instructs the defect detection engine 320 to execute an exploration run. As before, parameters used in this example are for purposes of illustration only and would be sourced from the job specification 305. Data with available ground truth (for example, data in a training and validation database 360 that has already been labeled, or previously simulated data results) is selected. Then, specified detections are performed using baseline and ground truth data by defect detection engine 320. Candidate defects are then stored in the defect database 325, along with a reference to the scenario used to generate the originating simulation (for simulated data) or an annotated scenario description of labeled road data. Next, the orchestrator 310 directs that related simulations with one or more altered parameters be generated, enabling exploration of the simulation space adjacent to the features identified in the set of candidate defects.

In certain embodiments, the simulated objects within each bounding box for each candidate defect can be identified using associated ground truth and it can be determined, from past runs of the workflow, whether parameter alterations in the current simulation scenario or world model had a material effect on the set of detections generated in the current run. If the material effect is trending positively (i.e., indicates a correlation) but is not yet sufficient to infer causation, the next step is to generate one or more new simulation scenarios or world models that further alter the same parameters associated with entities identified by the current set of detections. If, however, the material effect is trending negatively or there was no material effect, one or more new simulation scenarios or world models are generated that alter different parameters associated with the entities identified by the current set of detections.

In the case of new simulation scenarios or world models, new job specifications 305 are submitted for additional workflow runs against the new simulation instances. If sufficient causality between parameter alteration of one or more scenes or world model can be determined, the set of parameter alteration(s) are recorded with the set of candidate defects in the defect database 325, and the overall workflow terminates. If insufficient causality between parameter alteration of one or more scenes or world model can be determined, the set of defects is marked for human review and the overall workflow terminates.

Examples of parameters that can be altered include object properties (e.g., color, shape, size, presentation angle or actors, vehicles, road furniture, and the like), backgrounds (e.g., color, texture, brightness), scene characteristics (e.g., contrast, glare), environmental characteristics (e.g., haze, fog, snow, sleet, rain), and occlusion (slight occlusion, partial occlusion, heavy occlusion).

It is expected that this embodiment will be executed on a repeated basis, as a way of constantly searching for defects and identifying their triggering conditions. It will be apparent to one of skill in the art that labelling could be completely automated provided a sufficiently accurate automated perception labeling capability is available, or could be semi-automated.

The various embodiments of the invention can be advantageously used to identify good candidates for perception retraining, identify systematic perception system weaknesses, and identify systemic defects in simulator outputs. The invention can be implemented via physical devices (e.g., an augmentation physical computing module comprising hardware and software that modifies data streams sent to a separate perception system as well as a physical computing module that performs comparison) as well as via purely software mechanisms hosted on a convenient computing platform.

Certain aspects of the embodiments set forth herein include process steps. It should be noted that the process steps of the embodiments could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by a variety of operating systems. The embodiments can also be in a computer program product that can be executed on a computing system.

The embodiments also relate to a system for performing the operations herein. This system may be specially constructed for the purpose, e.g., a specifically constructed or selected array of computing devices, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. The memory/storage can be transitory or non-transitory. Memory can include any of the above and/or other devices that can store information/data/programs. Furthermore, the computing devices referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Although the present invention has been illustrated and described herein with reference to preferred embodiments and specific examples thereof, it will be readily apparent to those of ordinary skill in the art that other embodiments and examples may perform similar functions and/or achieve like results. All such equivalent embodiments and examples are within the spirit and scope of the present invention, are contemplated thereby, and are intended to be covered by the claims.

Claims

1. A computer-implemented method for improving the labeling performance of a machine-learned (ML) perception system on an input dataset, the method comprising processing, by a computing device, the input dataset through the perception system; identifying, by a computing device, one or more perception weaknesses;isolating, by a computing device, at least one scene from the input dataset that includes the one or more perception weaknesses;relabeling the one or more perception weaknesses in the at least one scene to obtain at least one relabeled scene; andretraining, by a computing device, the perception system with the at least one relabeled scene; whereby the labeling performance of the ML perception system is improved.

2. The computer-implemented method of claim 1, wherein the step of identifying one or more perception weaknesses is performed by a defect detection engine.

3. The computer-implemented method of claim 1, wherein the one or more perception weaknesses comprise labeled objects in bounding boxes.

4. The computer-implemented method of claim 1, wherein the step of processing the input dataset further comprises creating, by a computing device, an augmented dataset, processing, by a computing device, the augmented dataset through the perception system, and comparing, by a computing device, results of the input data set with results of the augmented dataset.

5. The computer-implemented method of claim 1, wherein the step of relabeling the at least one scene comprises sending the at least one scene to be labeled by a human.

6. The computer-implemented method of claim 1, wherein the input dataset is comprised of one or more of baseline data, augmented data, ground truth data, and simulated data.

7. The computer-implemented method of claim 1, further comprising filtering, by a computing device, the results of the processing step to concentrate on specific perception weaknesses of engineering interest.

8. The computer-implemented method of claim 1, further comprising repeating the method and comparing the perception weaknesses with the perception weaknesses identified in the processing step that occurred prior to the relabeling to determine an amount of improvement in the performance of the perception system.

9. The computer-implemented method of claim 8, further comprising repeating the steps of processing, identifying, isolating, relabeling and retraining until retraining the ML perception system with the newly labeled scenes does not meaningfully improve the labeling performance.

10. The computer-implemented method of claim 9, further comprising reengineering the ML perception system when the labeling performance does not meaningfully improve with the newly labeled scenes.

11. A computer-implemented method for evaluating the quality of simulated data used to train a machine-learned (ML) perception system comprising: processing, by a computing device, an input dataset comprising simulated data through a perception system to obtain one or more candidate defects;extracting, by a computing device a simulated object rendering related to the one or more candidate defects;comparing, by a computing device, the simulated object rendering to other examples of the object; andflagging, by a computing device, a simulated object having low confidence matching to the other examples.

12. The computer-implemented method of claim 11, wherein the simulated object rendering is compared with a learned model that is trained, by a computing device, on a library of previously labeled objects.

13. The computer-implemented method of claim 11, further comprising updating, by a computing device, a simulated world model by replacing flagged simulated objects with new simulated objects.

14. A computer-implemented method of determining a triggering condition for perception weaknesses identified during perception engine testing with simulated data, the method comprising: processing, by a computing device, an input dataset comprising simulated data through a perception system to get a first set of results;processing, by a computing device, an input dataset comprising augmented simulated data through the perception system to get a second set of results;comparing, by a computing device, the first set of results with the second set of results using a defect detection engine to identify perception weaknesses caused by simulated objects, the simulated objects defined by a plurality of parameters;altering, by a computing device, at least one parameter to create an altered simulated object; repeating the processing and comparing steps with the altered simulated object to determine whether the altered simulated object affected the perception weaknesses.

15. The computer-implemented method of claim 14, further comprising repeating the processing, comparing and altering steps until the triggering condition is identified.

16. A computer system for improving the labeling performance of a machine-learned (ML) perception system on an input dataset, the computer system comprising: a memory or other data storage facility;one or more processors; anda non-transitory computer-readable medium that stores instructions that when executed by the one or more processors, causes them to:process the input dataset through the perception system to identify one or more perception weaknesses;isolate at least one scene from the input dataset that includes the one or more perception weaknesses;relabel the one or more perception weaknesses in the at least one scene to obtain at least one relabeled scene; andretrain the perception system with the at least one relabeled scene;whereby the labeling performance of the ML perception system is improved.

17. A non-transitory computer-readable medium for improving the labeling performance of a machine-learned (ML) perception system on an input dataset, comprising instructions stored thereon, that when executed on a processor, perform the steps of: processing the input dataset through the perception system to identify one or more perception weaknesses;isolating at least one scene from the input dataset that includes the one or more perception weaknesses;relabeling the one or more perception weaknesses in the at least one scene to obtain at least one relabeled scene; andretraining the perception system with the at least one relabeled scene; whereby the labeling performance of the ML perception system is improved.