Patent Grant
11100371
The present invention, in some embodiments thereof, relates to a system for creating sensory data for testing an autonomous system, and, more specifically, but not exclusively, to creating sensory data for testing an advanced driver-assistance system or an autonomous driving system.
The term “autonomous driving system” refers to a vehicle that is capable of sensing its environment and moving safely with some human input. The term “advanced driver-assistance system (ADAS)” refers to a system that aids a vehicle driver while driving by sensing its environment. A vehicle comprising an ADAS may comprise one or more sensors, each capturing a signal providing input to the ADAS. There exist myriad conditions that may affect the signal captured by a sensor of the vehicle. For example, a first signal, captured by the sensor when mounted on the vehicle in a first position on the vehicle while the vehicle traverses a physical scene, for example a geographical area, may be different from a second signal, captured by the sensor when mounted on the vehicle in a second position on the vehicle while the vehicle traverses the physical scene. In addition, a physical characteristic of the sensor may affect the signal captured by the sensor. Some examples of a physical characteristic that may affect the captured signal are the sensor's sensitivity, the sensor's color intensity, the sensor's field of view, a material the sensor is made of, and the sensor's construction. There is a need to train, test, validate and verify an ADAS according to a target sensor.
It is an object of the present invention to provide a system and a method for creating synthetic data for a simulation.
The foregoing and other objects are achieved by the features of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.
According to a first aspect of the invention, a system for creating synthetic data for testing an autonomous system comprises at least one hardware processor adapted to execute a code for: using a machine learning model to compute a plurality of computed depth maps based on a plurality of real signals, the plurality of real signals are captured simultaneously from a common physical scene, each of the plurality of real signals are captured by one of a plurality of sensors, each of the plurality of computed depth maps qualifies one of the plurality of real signals; applying a point of view transformation to the plurality of real signals and the plurality of computed depth maps, to produce synthetic data simulating a possible signal captured from the common physical scene by a target sensor in an identified position relative to the plurality of sensors; and providing the synthetic data to at least one testing engine to test an autonomous system comprising the target sensor.
According to a second aspect of the invention, a method for creating synthetic data for testing an autonomous system comprises using a machine learning model to compute a plurality of computed depth maps based on a plurality of real signals, the plurality of real signals are captured simultaneously from a common physical scene, each of the plurality of real signals are captured by one of a plurality of sensors, each of the plurality of computed depth maps qualifies one of the plurality of real signals; applying a point of view transformation to the plurality of real signals and the plurality of computed depth maps, to produce synthetic data simulating a possible signal captured from the common physical scene by a target sensor in an identified position relative to the plurality of sensors; and providing the synthetic data to at least one testing engine to test an autonomous system comprising the target sensor.
According to a third aspect of the invention, a system for testing an autonomous system, comprising a target sensor comprises at least one hardware processor adapted to execute a code for: producing synthetic data simulating a possible signal captured from a common physical scene by the target sensor in an identified position relative to a plurality of sensors, where the synthetic data is produced using a plurality of real signals, the plurality of real signals are captured simultaneously from the common physical scene by the plurality of sensors; and testing the autonomous system using the synthetic data. Optionally, wherein producing the synthetic data using the plurality of real signals comprises: using a machine learning model to compute a plurality of computed depth maps based on the plurality of real signals, the plurality of real signals are captured simultaneously from the common physical scene, each of the plurality of real signals are captured by one of the plurality of sensors, each of the plurality of computed depth maps qualifies one of the plurality of real signals; applying a point of view transformation to the plurality of real signals and the plurality of computed depth maps, to produce synthetic data simulating the possible signal captured from the common physical scene by the target sensor in an identified position relative to the plurality of sensors; and providing the synthetic data to at least one testing engine to test the autonomous system.
According to a fourth aspect of the invention, a system for detecting a plurality of objects in sensor data comprises at least one hardware processor adapted to execute a code for: using a machine learning model to compute a plurality of computed depth maps based on a plurality of real signals, the plurality of real signals are captured simultaneously from a common physical scene, each of the plurality of real signals are captured by one of a plurality of sensors, each of the plurality of computed depth maps qualifies one of the plurality of real signals; and providing the plurality of computed depth maps to at least one object classification engine to detect at least one object in the plurality of real signals.
With reference to the first and second aspects, in a first possible implementation of the first and second aspects of the present invention, the at least one hardware processor is adapted to further execute code for: applying a style transformation to the synthetic data according to real target sensor data captured by the target sensor from another physical scene, to produce target synthetic data simulating target data captured by the target sensor; and using the target synthetic data as the synthetic data. Optionally, the at least one hardware processor is adapted to further execute code for: applying one or more physical-characteristic adjustments to the synthetic data to produce adjusted target synthetic data simulating target data captured by the target sensor; and using the adjusted target synthetic data as the synthetic data. Optionally, at least one of the one or more physical-characteristic adjustments is selected from a group of physical-characteristic adjustments consisting of: a geometric distortion transformation applied according to at least one physical characteristic of the target sensor, a change of image resolution, and a scan-pattern filter applied according to an identified scanning pattern of the target sensor. Optionally, the geometric distortion transformation is selected from a group of geometric distortion transformations comprising: a fisheye transform, a barrel distortion transform, a pincushion distortion transform, and a moustache distortion transform. Optionally, the synthetic data comprises a plurality of synthetic digital images, and applying the scanning-pattern filter according to the identified scanning pattern of the target sensor comprises using optical flow motion estimation to compute at least one additional synthetic digital image. Optionally applying the point of view transformation comprises applying a dynamic transformation to the identified position. Applying a style transformation and additionally or alternatively applying one or more physical-characteristic adjustments to the synthetic data increases accuracy of the synthetic data and thus increases accuracy of an output of a testing engine using the synthetic data.
With reference to the first and second aspects, in a second possible implementation of the first and second aspects of the present invention, the plurality of sensors are a plurality of cameras; the target sensor is a target camera; each of the plurality of real signals comprises a plurality of digital images captured from the common physical scene by respective camera of the plurality of cameras; and each of the plurality of computed depth maps comprises a plurality of depth images, each comprising depth information for a digital image of respective plurality of digital images qualified by the computed depth map. Optionally, each depth image of the plurality of depth images comprises depth information for each of a plurality of pixels of respective digital image. Optionally, the synthetic data comprises a plurality of synthetic digital images, and the at least one hardware processor is adapted to further execute code for: applying a hole-filling algorithm to the plurality of synthetic images of the synthetic data to produce filled target synthetic data simulating target data captured by the target sensor; and using the filled target synthetic data as the synthetic data. Applying a hole-filling algorithm to the plurality of synthetic images, of the synthetic data, increases accuracy of the synthetic data and thus increases accuracy of an output of a testing engine using the synthetic data.
With reference to the first and second aspects, in a third possible implementation of the first and second aspects of the present invention, the at least one hardware processor is adapted to further execute code for: computing a plurality of ground-level layer maps each for one of the plurality of real signals, using the plurality of computed depth maps; adding at least one additional object to the synthetic data according to the plurality of computed depth maps and the plurality of ground-level layer maps to produce augmented synthetic data; and using the augmented synthetic data as the synthetic data. Optionally, the at least one additional object is selected from a group of scene objects consisting of: a mark on a road, a speed bump, a depression in a road surface, an elevation in a road surface, a traffic light, a sign, a road side, a curb, an animal, a pedestrian, a bicycle, and a motorized vehicle. Optionally, the at least one hardware processor is adapted to further execute code for detecting, in the synthetic data, a plurality of existing objects, using the plurality of computed depth maps. Optionally, adding the at least one additional object to the synthetic data comprises using at least one of the plurality of existing objects. Augmenting the synthetic data by inserting one or more additional objects improves accuracy of an output of an autonomous system trained and additionally or alternatively tested by a testing engine using the synthetic data. Using a plurality of ground-level layer maps generated using the plurality of computed depth maps to insert the one or more additional objects in the synthetic data increases accuracy of the synthetic data, thus improving accuracy of an output of a testing engine using the synthetic data. Using at least one of the plurality of existing objects when adding the at least one additional object increases accuracy of the synthetic data comprising the at least one additional object, thus improving accuracy of an output of a testing engine using the synthetic data.
With reference to the first and second aspects, in a fourth possible implementation of the first and second aspects of the present invention, at least one first physical quality value of at least one physical quality, measured while capturing the plurality of real signals by the plurality of sensors, is greater than at least one second physical quality value of the at least one physical quality, measured while capturing the real target sensor data by the target sensor. Optionally, the at least one first physical quality value comprises a first value of a first distance the plurality of sensors traversed in the common physical scene while capturing the plurality of real signals; and the at least one second physical quality value comprises a second distance value of a second distance the target sensor traversed in the other physical scene while capturing the real target sensor data. Using real target sensor data captured over a second distance shorter than a first distance over which the plurality of real signals are captured reduces a cost of producing target synthetic data simulating target data captured by the target sensor over the first distance. Optionally, a ratio between the first distance value and the second distance value is greater than an identified ratio threshold. Optionally, the identified ratio threshold is selected from a group of ratio thresholds comprising 10, 100 and 1000.
With reference to the first and second aspects, in a fifth possible implementation of the first and second aspects of the present invention, the plurality of real signals are captured while the plurality of sensors are mounted on a vehicle in an identified pattern, the real target sensor data is captured while the target sensor is mounted on the vehicle at the identified position, and the identified position is an identified relative position relative to the identified pattern. Optionally, the identified pattern is a grid. Using a two-dimensional or three-dimensional pattern increases an amount of possible positions of the target sensor for which interpolation may be used to compute the point of view transformation compared to using a one dimensional pattern, thus increasing accuracy of the synthetic data for more possible positions than when using a one dimensional pattern.
With reference to the first and second aspects, in a sixth possible implementation of the first and second aspects of the present invention, the at least one hardware processor is adapted to further execute code for: applying a new point of view transformation to the plurality of real signals and the plurality of computed depth maps, to produce new synthetic data simulating a new possible signal captured from the common physical scene by a new target sensor in a new identified position relative to the plurality of sensors; and providing the new synthetic data to the at least one testing engine to test another autonomous system comprising the new target sensor. Optionally, the at least one hardware processor is adapted to further execute code for: applying a new style transformation to the new synthetic data according to new real target sensor data captured by the new target sensor from a new other physical scene, to produce new target synthetic data simulating new target data captured by the new target sensor; and using the new target synthetic data as the new synthetic data. Optionally, the at least one hardware processor is adapted to further execute code for: applying one or more new physical-characteristic adjustments to the synthetic data to produce new adjusted target synthetic data simulating new target data captured by the new target sensor; and using the new adjusted target synthetic data as the new synthetic data. Optionally, at least one of the one or more new physical-characteristic adjustments is selected from a new group of physical-characteristic adjustments consisting of: a new geometric distortion transformation applied according to at least one physical characteristic of the new target sensor, a new change of image resolution, and a new scan-pattern filter applied according to a new identified scanning pattern of the new target sensor. Optionally, the new geometric distortion transformation is selected from a group of geometric distortion transformations consisting of: a fisheye transform, a barrel distortion transform, a pincushion distortion transform, and a moustache distortion transform. Optionally, the new synthetic data comprises a plurality of new synthetic digital images. Optionally, applying the new scanning-pattern filter according to the new identified scanning pattern of the new target sensor comprises using optical flow motion estimation to compute at least one additional new synthetic digital image. Optionally, the new synthetic data comprises a new plurality of synthetic digital images, and the at least one hardware processor is adapted to further execute code for: applying a new hole-filling algorithm to the new plurality of synthetic images of the new synthetic data to produce new filled target synthetic data simulating new target data captured by the new target sensor; and using the new filled target synthetic data as the new synthetic data. Optionally, the at least one hardware processor is adapted to further execute code for: computing a plurality of ground-level layer maps, each for one of the plurality of real signals, using the plurality of computed depth maps; adding at least one new additional object to the new synthetic data according to the plurality of computed depth maps and the plurality of ground-level layer maps to produce new augmented synthetic data; and using the new augmented synthetic data as the new synthetic data. Optionally, the at least one new additional object is selected from a group of scene objects consisting of: a mark on a road, a speed bump, a depression in a road surface, an elevation in a road surface, a traffic light, a sign, a road side, a curb, an animal, a pedestrian, a bicycle, and a motorized vehicle. Optionally, the at least one hardware processor is adapted to further execute code for detecting, in the new synthetic data, a plurality of new existing objects, using the plurality of computed depth maps. Optionally, adding the at least one new additional object to the new synthetic data comprises using at least one of the plurality of new existing objects. Optionally, applying the new point of view transformation comprises applying a new dynamic transformation to the new identified position. Using the plurality of computed depth maps to produce new synthetic data simulating new target data captured by the new target sensor reduces costs of testing another autonomous system comprising the new target sensor.
With reference to the first and second aspects, in a seventh possible implementation of the first and second aspects of the present invention, the autonomous system is selected from a group of autonomous systems consisting of: an autonomous driving system, and an advanced driver-assistance system (ADAS).
With reference to the third aspect, in a first possible implementation of the third aspect of the present invention, the at least one hardware processor is adapted to further execute code for: computing a plurality of ground-level layer maps, each for one of the plurality of real signals, using the plurality of computed depth maps; accessing a plurality of detected objects, identified in the synthetic data by the autonomous system while testing the autonomous system; and validating the plurality of detected objects according to the plurality of ground-level layer maps, the plurality of computed depth maps and the synthetic data. Optionally, validating the plurality of detected objects according to the plurality of ground-level layer maps comprises applying at least one validation rule to the plurality of ground-level layer maps and at least one of the plurality of detected objects. Optionally, the at least one hardware processor is adapted to further execute code for: detecting in the synthetic data a plurality of target objects; and producing a ground-truth comprising a plurality of bounding-boxes each embedding one of the plurality of target objects, the ground-truth produced using the plurality of computed depth maps. Optionally, validating the plurality of detected objects further comprises validating the plurality of detected objects according to the ground-truth and the plurality of computed depth maps. Optionally, at least one of the plurality of bounding-boxes is one of: a two-dimensional bounding-box embedding one of the plurality of target objects, and a three-dimensional bounding-box embedding one of the plurality of target objects. Optionally, validating the plurality of detected objects according to the ground-truth and the plurality of computed depth maps comprises executing at least one other machine learning model. Validating the plurality of detected objects using a plurality of ground-level layer maps increases accuracy of an object validation compared to validating without using ground-level layer information. Producing the plurality of ground-level layer maps using the plurality of computed depth maps increases accuracy of the plurality of ground-level layer maps compared to other ground-level layer maps computed without using a plurality of depth maps, and thus increases accuracy of an object validation.
With reference to the fourth aspect, in a first possible implementation of the fourth aspect of the present invention, the at least one hardware processor is adapted to further execute code for: computing a plurality of ground-level layer maps, each for one of the plurality of real signals, using the plurality of computed depth maps; and validating the at least one object according to the plurality of ground-level layer maps, the plurality of computed depth maps and the synthetic data. Optionally, validating the at least one object according to the plurality of ground-level layer maps comprises applying at least one validation rule to the plurality of ground-level layer maps and the at least one object. Validating the at least one object according to the plurality of ground-level layer maps, generated using the plurality of computed depth maps, improves accuracy of an output of the at least one object classification engine.
Other systems, methods, features, and advantages of the present disclosure will be or become apparent to one with skill in the art upon examination of the following drawings and detailed description. It is intended that all such additional systems, methods, features, and advantages be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims.
Unless otherwise defined, all technical and/or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the invention, exemplary methods and/or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.
Some embodiments of the invention are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the invention. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the invention may be practiced.
In the drawings:
The present invention, in some embodiments thereof, relates to a system for creating sensory data for testing an autonomous system, and, more specifically, but not exclusively, to creating sensory data for testing an ADAS or an autonomous driving system.
The following description focuses on a non-limiting example of a system for testing an autonomous system, however synthetic data created according to the present invention may be used in a similar manner to alternatively or additionally verify the autonomous system, alternatively or additionally train the autonomous system and alternatively or additionally validate the autonomous system. As used herein, the term “testing engine” refers to an engine for performing one or more operations on an autonomous system, where the one or more operations are selected from a group of: testing the autonomous system, validating the autonomous system, verifying the autonomous system and training the autonomous system.
In addition, the following description focuses on a system for testing an ADAS, however the present invention is not limited to the field of automotive autonomous systems and may be used with an autonomous system from another field, for example an autonomous production process control system.
Some methods for testing an ADAS comprising a target sensor comprise capturing real testing data by the target sensor, mounted at a target position on a vehicle, while traversing the physical scene. Such methods require capturing new real testing data any time some of the sensor's physical characteristics are modified or when the sensor is mounted in a new position on the vehicle. For example, changing a hardware component of the sensor may change some of the sensor's physical characteristics such as its sensitivity to light, its color intensity and distortion of a signal captured by the sensor. Capturing the real testing data may be expensive, for example due to a physical quality measured while capturing the real testing data such as an amount of time required to capture the real testing data or a distance a vehicle is required to traverse to capture the real testing data. In addition, there may be a need to produce large data sets of testing data, further increasing a cost of capturing the real testing data, for example due to an increased amount of time to capture the real testing data or an increased distance over which the real testing data is captured.
As used henceforth, the term “synthetic data” means data generated by a computerized generation system and not captured by a sensor.
Using synthetic data to test the ADAS is an alternative to using the real testing data captured by the sensor. However, there is a need to produce synthetic testing data that exhibits characteristics according to the target sensor. For example, for visual data, there is a need to produce synthetic data according to the target sensor's position on the vehicle and according to the target sensor's sensitivity. There exist methods of producing synthetic data simulating a possible signal captured from a common physical scene by applying a point of view transformation to two or more signals captured by two or more sensors from the common physical scene. For example, there exist methods of computing a point of view transformation applied to two or more signals, each comprising a plurality of captured digital images captured by one of two or more image sensors, to produce a plurality of synthetic digital images simulating a possible signal captured by a target image sensor. Such methods may compute an interpolation or an extrapolation of the two or more signals according to a relation between a target position on the vehicle of the target image sensor and two or more known positions on the vehicle of the two or more image sensors. However, the plurality of synthetic digital images produced by such methods may be distorted compared to the possible signal, as the accuracy of the plurality of synthetic digital images is impacted by availability and quality of depth maps qualifying the plurality of images of the two or more signals. A depth map is an image or image channel that contains information relating to a distance of surfaces or points of scene objects from a viewpoint. When an image is captured by a sensor, a depth map contains information relating to a distance of surfaces or points of scene objects from the sensor. Estimating depth is an important component of understanding geometric relations within a scene. Some existing sensors do not always capture depth information correlated with a captured signal. Some other existing sensors provide low quality depth maps, qualifying only some of a plurality of pixels of each of the plurality of digital images of a signal. In addition, there is a need to align the two or more signals. There is a need to reduce the distortion of the plurality of synthetic digital images compared to the possible signal captured by a target image sensor.
The present invention proposes, in some embodiments thereof, capturing a plurality of real signals simultaneously from a common physical scene by a plurality of sensors and generating a plurality of computed depth maps based on the plurality of real signals, each computed depth map qualifying one of the plurality of real signals, and applying a point of view transformation to the plurality of real signals and the plurality of computed depth maps to produce synthetic data simulating a possible signal captured from the common physical scene by a target sensor in an identified position relative to the plurality of sensors.
Using a plurality of real signals captured simultaneously increases an alignment among the plurality of real signals and increases an accuracy of the synthetic data such that a distortion of the synthetic data with relation to the plurality of real signals is reduced compared to using some existing methods. In some embodiments of the present invention each of the plurality of real signals comprises a plurality of digital images, and each of the plurality of computed depth maps is a dense computed depth map, comprising depth information for each of a plurality of pixels of the plurality of digital images of the respective real signal qualified by the computed depth map. Using dense computed depth maps when applying the point of view transformation improves the accuracy of the synthetic data and thus ultimately improves accuracy of an output of a testing engine using data produced based on the synthetic data. Using interpolation to compute the point of view transformation produces a more accurate point of view transformation than using extrapolation to compute the point of view transformation. According to the present invention, in some embodiments the plurality of sensors is organized in an identified pattern, for example a two-dimensional pattern, for example a grid. Using a two-dimensional pattern increases an amount of possible positions of the target sensor for which interpolation may be used to compute the point of view transformation compared to using a one dimensional pattern, thus increasing accuracy of the synthetic data for more possible positions than when using a one dimensional pattern. Optionally, the identified pattern is three-dimensional.
The present invention further proposes, in some embodiments, applying a style transformation to the synthetic data according to real target sensor data captured by the target sensor from another physical scene, to produce target synthetic data simulating target data captured by the target sensor, and using the target synthetic data as the synthetic data in further processing. Applying a style transformation to the synthetic data according to real target sensor data captured by the target sensor increases accuracy of the synthetic data, and thus increases accuracy of an output of a testing engine using the synthetic data.
The present invention further proposes, in some embodiments, applying one or more physical characteristic adjustments to the synthetic data, to produce adjusted target synthetic data simulating target data captured by the target sensor, and using the adjusted target synthetic data as the synthetic data in further processing. Optionally, the one or more physical characteristic adjustments comprise a change of image resolution. Optionally, the one or more physical characteristic adjustments comprise applying a sensor scan-pattern filtering, according to an identified scanning pattern of the target sensor, for example when the target sensor has a scanning beam. Some examples of a sensor having a scanning beam are a radar and a light detection and ranging (LIDAR) sensor. Optionally, applying the scan-pattern filtering comprises using optical flow motion estimation, for example to compute one or more intermediate digital images and additionally or alternatively one or more points on at least one surface identified in one or more images of the synthetic data. Using optical flow motion estimation allows frame rate up-conversion and additionally or alternatively generating in-between frames data, increasing accuracy of the synthetic data after applying the scan-pattern filtering. Optionally, the one or more physical characteristics comprise applying a geometric distortion transformation according to one or more physical characteristics of the target sensor. Some more examples of a physical characteristic of a sensor are a lens distortion and a lens field of view. Applying one or more physical characteristics adjustments to the synthetic data increases accuracy of the synthetic data, and thus improves accuracy of an output of a testing engine using the synthetic data.
There may be a need to enrich the synthetic data, augmenting the synthetic data with additional objects. Some examples of an additional object are a mark on a road, for example a road mark on ground level, such as a lane mark, an arrow and a pedestrian crossing. Some other examples of an additional object are a depression in a road surface, for example a depression indicative of a hole in a road, and an elevation in a road surface, for example an elevation due to accumulated snow. Some more of an additional object are a sign, a traffic light, a curb, a speed bump, a pedestrian, an animal, a bicycle, and a motorized vehicle. There is a need to insert the additional objects into the synthetic data such that the additional objects are correctly aligned with other surfaces and objects in the synthetic data and are sized in proportion to the other surfaces and objects. To augment the synthetic data, the present proposes in some embodiments thereof proposes using the plurality of computed depth maps to compute a plurality of ground-level layer maps, each for one of the plurality of real signals, and inserting in the synthetic data one or more additional objects according to the plurality of computed depth maps using the plurality of ground-level layer maps. Optionally, each of the plurality of ground-level layer maps comprises a ground-level surface as captured in the respective real signal. Optionally, each of the plurality of ground-level layer maps comprises one or more height values of one or more objects captured in the respective real signal, where the one or more height values are relative to the ground-level surface. Augmenting the synthetic data by inserting one or more additional objects improves accuracy of an output of an autonomous system trained and additionally or alternatively tested by a testing engine using the synthetic data. Using a plurality of ground-level layer maps generated using the plurality of computed depth maps to insert the one or more additional objects in the synthetic data increases accuracy of the synthetic data, thus improving accuracy of an output of a testing engine using the synthetic data. Optionally, the one or more additional objects are inserted using one or more of a plurality of detected objects, detected in the synthetic data, for example in order to align the one or more additional objects in one or more scenes of the synthetic data. Using the one or more detected objects increases accuracy of the synthetic data comprising the one or more additional objects. In addition, in some embodiments the present invention further proposes producing the plurality of computed depth maps using a machine learning model trained based on a plurality of synthetic training signals, each simulating one of a plurality of signals captured by the plurality of sensors from a common training scene, and a plurality of training depth maps, each qualifying one of the plurality of synthetic training signals according to the common training scene such that the training depth map provides ground truth for the respective synthetic training signal. Using a plurality of synthetic training signals simulating a plurality of signals captured by the plurality of sensors increases an accuracy of an output of the machine learning model in response to the plurality of real signals captured by the plurality of sensors, compared to an alternative output in response to the plurality of signals captured by the plurality of sensors when the machine learning model is trained using an alternative plurality of synthetic training signals that do not simulate the plurality of signals captured by the plurality of sensors. Increasing the accuracy of the output of the machine learning model increases an accuracy of the synthetic data and thus increases accuracy of an output of a testing engine using the synthetic data. In some embodiments, the present invention further proposes applying another style transformation to the plurality of synthetic training signals according to the real target sensor data and training the machine learning model using a plurality of transformed synthetic training signals produced by applying the other style transformation. Training the machine learning model using signals transformed according to the real target sensor data improves the accuracy of an output of the machine learning model in response to the plurality of real signals, thus improving the accuracy of the synthetic data, and thus improves accuracy of an output of a testing engine using the synthetic data.
As used herein, the term self-supervised learning refers to implicitly and autonomously extracting correlations, embedded metadata, or domain knowledge available within the input to a machine learning model and using this extracted data as supervisory signals in training the machine learning model, without a need to explicitly label the input data. The present invention proposes, in some embodiments, to further use self-supervised learning when training the machine learning model. In such embodiments, a plurality of test depth maps produced by the machine learning model based on at least some of the plurality of real sensor signals are used with at least one of the plurality of real sensor signals to generate at least one reconstructed sensor signal associated with at least one sensor of the plurality of sensors. In such embodiments the at least one of the plurality of real sensors is associated with a second sensor of the plurality of sensors. According to the present invention, in such embodiments a difference between the at least one reconstructed sensor signal and at least one other real sensor signal associated with the at least one first sensor is used as a supervisory signal to the machine learning model such that at least one model value of the machine learning model is modified to minimize the difference. Using self-supervised learning when training the machine learning model improves accuracy of the plurality of computed depth maps, which in turn improves the accuracy of the synthetic data.
There may be a need to produce testing data for testing an autonomous system for more than one sensor. Costs of producing the testing data for the more than one sensor is impacted by a cost of producing the testing data for each of the more than one sensor. For example, when the testing data comprises real new testing data captured for each of the more than one sensors, costs of producing the testing data may be impacted by an amount of time to capture the new real new testing data or a distance over which the real new testing data is captured. The present invention, in some embodiments thereof, proposes using the plurality of real signals and the plurality of computed depth maps to produce more testing data for more than one target sensor. In such embodiments, the present invention proposes applying a new point of view transformation to the plurality of real signals and the plurality of computed depth maps to produce new synthetic data simulating a new possible signal captured from the common physical scene by a new target sensor in a new identified position relative to the plurality of sensors, and optionally applying a new style transformation according to new real target sensor data captured by the new target sensor from a new physical scene, to produce new synthetic data simulating new target data captured by the new target sensor. The present invention further proposes, in some embodiments, applying one or more new physical characteristic adjustments to the new synthetic data to produce new adjusted target synthetic data simulating the new target data, and using the adjusted target synthetic data as the new synthetic data in further processing. Optionally, the one or more new physical characteristic adjustments comprise a new change of image resolution. Optionally, the one or more new physical characteristic adjustments comprise applying a new sensor scan-pattern filtering, according to a new identified scanning pattern of the new target sensor. Optionally, applying the new scan-pattern filtering comprises using optical flow motion estimation, for example to compute one or more new intermediate digital images and additionally or alternatively one or more points on at least one surface identified in one or more new images of the new synthetic data. Using optical flow motion estimation allows frame rate up-conversion, increasing accuracy of the new synthetic data after applying the new scan-pattern filtering. Optionally, the one or more new physical characteristic adjustments comprise applying a new geometric distortion transformation according to one or more new physical characteristics of the new target sensor. Using the plurality of real signals and the plurality of computed depth maps to produce new synthetic data for a new target sensor reduces overall costs of producing a plurality of synthetic data sets simulating data captured by the more than one sensor.
In addition, in some embodiments thereof, the present invention proposes using real target sensor data such that a first physical quality value of a physical quality measured when capturing the plurality of real signals is different from a second physical quality value of the physical quality measured when capturing the real target sensor data. For example, a first distance that a vehicle traverses in the common physical scene when the plurality of real signals are captured may be greater than a second distance another vehicle traverses the other physical scene when capturing the real target sensor signal is captured. In another example, a first atmospheric pressure value measured when capturing the plurality of real signals may be greater than a second atmospheric pressure value when capturing the real target sensor data. The difference between the first physical quality value and the second physical quality value facilitates reducing costs of capturing the real target sensor data, for example reducing cost of traversing a long distance or reducing cost of producing a high atmospheric pressure, which in turn reduces a cost of producing the synthetic data compared to a cost of capturing real target sensor data such that the second physical quality value is equivalent to the first physical quality value. In addition, a cost of producing the new synthetic data according to the present invention is lower than a cost of capturing equivalent target sensor data by the target sensor such that an equivalent physical quality value measured when capturing the target sensor data is equivalent to the first physical quality value.
In addition, in some embodiments, the present invention proposes using the synthetic data generated according to the present invention to test an autonomous system comprising the target sensor. Using synthetic data generated according to the present invention reduces cost of testing an autonomous system comprising the target sensor and increases accuracy of an output of a system for testing the autonomous system compared to testing the autonomous system with the real target sensor data.
There exist other uses of dense depth maps besides producing simulation data for a testing engine. In some embodiments the present invention proposes using the plurality of computed depth maps when detecting, and additionally or alternatively classifying, one or more objects in the plurality of signals. In such embodiments, the present invention proposes providing the plurality of computed depth maps to one or more object classification engines to detect, and additionally or alternatively classify, one or more objects in the plurality of signals. An output of an object classification engine using the plurality of computed depth maps produced according to the present invention is more accurate than another output when using other depth maps of lesser quality. Some object detection systems associate each of the one or more objects with a confidence value, indicative of a degree of confidence the respective object is identified, and additionally or alternatively classified, correctly. The present invention further proposes, in some embodiments thereof, analyzing the one or more objects detected in the plurality of signals using a plurality of ground-level layer maps, each generated for one of the plurality of real signals using the plurality of computed depth maps. Optionally, each of the plurality of ground-level layer maps comprises a ground-level surface as captured in the respective real signal. Analyzing the one or more objects using a plurality of ground-level layer maps, generated using the plurality of computed depth maps, improves accuracy of an output of the object classification engine, optionally increasing a confidence score of at least one of the one or more objects, and additionally or alternatively optionally reducing a confidence score of at least one other of the one or more objects. Optionally, the one or more objects are analyzed additionally using the plurality of computed depth maps, further improving accuracy of an output of the object classification engine. Optionally, when an object of the one or more objects has a modified confidence score less than an identified confidence threshold value, the object may be eliminated from the one or more detected objects. In addition, validating one or more object detection systems according to the present invention allows comparing performance of the one or more object detection systems, and additionally or alternatively allows comparing performance of one or more versions of the object detection system, facilitating accelerating one or more processes of handling faults in an object detection system, thus reducing cost of development and cost of operation of the object detection system. In addition, validating one or more object detection systems according to the present invention allows comparing performance of the one or more object detection systems automatically, and additionally of alternatively comparing performance of one or more versions of the object detection system automatically, further facilitating reducing cost of development and cost of operation of the object detection system.
Another use of dense depth maps is in validation of a system having one or more components for detecting one or more objects in sensor data, for example an autonomous system such as an autonomous driving system or an advanced driver-assistance system (ADAS). To validate a system that detects objects in sensor data there is a need to validate that one or more objects detected in sensor data are detected correctly. Level of detection correctness may be measured using a recall metric value, indicative of a ratio between an amount of correctly detected objects and an amount of objects present in the sensor data, that should have been detected. The amount of objects present in the sensor data is a sum of the amount of correctly detected objects and an amount of falsely undetected objects (known as an amount of “false negative”). There is a need to increase the recall metric value. Additionally, or alternatively, level of detection correctness may be measured using a precision metric value, indicative of a ratio between an amount of correctly detected objects and an amount of detected objects. The amount of detected objects is a sum of the amount of correctly detected objects and an amount of incorrectly detected objects (known as an amount of “false positive”). There is a need to increase the precision metric value. To do so, the present invention in some embodiments thereof proposes analyzing the one or more detected objects using the plurality of ground-level layer maps for the purpose of validating the system. Validating an object using a plurality of ground-level layer maps increases accuracy of an object validation compared to validating without using ground-level layer information. Producing the plurality of ground-level layer maps using the plurality of computed depth maps increases accuracy of the plurality of ground-level layer maps compared to other ground-level layer maps computed without using a plurality of depth maps, and thus increase accuracy of an object validation. Optionally, the system that detects objects is a system for testing an autonomous system. In such embodiments the plurality of ground-level layer maps are used to validate one or more test objects detected by the autonomous system while being tested. Validating the one or more tested objects using the plurality of ground-level layer maps computed using the plurality of computed depth maps increases accuracy of a validation of the autonomous system being tested.
In addition, the present invention in some embodiments thereof proposes analyzing the one or more test objects using a ground truth computed using the plurality of computed depth maps. In such embodiments, the present invention proposes detecting one or more target objects in the plurality of real signals, and computing, using the plurality of computed depth maps, a ground-truth comprising one or more bounding-boxes, each embedding one of the plurality of target objects. Optionally, the ground-truth is computed using a machine learning model, for example a neural network. Optionally, the machine learning model is trained using the plurality of computed depth maps. Validating the one or more test objects using the ground-truth computed using the plurality of computed depth maps increases accuracy of a validation of the autonomous system being tested. For example, an object present in the ground-truth is expected to be detected by the tested autonomous system; when not detected by the tested autonomous system the recall metric value is reduced. Optionally, the one or more test objects are analyzed using the plurality of ground-level layer maps computed using the plurality of computed depth maps and using the ground-truth computed using the plurality of computed depth maps, further increasing accuracy of a validation of the autonomous system.
Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangement of the components and/or methods set forth in the following description and/or illustrated in the drawings and/or the Examples. The invention is capable of other embodiments or of being practiced or carried out in various ways.
The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.
The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
For brevity, the term “processor” is used to mean “at least one hardware processor”. The processor may be any kind of programmable or non-programmable circuitry that is configured to carry out the operations described below.
Reference is now made to
Before describing a possible method for creating synthetic data, reference is now made also to
Optionally, depth maps generator 201 is a machine learning model, for example a neural network. Optionally, a synthetic data generator 204 generates a large synthetic training dataset 214 simulating a large real training dataset captured by the plurality of sensors, and optionally depth maps generator 201 is trained using large synthetic training dataset 214. Optionally, another style transformation 205 is applied to large synthetic training dataset 214 according to small real dataset 220 captured by the target sensor. Optionally, an output of the other style transformation is a large synthetic target specific training dataset 215. Optionally, depth maps generator 201 is trained using large synthetic target specific training dataset 215.
In some embodiments of the present invention, system 100 implements the following optional method to produce synthetic data for testing an autonomous system according to flow of data 200.
Reference is now made also to
In 310, processor 101 optionally applies a point of view transformation to the plurality of real signals and the plurality of computed depth maps, to produce synthetic data, for example large synthetic target dataset 212, simulating a possible signal captured from the common physical scene by target sensor 130 in an identified position 230 relative to the plurality of sensors, and in 340 processor 101 optionally provides the synthetic data to at least one testing engine to test an autonomous system comprising the target sensor. Optionally, the testing engine additionally or alternatively performs one or more operations on the autonomous system, where the one or more operations are selected from a group of: validating the autonomous system, verifying the autonomous system and training the autonomous system. Optionally, the autonomous system is an autonomous driving system. Optionally, the autonomous system is an ADAS. Optionally, the autonomous system is a production process control system. Optionally, in 310 processor 101 applies dynamic transformation 208 to target sensor position 230, to simulate one or more dynamic changes in the target sensor position, for example due to mounting instability and/or shakiness.
Optionally, the synthetic data comprises a plurality of digital images, for example when the plurality of sensors is a plurality of cameras. Optionally, in 360 processor 101 applies a hole-filling algorithm to the plurality of digital images of the synthetic data, to produce filled target synthetic data, for example large synthetic filled target dataset 217, simulating target data captured by target sensor 130, and optionally uses the filled target synthetic data as the synthetic data, for example providing the filled target synthetic data to the at least one testing engine. Optionally, processor 101 applies the hole-filling algorithm by executing at least one prediction engine to predict at least one color of at least one pixel of at least one of the plurality of digital images of the synthetic data. Optionally, the at least one prediction engine is at least one color prediction machine learning model.
Optionally, in 380 processor 101 applies one or more physical characteristic adjustments to the synthetic data to produce adjusted target synthetic data, for example large synthetic adjusted target dataset 216, simulating target data captured by target sensor 130. Optionally, processor 101 uses the filled target synthetic data produced in 360 as the synthetic data in 380.
Reference is now made also to
Reference is now made again to
Optionally, at least one first physical quality value of at least one physical quality, measured while capturing the plurality of real signals by the plurality of sensors, is greater than at least one second physical quality value of the at least one physical quality, measured while capturing the real target sensor data by target sensor 130. The physical quality may be indicative of an amount of use of a sensor when capturing a signal. Some examples of a physical quality are a distance a vehicle traverses while capturing the signal, an amount of time for capturing the signal, an atmospheric pressure while capturing the signal, an environment temperature while capturing the signal, and an amount of light while capturing the signal. Optionally, the at least one first physical quality value comprises a first value of a first distance the plurality of sensors traversed in the common physical scene while capturing the plurality of real signals, and the at least one second physical quality value comprises a second distance value of a second distance target sensor 130 traversed in the other physical scene while capturing the real target sensor data. Optionally, the at least one first physical value is indicative of greater use of the plurality of sensors to capture the plurality of real signals than use of target sensor 130 to capture the real target sensor data. For example, a ratio between the first distance value and the second distance value may be greater than an identified ratio threshold. Optionally, the identified ratio threshold is 10, optionally indicating that a first distance traversed by the plurality of sensors while capturing the plurality of real signals is at least 10 times greater than a second distance traversed by target sensor 130 while capturing the real target sensor data. Optionally, the identified ratio threshold is 100. Optionally the identified ratio threshold is 1000.
Processor 101 optionally applies in 350 another style transformation to the plurality of synthetic training signals produced in 301 and to the real target sensor data to produce a plurality of transformed synthetic training signals, for example large synthetic target specific training dataset 215 and optionally uses the plurality of transformed synthetic training signals in 304 when training the machine learning model. Training the machine learning model in 304 optionally comprises a plurality of training iterations using the plurality of synthetic training signals and additionally or alternatively the plurality of transformed synthetic training signals. Optionally, training the machine learning model in 304 further comprises a plurality of self-supervised training iterations.
Reference is now made also to
In some embodiments of the present invention, system 100 may be used additionally to produce synthetic data for a new target sensor, for example target sensor 140.
Reference is now made again to
Optionally, the new synthetic data comprises a new plurality of digital images, for example when the plurality of sensors is a plurality of cameras. Optionally, in 365 processor 101 applies a new hole-filling algorithm to the new plurality of digital images of the new synthetic data, to produce new filled target synthetic data, simulating target data captured by new target sensor 140, and optionally uses the new filled target synthetic data as the new synthetic data, for example providing the new filled target synthetic data to the at least one testing engine. Optionally, processor 101 applies the new hole-filling algorithm by executing at least one new prediction engine to predict at least one new color of at least one new pixel of at least one of the new plurality of digital images of the new synthetic data. Optionally, the at least one new prediction engine is at least one new color prediction machine learning model.
Optionally, in 385 processor 101 applies one or more new physical characteristic adjustments to the new synthetic data, to produce new adjusted target synthetic data simulating target data captured by new target sensor 140 Optionally, processor 101 uses the new filled target synthetic data produced in 365 as the new synthetic data in 385.
Reference is now made also to
Reference is now made again to
Processor 101 optionally applies in 355 a new other style transformation to the plurality of synthetic training signals produced in 301 and to the new real target sensor data to produce a new plurality of transformed synthetic training signals, and optionally uses the new plurality of transformed synthetic training signals in 304 when training the machine learning model.
In some embodiments of the present invention, system 100 is further used to augment synthetic data with one or more additional objects, for example to augment synthetic data generated using method 300 with one or more additional objects. To augment synthetic data, in some embodiments of the present invention system 100 further implements the following optional method.
Reference is now made also to
Reference is now made also to
To test the autonomous system, in some embodiments system 500 implements the following non-mandatory method.
Reference is now made also to
Optionally, system 600 implements method 300 to produce the synthetic data. Optionally, processor 501 produces a plurality of synthetic training signals, each simulating one of a plurality of signals simultaneously captured from a common training scene by the plurality of sensors, and a plurality of training depth maps each qualifying one of the plurality of synthetic training signals according to the common training scene. Optionally, processor 501 trains a machine learning model based on the plurality of synthetic training signals and the plurality of training depth maps. Optionally, processor 501 uses the machine learning model to compute a plurality of computed depth maps based on the plurality of real signals. Optionally, the plurality of real signals are captured simultaneously from the common physical scene. Optionally, each of the plurality of real signals are captured by one of the plurality of sensors. Optionally, each of the plurality of computed depth maps qualifies one of the plurality of real signals. Optionally, processor 501 applies a point of view transformation to the plurality of real signals and the plurality of computed depth maps, to produce synthetic data simulating the possible signal captured from the common physical scene by the target sensor in an identified position relative to the plurality of sensors.
Optionally, processor 501 receives the synthetic data from system 100, optionally via at least one digital communication network interface 502.
In 610, processor 501 optionally provides the synthetic data to at least one testing engine to test an autonomous system comprising a target sensor and optionally tests the autonomous system using the synthetic data. Optionally, the testing engine additionally or alternatively performs one or more operations on the autonomous system, where the one or more operations are selected from a group of: validating the autonomous system, verifying the autonomous system and training the autonomous system. Optionally the autonomous system is an ADAS. Optionally the autonomous system is an autonomous driving system.
In some embodiments of the present invention, the plurality of computed depth maps created by system 100 implementing flow of operations 300 is used for detecting one or more objects in the plurality of real signals.
Reference is now made also to
To detect one or more objects in the plurality of real signals, in some embodiments system 700 implements the following non-mandatory method.
Reference is now made also to
Optionally, processor 701 stores the plurality of ground-level layer maps generated in 815 and additionally or alternatively the plurality of computed depth maps generated in 807 in at least one non-volatile digital storage 703. Optionally, in 820 processor 701 retrieves the plurality of computed depth maps and additionally or alternatively the plurality of ground-level layer maps from at least one non-volatile digital storage 703. Retrieving the plurality of ground-level layer maps generated in 815 and additionally or alternatively the plurality of computed depth maps generated in 807 from at least one non-volatile digital storage 703 optionally reduces an amount of computation resources required to validate the one or more detected object.
In some embodiments of the present invention, system 700 is used to validate a system for testing an autonomous system. To validate a system for testing an autonomous system, system 700 implements in such embodiments the following optional method. Reference is now made also to
Optionally, processor 701 stores the plurality of ground-level layer maps generated in 915 and additionally or alternatively the plurality of computed depth maps in at least one non-volatile digital storage 703. Optionally, in 923 processor 701 retrieves the plurality of computed depth maps from at least one non-volatile digital storage 703. Optionally, in 930 processor 701 retrieves the plurality of ground-level layer maps from at least one non-volatile digital storage 703. Retrieving the plurality of ground-level layer maps generated in 915 and additionally or alternatively the plurality of computed depth maps from at least one non-volatile digital storage 703 optionally reduces an amount of computation resources required to validate the plurality of detected objects.
Optionally, in 910 processor 701 tests the autonomous system using synthetic data generated for a first target sensor and additional synthetic data generated for a second target sensor. Optionally, the additional synthetic data is generated for the second target sensor using method 300. Optionally, the plurality of detected objects identified in the synthetic data by the autonomous system while testing the autonomous system are identified by the autonomous system using one or more sensor fusion methods. Optionally the first target sensor is a digital camera. Optionally the second target sensor is a LIDAR. Optionally, in 923 processor 701 uses one or more sensor fusion methods to detect the one or more target objects in the synthetic data and the additional synthetic data.
It should be appreciated that all the methods described above for computing a plurality of depth maps, for applying a point of view transformations, and for computing a plurality of ground-level layer maps, may be applied any of a plurality of types of target sensors, including but not limited to an RGB camera, sensitive to red, green and blue, a gray-scale camera, a fish-eye camera, a Red/Clear (RCCC) camera, and an RGRB camera. It should also be appreciated that all the methods described above for a plurality of depth maps, for applying a point of view transformations, and for computing a plurality of ground-level layer maps, may use a plurality of cameras that create a 360-degree view surrounding a vehicle, for creating depth maps and point of view transformations over 360-degree view surrounding the vehicle.
The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
It is expected that during the life of a patent maturing from this application many relevant point of view transformations and style transformations will be developed and the scope of the terms “point of view transformation” and “style transformation” are intended to include all such new technologies a priori.
As used herein the term “about” refers to ±10%.
The terms “comprises”, “comprising”, “includes”, “including”, “having” and their conjugates mean “including but not limited to”. This term encompasses the terms “consisting of” and “consisting essentially of”.
The phrase “consisting essentially of” means that the composition or method may include additional ingredients and/or steps, but only if the additional ingredients and/or steps do not materially alter the basic and novel characteristics of the claimed composition or method.
As used herein, the singular form “a”, “an” and “the” include plural references unless the context clearly dictates otherwise. For example, the term “a compound” or “at least one compound” may include a plurality of compounds, including mixtures thereof.
The word “exemplary” is used herein to mean “serving as an example, instance or illustration”. Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and/or to exclude the incorporation of features from other embodiments.
The word “optionally” is used herein to mean “is provided in some embodiments and not provided in other embodiments”. Any particular embodiment of the invention may include a plurality of “optional” features unless such features conflict.
Throughout this application, various embodiments of this invention may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.
Whenever a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integral) within the indicated range. The phrases “ranging/ranges between” a first indicate number and a second indicate number and “ranging/ranges from” a first indicate number “to” a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numerals therebetween.
It is appreciated that certain features of the invention, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment of the invention. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.
All publications, patents and patent applications mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual publication, patent or patent application was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention. To the extent that section headings are used, they should not be construed as necessarily limiting. In addition, any priority document(s) of this application is/are hereby incorporated herein by reference in its/their entirety.
This application is a Continuation-in-Part (CIP) of U.S. patent application Ser. No. 16/237,806 filed on Jan. 2, 2019, the contents of which are incorporated by reference as if fully set forth herein in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9507346 | Levinson | Nov 2016 | B1 |
| 9606539 | Kentley | Mar 2017 | B1 |
| 9916703 | Levinson | Mar 2018 | B2 |
| 10019005 | Wang et al. | Jul 2018 | B2 |
| 10019011 | Green et al. | Jul 2018 | B1 |
| 10031526 | Li et al. | Jul 2018 | B1 |
| 10395117 | Zhang | Aug 2019 | B1 |
| 10580158 | Mousavian | Mar 2020 | B1 |
| 10816993 | Tran | Oct 2020 | B1 |
| 10832418 | Karasev | Nov 2020 | B1 |
| 20070088469 | Schmiedel et al. | Apr 2007 | A1 |
| 20140309836 | Ollis | Oct 2014 | A1 |
| 20160137206 | Chandraker et al. | May 2016 | A1 |
| 20160314224 | Wei | Oct 2016 | A1 |
| 20170123428 | Levinson et al. | May 2017 | A1 |
| 20170124476 | Levinson | May 2017 | A1 |
| 20170243083 | Wang | Aug 2017 | A1 |
| 20180032863 | Graepel et al. | Feb 2018 | A1 |
| 20180045519 | Ghadiok et al. | Feb 2018 | A1 |
| 20180259976 | Williams | Sep 2018 | A1 |
| 20180275658 | Iandola et al. | Sep 2018 | A1 |
| 20180275667 | Liu et al. | Sep 2018 | A1 |
| 20180315158 | Nurvitadhi | Nov 2018 | A1 |
| 20180322387 | Sridharan | Nov 2018 | A1 |
| 20190039625 | Stefan et al. | Feb 2019 | A1 |
| 20190147245 | Qi | May 2019 | A1 |
| 20190235521 | Mudalige | Aug 2019 | A1 |
| 20200041261 | Bernstein | Feb 2020 | A1 |
| 20200193630 | Mousavian | Jun 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| WO 2018080533 | May 2018 | WO |
| Entry |
|---|
| Notice Of Allowance dated Jul. 1, 2019 From the U.S. Appl. No. 16/237,806. (7 pages). |
| Official Action dated Apr. 12, 2019 From the U.S. Appl. No. 16/237,806. (8 pages). |
| European Search Report and the European Search Opinion dated Apr. 22, 2020 From the European Patent Office Re. Application No. 20150081.6. (9 Pages). |
| Ros et al. “The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes”, 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, XP033021505, Las Vegas, NV, USA, Jun. 27-30, 2016, p. 3234-3243, Jun. 27, 2016. |
| Number | Date | Country | |
|---|---|---|---|
| 20200210779 A1 | Jul 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16237806 | Jan 2019 | US |
| Child | 16594200 | US |
