Patent Application
20170109458
The present disclosure generally relates to vehicular systems and, more particularly, to a testbed for providing a testing and verification environment for developing, training and proving algorithms for detecting lane boundaries in a driving environment.
Generally speaking, it is imperative to have well proven algorithms for interpreting sensor data in order to provide, enable, or otherwise support functionalities such as driver assistance, controlling vehicle dynamics, and/or autonomous driving. In particular, algorithms for detecting boundaries of driving lanes are vital. Presently, real-world sensor data and ground-true information are relied upon for developing, training, testing and proving such algorithms. However, it is expensive, in terms of time, money and resources, to acquire a useful amount of real-world data and ground-true information.
Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various figures unless otherwise specified.
In the following description, reference is made to the accompanying drawings that form a part thereof, and in which is shown by way of illustrating specific exemplary embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the concepts disclosed herein, and it is to be understood that modifications to the various disclosed embodiments may be made, and other embodiments may be utilized, without departing from the scope of the present disclosure. The following detailed description is, therefore, not to be taken in a limiting sense.
In the development of lane boundary detection algorithms to detect lane boundaries with a variety of markings or without markings, diverse sets of sensor data are needed to train, develop, test and prove the lane boundary detection algorithms and additional downstream functions associated with the algorithms. However, it usually takes considerable money, time and resources to acquire real-world sensor data. For example, to obtain real-word sensor data, sensors need to be physically mounted on a vehicle, and actual driving runs need to be performed on various types of road and for various traffic conditions for the sensors to collect sensor data for each scenario. In addition, ambient variables such as weather, temperature, wind, light conditions and other weather-related factors tend to increase the number of sets of sensor data to be collected by multiple orders of magnitude. In general, sensor data for thousands of miles of road are needed to develop a lane boundary detection algorithm, and thus considerable amount of time, money and resources are required to acquire such data.
The scenarios are further complicated, and the sensor data to be collected further expanded, when different types of vehicles (such as different makes/models) with different kinds of sensors equipped at different locations of the vehicles are taken into consideration. Moreover, characteristics of a particular vehicle, as well as characteristics of the various sensors mounted thereon, may change or drift over time due to factors such as aging or accumulation of odometer mileage, among others. In other words, a lane boundary detection algorithm that is developed and well-trained for a new vehicle with low mileage may not be as effective or accurate after the vehicle is put to service for a few years.
Furthermore, not all types of sensors produce data in mutually compatible formats. Consequently, significant effort is needed for post-processing the massive amount of real-world sensor data after acquisition into a format that can be used by a lane boundary detection algorithm. In addition, in developing algorithms that have embedded supervised learning functions built in, it is needed to supply such an algorithm with actual lane boundary information, referred to as “ground truth information” herein, so that the algorithm can compare its own lane boundary designation with true or known-to-be-correct lane boundary locations, thereby improving the algorithm's detection ability by way of machine learning. This need of obtaining the ground truth information further exacerbates the burden of using real-world data for developing lane boundary detection algorithms.
The present disclosure provides a solution to overcome the difficulties, and reduce the cost, of acquiring sensor data needed for a lane boundary detection algorithm by generating the sensor data utilizing a virtual environment. The solution also enables generation of ground truth information with relative ease for algorithm self-training. In acquiring sensor data, various scenarios, conditions and parameters as mentioned above can be easily set up virtually in the virtual environment, and a virtual vehicle equipped with virtual sensors can traverse or drive about the virtual environment in a simulation and virtually collect (i.e., generate by simulation) the sensor data needed for the lane boundary detection algorithm. The virtual environment, virtual vehicle and virtual sensors are modeled to closely match the corresponding real-world environment, real-world vehicle and real-world sensors such that the simulation-generated virtual sensor data can substantially represent the same information collected by the real-world sensor in the real-world environment. Ground-true lane boundary information is readily available as it is defined as part of the virtual environment.
In some embodiments, road surface 105 may be provided with one or more road structures such as traffic dividers dividing a driving lane from another driving lane. In some embodiments, one or more roadside structures such as road shoulders, side rails or curbs 150 may be provided alongside the road surface 105 in a direction parallel with the automobile traffic, on either side or both sides of road surface 105. In some embodiments, objects such as traffic signs 160, commercial signs or billboards may also be provided alongside the road surface 105 in a direction parallel with the automobile traffic. Traffic signs or other signs may also be provided above road surface 105 where the automobile traffic is not affected.
In some environments, objects such as bushes, trees or other plants 170, and other structures such as lampposts, power posts, utility poles or buildings may be provided alongside the road surface 105 in a direction parallel with the automobile traffic.
A virtual vehicle with one or more virtual sensors mounted thereon may be utilized in environment 100 for generation of sensor data. For example, a vehicle 190 as illustrated in
The approach taken by the present disclosure to realize a low-cost and efficient solution for acquisition of sensor data is by substantially transforming from real world to a virtual space items and objects illustrated in
In some embodiments, lane boundary detection testbed 200 may include computer hardware and computer software. The computer hardware of lane boundary detection testbed 200 may include one or more processors 202, memory 290, a user interface 204, other hardware 206 such as a field-programmable gate array (FPGA) or a graphics processing unit (GPU), or the like or a combination or sub-combination thereof. Memory 290 may be operably connected to or otherwise accessible by the one or more processors 202, and may be configured to store the computer software for execution by the one or more processors 202.
In some embodiments, the one or more processors 202 may execute a lane boundary detection algorithm 270 to generate algorithm output 280. The lane boundary detection algorithm 270 may enable the one or more processors 202 to determine a “most probable” or “most likely” location where a lane boundary may be by receiving and analyzing sensor data 250 produced by virtual sensor models 220 modeling one or more real-world sensors of interest.
User interface 204 may enable a user, e.g., an engineer, technician or the like, to interact with, run, customize, or control various aspects of lane boundary detection testbed 200. In some embodiments, user interface 204 may include one or more keypads, keyboards, touch screens, pointing devices, or the like or a combination or sub-combination thereof.
In some embodiments, memory 290 may store data, codes and/or instructions pertaining to or otherwise defining one or more virtual driving environments 210. The one or more virtual driving environments 210 may contain various virtual objects, structures and markings as shown in
In some embodiments, virtual driving environment 210 may include a three dimensional mesh defining in a virtual space locations, orientations, sizes, shapes, colors, surface reflectiveness and other characteristics of some or all of the stationary objects, structures, markings and lines present in environment 100 as illustrated in
In some embodiments, each sensor model 220 may be a software model that defines or predicts for certain situations or views the output of a corresponding real-world sensor. In certain embodiments, each sensor model 220 may be provided with information (e.g., data from a virtual driving environment 210) characterizing various views of a road surface, e.g., road surface 105. With this information, each sensor model 220 may predict what an actual sensor presented with those views in the real world would output.
In some embodiments, real-world sensors of interest may include transducers that sense or detect some characteristic of an environment and provide a corresponding output (e.g., an electrical or optical signal or an image) that defines that characteristic. For example, one or more real-world sensors of interest may be accelerometers that output an electrical signal characteristic of the proper acceleration being experienced thereby. Such accelerometers may be used to determine the orientation, acceleration, velocity, and/or distance traveled by a vehicle. Other real-world sensors of interest may include cameras, laser scanners, light-detection-and-ranging (LIDAR) scanners, ultrasonic transducers, radar devices, gyroscopes, inertial measurement units, revolution counters or sensors, strain gauges, temperature sensors, or the like.
Each sensor model 220 may be used to model the output produced by a real-world sensor of interest. For example, sensor model 220 may be used to model sensors 191, 192 and 193 mounted on vehicle 190, as illustrated in
Each sensor model 220 may produce an output of any suitable format. For example, in some embodiments, a sensor model 220 may output an analog signal that a corresponding real-world sensor would produce. Alternatively, a sensor model 220 may output a processed signal, such as a digitized and filtered version of an analog signal. For example, a sensor model 220 may output a processed signal such as one output by a data acquisition system. Accordingly, in some embodiments, the output of a sensor model 220 may be a conditioned, digital version of the signal that a corresponding real-world sensor would produce.
Each of the one or more vehicle models 230 is configured to model a respective sensor-mounted vehicle that traverses on a road surface in a driving environment, such as vehicle 190 of
In general, a vehicle model 230 may include two sub-models: a vehicle-stationary model 232 and a vehicle-dynamic model 234. With the two sub-models, the traversal of a vehicle within virtual driving environment 210 may be modeled to a reasonably accurate degree. Vehicle-stationary model 232 may be a software model that defines certain stationary characteristics of a corresponding type of vehicle. In some embodiments, a set of parameters may be used to record dimensions of the corresponding type of vehicle. The set of parameters may also include information regarding planned locations of one or more sensors mounted on the corresponding type of vehicle. Vehicle-dynamic model 234 may be a software model that defines certain dynamic characteristics of a corresponding type of vehicle in response to external forces or impacts. In some embodiments, vehicle-dynamic model 234 may include characteristics of chassis and/or suspension dynamics of a corresponding type of vehicle to certain fidelity.
In some embodiments, vehicle-dynamic model 234 may be provided with one or more driver inputs (e.g., one or more values characterizing parameters such as velocity, drive torque, brake actuation, steering input, or the like or combinations or sub-combinations thereof) and information (e.g., data from a virtual driving environment 210) characterizing a road surface. With these inputs and information, vehicle-dynamic model 234 may predict motion states of the body of a corresponding type of vehicle.
The parameters of vehicle-dynamic model 234 may be determined or specified in any suitable manner. In some embodiments, certain parameters of vehicle-dynamic model 234 may be derived from previous knowledge of the mechanical properties (e.g., geometries, inertia, stiffness, damping coefficients, etc.) of a corresponding real-world vehicle. The parameters may be different for different types of vehicles.
Simulation module 240 may be programmed to cause the one or more processors 202 to take a virtual driving environment 210, one or more sensor models 220 and a vehicle model 230 as inputs and subsequently generate an output modeling a real-world output produced by one or more corresponding real-world sensors mounted on a corresponding real-world vehicle (e.g., the vehicle modeled by the vehicle model 230) traversing a real-world driving environment modeled by (e.g., substantially or exactly matching) the virtual driving environment 210. In some embodiments, at least a portion of the output generated by simulation module 240 may be stored in memory 290 as sensor data 250. As previously stated, driving environment 100 may include one or more lane markings such as solid line 110, broken line 120 and surface reflectors 130. In this case, sensor data 250 may include data that directly characterize locations of lane boundaries. Alternatively, in some embodiments, driving environment 100 may not include any or some of the lane markings. In this case, sensor data 250 may include data that characterize other virtual objects defined in virtual driving environment 210, either stationary or mobile, such as text 140, road shoulders or side rails 150, signs 160, trees or other plants 170, as well as other vehicles 181, 182 and 183. Though indirect, data of these virtual objects stored as sensor data 250 may still be utilized by the one or more processors 202 to infer locations of lane boundaries through execution of the lane boundary detection algorithm 270.
In some embodiments, lane boundary detection testbed 200 may include in simulation module 240 one or more bias modules 242. A purpose of a bias module 242 is to account for secondary effects such as weather conditions, time of day, sensor aging and vehicle aging. In real-world scenarios, a sensor may not maintain same characteristics or performance over an indefinitely long period of time. More likely than not, real-world sensors may experience certain aging effects and their characteristics may change or drift over time. Similar situation may happen for real-world vehicles. A bias module 242 included in simulation module 240 may be programmed to account for such effects due to weather conditions, time of day, sensor aging and vehicle aging by causing the one or more processors 202 to adjust either the output generated by simulation module 240 or the stored sensor data 250 according to a set of bias parameters. In some embodiments, a bias module 242 may be programmed to account for various weather conditions. In some embodiments, a bias module 242 may be programmed to account for lighting conditions that may change from dawn to dust depending on different times of the day. In some embodiments, a bias module 242 may cause the one or more processors 202 to adjust or “bias” one or more sensor models 220 to account for such secondary effects.
An example implementation of bias module 242 may be further illustrated by the following examples. In some embodiments, a sensor model 220 may be a virtual camera that models a real-world visual camera, and thus the corresponding sensor data 250 may be one or more visual images. If the virtual weather condition is raining, the image perceived by the virtual camera would become blurred due to rain and may be disturbed by motion of windshield wipers, as compared to a clear image otherwise perceived under a normal weather condition. As another example, the image perceived under a bright sunlight may have less contrast and thus having a “white out” effect as compared to a clear image under normal daylight condition. These visual effects on the images (e.g. sensor data 250) generated by simulation module 240 may be produced by one or more bias modules 242. In short, the one or more bias modules 242 not only account for various secondary effects, but also facilitate generating a large amount of sensor data 250 under various conditions in an efficient and low-cost way.
In the example shown in
In some embodiments, simulation module 240 may include ground truth annotation module 244 programmed to cause the one or more processors 202 to couple sensor data 250 with one or more annotations of a first kind. For example, annotation 350a and annotation 350b shown in
For example, one or more annotations, including annotations 350a and 350b, may provide “true locations” of the boundaries of lane 115 within which virtual vehicle 190 is driven for a particular run. The true locations of the lane boundaries are defined according to the spatial definition of the lane 115 within the virtual driving environment 100. Annotations 350a and 350b may be linked, tied to, overlaid upon, or otherwise associated with particular portions of data streams 391, 392 and 393. Accordingly, the ground truth information corresponding to a particular lane 115 may be linked to the portion of data streams 391, 392 and 393 that reflect the perception of virtual sensors 191, 192 and 193 of that boundaries of lane 115. In some embodiments, not all of data streams 391, 392 and 393 may have ground truth information annotated at same temporal portions thereof.
In some embodiments, simulation module 240 may include lane boundary annotation module 246 programmed to cause processor 202 to couple sensor data 250 with one or more annotations of a second kind. For example, annotation 350a and annotation 350b shown in
At 410, example process 400 may involve processor 202 setting up virtual driving environment 210 according to various virtual objects (e.g. as text 140, road shoulders or side rails 150, signs 160, trees or other plants 170, as well as other vehicles 181, 182 and 183), lane markings (e.g. solid line 110, broken line 120 and surface reflectors 130) and virtual sensors (e.g. sensors 191, 192 and 193) shown in
At 420, example process 400 may involve processor 202 determining whether a traversal has reached a predetermined destination. For instance, after generating virtual driving environment 210, processor 202 may determine whether or not a traversal has reached a predetermined destination. On one hand, if it is determined that the traversal has not yet reached the predetermined destination, processor 202 may determine to traverse virtual sensors to a next location within the virtual driving environment, and example process 400 may proceed to 430. On the other hand, if it is determined that the traversal has reached the predetermined destination, processor 202 may determine to proceed to 470.
At 430, in response to a determination that the traversal has not yet reached the predetermined destination, example process 400 may involve traversing the virtual sensors to a next location using a vehicle-dynamic model 234 of a vehicle model 230. Block 430 may be followed by block 440.
At 470, in response to a determination that the traversal has reached the predetermined destination, example process 400 may involve analyzing (by processor 202 executing lane boundary detection algorithm 270) sensor data 250 and subsequently determining (also by processor 202 executing lane boundary detection algorithm 270) lane boundary locations in virtual driving environment 210. At 470, example process 400 may also involve annotating sensor data 250 with the detected lane boundary. Example process 400 may end after 470.
At 440, example process 400 may involve processor 202 recording sensor data 250 generated by simulation module 240 characterizing virtual driving environment 210 as perceived by a virtual sensor modeled by a sensor model 220. Block 440 may be followed by block 450.
At 450, example process 400 may involve ground truth annotation module 244 of simulation module 240 annotating sensor data 250 with ground truth information such as locations of lane boundaries as defined in virtual driving environment 210 at various portions in time of the sensor data 250. Block 450 may be followed by block 460.
At 460, example process 400 may involve processor 202 determining, in response to recording annotated data characterizing virtual driving environment 210 as perceived by a virtual sensor, whether the virtual sensor is the last among the virtual sensors sensing the virtual driving environment 210 whose perceptions need to be recorded. On one hand, if it is determined that the virtual sensor is not the last among the virtual sensors sensing the virtual driving environment 210 whose perceptions need to be recorded, processor 202 may proceed to 440 to record annotated data characterizing virtual driving environment 210 as perceived by a next virtual sensor. On the other hand, if it is determined that the virtual sensor is already the last among the virtual sensors sensing the virtual driving environment 210 whose perceptions need to be recorded, processor 202 may proceed to 420 to check again if the traversal has reached the predetermined destination.
At 510, example process 500 may involve one or more processors 202 generating a virtual driving environment 100 that includes one or more driving lanes 105, a virtual vehicle 190, and virtual sensors 191, 192 and 193. Virtual sensors 191, 192 and 193 are mounted on virtual vehicle 190.
At 520, example process 500 may involve one or more processors 202 traversing virtual sensors 191, 192 and 193 as mounted on virtual vehicle 190 within virtual driving environment 100 to generate simulated sensor data 250. Block 520 may be followed by block 530.
At 530, example process 500 may involve one or more processors 202 executing one or more lane boundary detection algorithms 270 to process simulated sensor data 250 to detect one or more driving lanes 105, designating locations of boundaries of one or more driving lanes 105 as algorithm output 280. Block 530 may be followed by block 540.
At 540, example process 500 may involve one or more processors 202 recording algorithm output 280. Block 540 may be followed by block 550.
At 550, example process 500 may involve one or more processors 202 annotating, through lane boundary annotation module 246, the simulated sensor data 250 with algorithm output 280. Block 550 may be followed by block 560.
At 560, example process 500 may involve one or more processors 202 generating ground truth information for the at least one of the one or more driving lanes 105. The ground truth information may represent one or more actual locations of the one or more boundaries of the at least one of the one or more driving lanes 105 within the virtual driving environment 100. Additionally, example process 500 may involve one or more processors 202 recording a difference between the ground truth information and the output of the algorithm 280. Alternatively or additionally, example process 500 may involve one or more processors 202 annotating the simulated sensor data 250 with the ground truth information. Block 560 may be followed by block 570.
At 570, example process 500 may involve one or more processors 202 annotating, through ground truth annotation module 244, the simulated sensor data 250 with ground truth information of one or more driving lanes 105.
In some embodiments, the virtual driving environment 100 may also include a plurality of lane markings corresponding to the one or more driving lanes 105, such as line 110, a line 120, surface reflectors 130, and markings or text 140. In some embodiments, the virtual driving environment 100 may also include a plurality of virtual objects, either stationary or mobile relative to the virtual driving environment, such as road shoulders or side rails 150, signs 160, trees or other plants 170, and other vehicles 181, 182 and 183. Each of the lane markings and the virtual objects may be sensible by the one or more virtual sensors, such as sensors 191, 192 and 193 mounted on virtual vehicle 190. In some embodiments, the simulated sensor data 250 may characterize the virtual driving environment 100 as perceived by one or more virtual sensors 191, 192 and 193 sensing the plurality of lane markings and virtual objects.
In some embodiments, virtual sensors 191, 192 and 193 may include a virtual camera, and simulated sensor data 250 may include one or more virtual images of the virtual driving environment 100 as perceived by the virtual camera. In some embodiments, example process 500 may also involve displaying on the one or more virtual images a plurality of overlaid markings. The plurality of overlaid markings may indicate one or more locations of one or more boundaries of at least one of the one or more driving lanes 105.
In some embodiments, algorithm output 280 may include one or more locations of one or more boundaries of at least one of the one or more driving lanes designated by lane boundary detection algorithm 270 as “more probable” locations of boundaries of one or more driving lanes 105. In some embodiments, the one or more locations of the one or more boundaries of the at least one of the one or more driving lanes may include a plurality of points each with a respective spatial coordinate within the virtual driving environment 100. The plurality of points may collectively correspond to the one or more locations of the one or more boundaries of the at least one of the one or more driving lanes. Additionally, example process 500 may also involve annotating the simulated sensor data 250 with the spatial coordinates of the plurality of points.
In some embodiments, in recording algorithm output 280, example process 500 may involve recording a timestamp of the output of the algorithm 280.
In some embodiments, virtual sensors 191, 192 and 193 may be mounted on virtual vehicle 190 according to vehicle-stationary model 232 modeling a location of virtual sensors 191, 192 and 193 with respect to virtual vehicle 190. In some embodiments, virtual vehicle 190 may traverse within virtual environment 100 according to vehicle-dynamic model 234 modeling motions of virtual vehicle 190.
The articles “a” and “an” are used herein to refer to one or to more than one (i.e., to at least one) of the grammatical object of the article. By way of example, “a user” means one user or more than one users. Reference throughout this specification to “one embodiment,” “an embodiment,” “one example,” or “an example” means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” “one example,” or “an example” in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, databases, or characteristics may be combined in any suitable combinations and/or sub-combinations in one or more embodiments or examples. In addition, it should be appreciated that the figures provided herewith are for explanation purposes to persons ordinarily skilled in the art and that the drawings are not necessarily drawn to scale.
Embodiments in accordance with the present disclosure may be embodied as an apparatus, method, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware-comprised embodiment, an entirely software-comprised embodiment (including firmware, resident software, micro-code or the like), or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module,” or “system.” Furthermore, embodiments of the present disclosure may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.
The flow diagrams and block diagrams in the attached figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It will also be noted that each block of the block diagrams and/or flow diagrams, and combinations of blocks in the block diagrams and/or flow diagrams, may be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flow diagram and/or block diagram block or blocks.
Although the present disclosure is described in terms of certain embodiments, other embodiments will be apparent to those of ordinary skill in the art, given the benefit of this disclosure, including embodiments that do not provide all of the benefits and features set forth herein, which are also within the scope of this disclosure. It is to be understood that other embodiments may be utilized, without departing from the scope of the present disclosure.
