Patent Application
20190068943
The present disclosure relates to an environment perception system for an autonomous vehicle.
This section provides background information related to the present disclosure, which is not necessarily prior art.
An autonomous vehicle typically includes an environment perception system that gathers data regarding the surrounding environment, and inputs that data to an autonomous drive system, which drives the vehicle. Thus, it is imperative that the environment perception system accurately and quickly perceive the environment about the vehicle. While existing autonomous vehicle environment perception systems are suitable for their intended use, they are subject to improvement. The present teachings include an autonomous vehicle environment perception system that provides numerous advantages over existing systems, as explained herein and as one skilled in the art will recognize.
This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.
The present teachings provide for an environment perception system for perceiving an environment about a vehicle having an autonomous drive system. The environment perception system includes a first camera that captures first images of the environment. A first camera controller generates first image data based on first images captured by the first camera. A second camera captures second images of the environment. A second camera controller generates second image data based on second images captured by the second camera. A ranging scanner measures distances between the vehicle and objects within the environment. A ranging controller generates ranging data based on the distances measured by the ranging scanner. A fusion controller generates three-dimensional data representative of the environment based on the first image data, the second image data, and the ranging data.
Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.
The drawings described herein are for illustrative purposes only of select embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.
Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.
Example embodiments will now be described more fully with reference to the accompanying drawings.
The environment perception system 10 includes a first camera 20 and a second camera 22. The first camera 20 and the second camera 22 can be mounted at any suitable position about the vehicle 12. In the example of
The first and second cameras 20 and 22 can be multi-pose cameras controlled by two separate electric motor-based actuators. Steering angle of the vehicle 12, that which is available from controller area network (CAN) messages, can be used to supply information to the actuators to synchronize the positions of the first and second cameras 20 and 22 with the steering wheel of the vehicle 12. By synchronizing the angles of the first and second cameras 20 and 22 with the angle of the steering wheel of the vehicle 12, the first and second cameras 20 and 22 can closely track the steering wheel position to acquire a wide perspective of the environment.
The environment perception system 10 further includes a ranging scanner 24 that measures distances between the subject vehicle 12 and other objects within the scene in the environment about the subject vehicle 12. Any suitable ranging scanner can be used, such as a light detection and ranging (LIDAR) scanner. The ranging scanner 24 can be arranged at any suitable position about the subject vehicle 12, such as mounted to the roof thereof between the first camera 20 and the second camera 22 as illustrated in
The environment perception system 10 further includes a first camera controller 30 and a second camera controller 32. In this application, including the definitions below, the term “controller” may be replaced with the term “circuit.” The term “controller” may refer to, be part of, or include processor hardware (shared, dedicated, or group) that executes code, and memory hardware (shared, dedicated, or group) that stores code executed by the processor hardware. The code is configured to provide the features of the controllers described herein. The term memory hardware is a subset of the term computer-readable medium. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium is therefore considered tangible and non-transitory. Non-limiting examples of a non-transitory computer-readable medium are nonvolatile memory devices (such as a flash memory device, an erasable programmable read-only memory device, or a mask read-only memory device), volatile memory devices (such as a static random access memory device or a dynamic random access memory device), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc). The controllers described herein may also include a field-programmable gate array (FPGA).
The first camera controller 30 receives images (first images) captured by the first camera, and generates first image data based on the images captured on the first camera 20. Similarly, the second camera controller 32 receives images (second images) captured by the second camera 22 and generates second image data based on the images captured by the second camera 22. The first image data generated by the first camera controller 30 and the second image data generated by the second camera controller 32 can be in any form suitable for identifying and tracking objects in a scene of an environment about the subject vehicle 12. For example, the first image data and the second image data can be first point cloud data and second point cloud data respectively. The first and second point cloud data can include any suitable data of the images captured (and the objects captured in the images) such as, but not limited to, the following: X, Y, Z coordinate data; red, green, blue (RGB) color data; and/or intensity data (I) representing a depth of the image captured.
The system 10 further includes a third camera controller 34. The third camera controller 34 receives the first images captured by the first camera 20 and the second images captured by the second camera 22. Based on the first and second images combined, the third camera controller 34 generates third image data, such as third point cloud data. The third point cloud data can include any suitable data of the first and second images, such as, but not limited to, the following: X, Y, Z coordinate data; and/or intensity data (I) representing depth of the captured images.
The environment perception system 10 further includes a ranging controller 40. The ranging controller 40 generates ranging data based on the distances measured by the ranging scanner 24. The ranging controller 40 can be a standalone controller, or can be integrated with the ranging scanner 24. The ranging data can include ranging point cloud data of the scene being captured in the environment about the subject vehicle 12.
The environment perception system 10 further includes a fusion controller 50. Each one of the controllers 30, 32, 34, 40, and/or 50 can be any suitable controller, such as a field-programmable gate array (FPGA). The fusion controller 50 generates three-dimensional data representative of the environment about the subject vehicle 12, such as of a particular scene within the environment. The three-dimensional data is based on, for example, the first image data generated by the first camera controller 30 (such as first point cloud data), the second image data generated by the second camera controller 32 (such as second point cloud data), the third image data generated by the third camera controller 34 (such as the third point cloud data), and the ranging data generated by the ranging controller 40 (such as ranging point cloud data). The three-dimensional data generated by the fusion controller 50 includes one or more of the following: X, Y, Z coordinate data; red, green, blue color data; and intensity data representing image depth.
The three-dimensional data generated by the fusion controller 50 is input to a scene perception controller 60. The first images captured by the first camera 20 and the second images captured by the second camera 22 are also input directly to the scene perception controller 60. Based on these inputs, the scene perception controller 60 generates scene perception data. The scene perception data provides information of the environment about the subject vehicle 12, and specifically of particular scenes of the environment captured by the system 10, for use by the autonomous drive system 70 for vehicle path planning and decision making.
For example, the scene perception data can identify a gap in traffic that the subject vehicle 12 may pass through. Based on this scene perception data, the autonomous drive system 70 can safely and successfully drive the subject vehicle 12 through the gap in traffic. Similarly, the scene perception data can indicate to the autonomous drive system 70 that a lane adjacent to the subject vehicle 12 is clear for the vehicle 12 to use for passing. The autonomous drive system 70 can then steer the subject vehicle 12 to the passing lane in order to pass slower traffic (or avoid an obstacle). The scene perception data can also include data identifying traffic signs or traffic lights. If the scene perception data includes a stop sign or a red traffic light, for example, upon receipt of the data the autonomous drive system 70 will stop or otherwise operate the subject vehicle 12 so as to be in compliance with the stop sign or traffic signal (or any other type of traffic regulation).
Positioning the first camera 20 and the second camera 22 on opposite sides of the vehicle 12, as illustrated in
An additional advantage of the environment perception system 10 includes enhanced depth perception (generation of three-dimensional data) of particular scenes captured in the environment of the subject vehicle 12 due to the use of both first and second cameras 20 and 22, and the ranging scanner 24. By combining data from the first camera 20, the second camera 22, and the ranging scanner 24 using the fusion controller 50 and the scene perception controller 60, the system 10 according to the present teachings provides improved depth perception as compared to systems that employ only cameras or ranging scanners.
Additional advantages of the environment perception system 10 include use of the dedicated first camera controller 30 to process the first images captured by the first camera 20, and the dedicated second camera controller 32 to process the second images captured by the second camera 22. By using the separate first and second camera controllers 30 and 32, as well as the separate third camera controller 34 and ranging controller 40, the system 10 advantageously makes use of distributed computation (or processing) to prevent system overload.
Using both the first camera 20 and the second camera 22 also advantageously allows for use of data from one of the first and second cameras 20 and 22 to safely guide the vehicle 12 (such as to a road shoulder) in the event that one of the first camera 20 and the second camera 22 fails.
The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.
Example embodiments are provided so that this disclosure will be thorough, and will fully convey the scope to those who are skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.
The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” may be intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises,” “comprising,” “including,” and “having,” are inclusive and therefore specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order discussed or illustrated, unless specifically identified as an order of performance. It is also to be understood that additional or alternative steps may be employed.
When an element or layer is referred to as being “on,” “engaged to,” “connected to,” or “coupled to” another element or layer, it may be directly on, engaged, connected or coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being “directly on,” “directly engaged to,” “directly connected to,” or “directly coupled to” another element or layer, there may be no intervening elements or layers present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between” versus “directly between,” “adjacent” versus “directly adjacent,” etc.). As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.
Although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms may be only used to distinguish one element, component, region, layer or section from another region, layer or section. Terms such as “first,” “second,” and other numerical terms when used herein do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.
Spatially relative terms, such as “inner,” “outer,” “beneath,” “below,” “lower,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. Spatially relative terms may be intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, the example term “below” can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
