Patent Application
20190154823
As is known in the art, radar sensors are increasingly being used within automobiles and other vehicles to provide information to drivers about the presence of people and vehicles in a vicinity of the automobiles. Radar sensors may be programmed to perform functions such as blind spot detection (BSD), lane change assist (LCA), cross traffic alert (CTA), rear detection, and others to enhance safety and driver awareness on the road.
Known methods for rear pedestrian detection (RPD) require a moving pedestrian in order to be able to detect and classify an object as pedestrian. These known methods do not work for detecting pedestrians which are not moving. This is caused by the low and diffuse radar cross section (RCS) and due to the fact that a static object does not have a velocity distribution. It is very challenging to distinguish between infrastructure and a static pedestrian due to co-range and co-Doppler distortion. There are a lot of scenarios in which it is very important to detect pedestrians which are not moving. This applies for example for parking maneuvers.
In non-RPD scenarios detections are desired in a further range (e.g., greater than twenty meters) and with a generally fast moving target. There is also typically good target to infrastructure Doppler separation. Uncertainty in these scenarios is usually related to noise and phase curve effects which hinder assigning azimuth correctly to these detections.
In RPD scenarios, the challenges are different. The vehicle is typically moving very slowly, is at close range (less than or equal to 15 meters, though farther is possible), is detecting static or very slow moving targets (below 10 kph). Uncertainty is affected by co-range, co-doppler and scintillation resulting from specular and multifaceted reflectors (parked cars, large plate target in field of view, etc.). These effects are rarely observed simultaneously by multiple sensors. For example, if there is a vehicle behind you, the left sensor will detect it in one location (rear bumper) while the right sensor might detect the fender of the same vehicle. The detection coordinates will be different but for the same target.
RPD also suffers from multipath effects. These effects are dependent on the layout of the scenario and occur more often for complicated scenarios. In an environment where there is a single person in an open field behind you, this is not much of an issue however for more complicated issues such as multiple targets in a crowded parking lot, multipath effects become an issue. Fading, which is due to multiple wavefronts received at slightly different times due to ground bounce, has the effect of distorting the magnitude, range and azimuth information of target. This increases the 2D spatial uncertainty factor for any given detection. This effect is influenced by the physical height of the sensor above the ground. Due to all the above described factors, a technique is needed which can resolve static or slow moving RPD targets.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Note that each of the different features, techniques, configurations, etc. discussed in this disclosure can be executed independently or in combination. Accordingly, embodiments of the concepts described herein can be embodied and viewed in many different ways. For additional details, elements, and/or possible perspectives (permutations) of the concepts described herein, the reader is directed to the Detailed Description section and corresponding figures of the present disclosure as further discussed below.
In accordance with the concepts, systems and techniques described herein, it has been appreciated that existing techniques for detecting rear objects (including, but not limited to pedestrians) provide poor performance. Accordingly, there is a need for an improved rear object detection technique which is capable of detecting objects (including, but not limited to pedestrians) having a low and/or diffuse radar cross section (RCS).
It has been found that the concepts, systems, devices and techniques described herein may be used in both a rear-looking object detection system (i.e. to detect one or more objects behind a rear of a vehicle when a vehicle is still or is backing up) as well as in a forward-looking object detection system (i.e. to detect one or more objects in front of a vehicle when a vehicle is moving forward). Vehicles may include either or both of such forward and/or rear object detection systems. Furthermore, such forward and/or rear object detection systems may find use in autonomous driving systems.
In accordance with a first aspect of the concepts, systems, devices and methods described herein, an object detection method for detecting objects (including, but not limited to pedestrians) in a front or rear of a vehicle includes acquiring a plurality of detections for an area around at least a portion of a vehicle from at least one radar sensor. The method further includes integrating at least some of the plurality of detections to generate at least one image mask. Additionally, the method requires applying the at least one image mask to a host-compensated image of the area around at least a portion of the vehicle to determine a presence of static or substantially static objects within the image. The objects may, for example, be pedestrians. The objects may be static (i.e. non-moving) or substantially static (slowly moving relative to the speed of the vehicle in which the front and/or rear object detection system is disposed. The concepts, systems, devices and methods described herein may also be applied to pedestrian detection or object detection at low speed in general.
Such a technique is useful for detecting objects (including, but not limited to pedestrians) having a low and/or diffuse radar cross section (RCS).
In embodiments, prior to the integrating at least some of the plurality of detections to generate at least one image mask, at least some detections from the sensors may be combined when there are more than one sensor. In embodiments, the mask may be a two-dimensional mask. In embodiments the two dimensions may comprise range and azimuth. In embodiments the detections may be weighted by the image mask prior to updating the host-compensated image.
In embodiments the method may further include calculating a confidence factor from a range confidence factor and an azimuth confidence factor. In some embodiments a range confidence factor may be calculated from a number of counts comprising a distance between a peak value and an average value divided by a distance between smoothed data cross over points with average in a range histogram. In embodiments an azimuth confidence factor may be calculated from a number of counts comprising a distance between a peak value and an average value divided by range, the range determined in accordance with the formula: a number of degrees between smoothed data cross over points with average multiplied by (Pi divided by 180) multiplied by a propagated range in an azimuth histogram. In embodiments detections that may be associated with multiple different sensors may be weighted higher.
Other arrangements of embodiments of the concepts described herein that are disclosed herein may include software programs to perform methods and operations summarized above and disclosed in detail below. More particularly, a computer program product is one embodiment may have a computer-readable medium including computer program logic encoded thereon that when performed in a computerized device provides associated operations providing pedestrian detection as explained herein. The computer program logic, when executed on at least one processor with a computing system, causes the processor to perform the operations (e.g., the methods) indicated herein as embodiments of the broad concepts described. Such arrangements/embodiments are typically provided as software, code and/or other data structures arranged or encoded on a computer readable medium such as an optical medium (e.g., CD-ROM), floppy or hard disk or other a medium such as firmware or microcode in one or more ROM or RAM or PROM chips or as an Application Specific Integrated Circuit (ASIC) or as downloadable software images in one or more modules, shared libraries, etc. The software or firmware or other such configurations may be installed onto a computerized device to cause one or more processors in the computerized device to perform the techniques explained herein as embodiments of the described concepts. Software processes that operate in a collection of computerized devices, such as in a group of data communications devices or other entities can also provide the system which implements the described concepts. The system(s), device(s) and technique(s) described herein can be distributed between many software processes on several data communications devices, or all processes could run on a small set of dedicated computers, or on one computer alone to implement the described concepts.
Details relating to this and other embodiments are described more fully herein.
Objects, aspects, features, and advantages of embodiments disclosed herein will become more fully apparent from the following detailed description, the appended claims, and the accompanying drawings in which like reference numerals identify similar or identical elements. Reference numerals that are introduced in the specification in association with a drawing figure may be repeated in one or more subsequent figures without additional description in the specification in order to provide context for other features. For clarity, not every element may be labeled in every figure. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments, principles, and concepts. The drawings are not meant to limit the scope of the claims included herewith.
While a system using two sensors are shown and described, it should be appreciated that the system could be used with only a single sensor. Further, while object/pedestrian detection at a rear of a vehicle is described, it should be appreciated that the same concepts equally apply to detection of an object/pedestrian at a front of a vehicle. Such a sensing system disposed for operation at a front of a vehicle (e.g. in a vehicle moving in a forward direction), may be useful in autonomous driving applications and for object/pedestrian detection at low speed in general.
A first field of view 106 is shown for sensor 104a and a second field of view 108 is shown for sensor 104b. There is an area of overlap 110 for the two fields of view 106 and 108. There is also an area 112 between the fields of view of sensors 104a and 14b. In this particular embodiment, each field of view 106 and 108 have approximately a 150 degree area, with 22 degrees of each field of view overlapping.
Referring to
In one particular embodiment, Rear Sensor Fusion may be used. This technique improves image resolution for diffuse scatterers defined as energy scattered over a wide range of angles (humans) and having a diminishing effect at higher range. This technique provides little to no improvement for specular scatterers, defined as energy scattered over a very narrow range of angles (large flat plates).
At least one embodiment, the object/pedestrian detection technique described herein uses what is referred to as Type 1 Integration. Type 1 integration uses a group of range-azimuth histograms produced from detections to create image masks of the area behind the vehicle. The masks provide a two-dimensional (2D) probability distribution within an image map. The masks may be applied to a host-compensated image of the area behind the vehicle to determine a presence of static objects within the image.
The host compensated image, is for determining static objects only. Moving objects are determined from the statistics of the masks themselves. A mask associated with a group having very low spatial uncertainty (high confidence), will get promoted to an object even if little or no energy corresponds to the objects location in the host compensated image. This is because the object is moving. Only static objects will appear “bright” in the host compensated image. Objects that spawn from the host-compensated image can be classified as static versus. groups with very mature statistics but do not have significant image energy are considered non-static. This distinguishable classification will impact the time-to-collision estimation with the host.
In at least one embodiment, the technique may further incorporate Type 2 Integration which involves image processing Synthetic Aperture Radar (SAR) techniques. This improves the signal-to-Noise (SNR) for static targets within the map. Pixels containing detections from static objects will accumulate the fastest, while pixels having detections from dynamic targets appear smeared as energy will be spread out across more adjacent pixels. The type 1 integration assists the type 2 integration by providing spatial weighting to the 2D map.
Current data is shown in FFT 402, which collects range and doppler information from the one or more sensors. FFT 402 provides this data to DETC 404 which performs detection processing. A local maxima is determined from the 2D Range-Doppler ITT. Thresholds are based on noise estimates in the vicinity of detections.
Block 406 shows data for a detection. The data includes range, doppler and azimuth information for each detection. For each sensor, detections are collected according to range to create a group. For each detection, if the range and Doppler associated with that detection are within a range window and doppler window of an active group, that detection is added to the group, otherwise a new group is provided.
Detections from other sensors, shown in box 410 are also processed. Each sensor gets the data from the other sensor. For example, the left sensor may receive the right sensor data and the right sensor may receive the left sensor. Integration from multiple sensors provides the advantage of observing the scene from multiple aspects.
Box 408 is used for synchronizing the time stamps of detections of one sensor with the detections of the other sensor. This is referred to herein as fusing the detections together. The link groups box 408 combines the data from box 406 with the data from box 410 according to range. Box 408 also performs type 1 integration and feature extraction.
Box 412 utilizes the data from box 408 to take the two linear features of range and azimuth and histogramming them over time. This also results in forming statistics about range uncertainty and azimuth. Histogram masks can be used to weight detections from same sensor.
The current detections are weighted by the integrated statistics of the same sensor. Cross correlation is provided by weighting the current detections from one sensor by the integrated statistics from another sensor, and vice-versa. Associated detections from multiple sensors provides an advantage to diffuse scatterers (e.g., humans), seen at the same physical location by each sensor, and also suppresses sporadic detections, which are rarely observed simultaneously by multiple sensors at the same location. The contribution to the integrated statistics is based on the amount of association (number of associated detections from other sensors).
For the group range and azimuth data histograms, the peak bin represents the maximum likelihood of the group's range and azimuth. The spread of energy across bins represents the uncertainty factor in range and azimuth, which can lead to a confidence metric in two dimensions. The vector direct product (uv) of the two histograms creates a two dimension weighting function, or mask, containing the combined effect of the group's range and azimuth statistics.
In box 414 vector multiplication is performed to create from two one dimensional vectors a two dimensional mask in range and azimuth for that particular group. All the masks for the group are built and in box 416 they are emerged to form an image mask. This box results in an overall mask of a probability distribution of where any given detection would be coming in the future should reside. If this is not the case, then the detection is not for real target.
In box 418 all current detections from the current sensor (box 406) and all detections form other sensors (box 410) are weighted by the image mask (box 416), which provides spatial weighting for every detection before placing detections into an image (box 420). From the integrated image (box 420) the detection of objects can be determined (box 422).
In box 424 the integrated image is propagated over time based on the host dynamic (box 426). The image mask provides the statistical probability for predicting the location of any future detection. Scaling detections by the image mask provides the proper spatial weighting in that detections that are consistent with the integrated statistics of an active group are amplified, while detections that do not coincide with an active group in range and azimuth are suppressed. Host motion compensation requires rotation and translation of the full map referenced to a common system origin.
A weighting factor based on how many detections from the other sensors can be associated with this detection are used. Sensitivity to an object seen simultaneously by multiple sensors at the same range and azimuth is improved while objects such as road irregularities and parked cars are not as correlated as often. This provides an advantage to a human target if co-range with strong reflectors such as parked cars and infrastructure.
Referring now to
In this example, the right vehicle data 508 is more pronounced than the left vehicle data 510 since the right sensor detected the right vehicle but not the left vehicle, while the left sensor detected the right vehicle and the left vehicle.
Referring to
Referring now to
Graph 720 shows the left sensor range data for group 2 (ground). The range histogram 722 shows a small peak at about 10.5 meters, a larger peak at about 11.2 meters, and another peak at 11.4 meters. The smoothed data 724 shows a curve having a small peak at about 10.5 meters and a larger peak area at about 11.3 meters. This indicates the presence of a ground clutter at that distance from the source vehicle. Graph 730 shows the left sensor azimuth data for group 1 (left vehicle). The azimuth histogram 732 shows a peak at about −11 degrees. The smoothed data 734 shows a curve having a peak at about −10 degrees. This indicates the presence of ground clutter at that angle from the source vehicle.
Graph 740 shows the left sensor range data for group 3 (mannequin). The range histogram 742 shows a plateau from about 6 meters to 6.2 meters, a larger peak at about 6.4 meters, another peak at 6.6 meters and another peak at 6.8 meters. The smoothed data 744 shows a curve having a small peak at about 6.6 meters and extending from 6 meters to 7 meters. This indicates the presence of a mannequin at that distance from the source vehicle. Graph 750 shows the left sensor azimuth data for group 3 (mannequin). The azimuth histogram 752 shows several smaller peaks at −27 degrees, −22 degrees, −17 degrees, −6 degrees, 12 and 20 degrees. Also shown are several larger peaks occurring at −14 degrees, −10 degrees, 0 degrees, 3 degrees and 7 degrees. The smoothed data 754 shows a curve having a larger peak at about −10 degrees and another peak at 5 degrees. This indicates the presence of the mannequin behind the source vehicle.
Graph 800 shown in
In
Brange=(Bdegrees*π/180)*propagated_range
The azimuth CF is then calculated by taking the value of Acounts and dividing it by the value of Brange. The total CF can then be determined by multiplying the range CF by the azimuth CF. The spatial uncertainty is proportional to 1/CF.
Referring to
The presence of the pedestrian/mannequin is shown by object 1006 and the uncertainty boxes 1016 and 1020 surrounding object 1002. Ground object 1008 is shown with uncertainty box 1018.
Referring now to
Further, the processes and operations described herein can be performed by a computer especially configured for the desired purpose or by a general-purpose computer especially configured for the desired purpose by another computer program stored in a computer readable storage medium or in memory.
Referring to
Processing block 1306 discloses integrating at least some of the plurality of detections to generate at least one image mask. As shown in processing block 1308, the mask may be a two-dimensional mask. As further shown in processing block 1310 the two dimensions comprises range and azimuth.
Processing block 1312 shows applying the at least one image mask to a host-compensated image of the area surrounding at least a portion of the vehicle to determine a presence of static objects within the image. Processing block 1314 shows the detections are weighted by the image mask.
Processing continues with processing block 1316 which depicts calculating a confidence factor from a range confidence factor and an azimuth confidence factor. As shown in processing block 1318, range confidence factor is calculated from a number of counts comprising a distance between a peak value and an average value divided by a distance between smoothed data cross over points with average in a range histogram. As shown in processing block 1320, the azimuth confidence factor is calculated from a number of counts comprising a distance between a peak value and an average value divided by range, the range determined in accordance with the formula:
a number of degrees between smoothed data cross over points with average multiplied by (Pi divided by 180) multiplied by a propagated range in an azimuth histogram.
Processing block 1322 shows wherein detections that are associated with multiple different sensors are weighted higher.
As shown in
The processes of
Processor 1402 may be implemented by one or more programmable processors executing one or more computer programs to perform the functions of the system. As used herein, the term “processor” describes an electronic circuit that performs a function, an operation, or a sequence of operations. The function, operation, or sequence of operations may be hard coded into the electronic circuit or soft coded by way of instructions held in a memory device. A processor may perform the function, operation, or sequence of operations using digital values or using analog signals. In some embodiments, the processor can be embodied in one or more application specific integrated circuits (ASICs). In some embodiments, the processor may be embodied in one or more microprocessors with associated program memory. In some embodiments, the processor may be embodied in one or more discrete electronic circuits. The processor may be analog, digital, or mixed-signal. In some embodiments, the processor may be one or more physical processors or one or more “virtual” (e.g., remotely located or “cloud”) processors.
Various functions of circuit elements may also be implemented as processing blocks in a software program. Such software may be employed in, for example, one or more digital signal processors, microcontrollers, or general-purpose computers. Described embodiments may be implemented in hardware, a combination of hardware and software, software, or software in execution by one or more physical or virtual processors.
Some embodiments may be implemented in the form of methods and apparatuses for practicing those methods. Described embodiments may also be implemented in the form of program code, for example, stored in a storage medium, loaded into and/or executed by a machine, or transmitted over some transmission medium or carrier, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation. A non-transitory machine-readable medium may include but is not limited to tangible media, such as magnetic recording media including hard drives, floppy diskettes, and magnetic tape media, optical recording media including compact discs (CDs) and digital versatile discs (DVDs), solid state memory such as flash memory, hybrid magnetic and solid state memory, non-volatile memory, volatile memory, and so forth, but does not include a transitory signal per se. When embodied in a non-transitory machine-readable medium and the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the method.
When implemented on one or more processing devices, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits. Such processing devices may include, for example, a general purpose microprocessor, a digital signal processor (DSP), a reduced instruction set computer (RISC), a complex instruction set computer (CISC), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic array (PLA), a microcontroller, an embedded controller, a multi-core processor, and/or others, including combinations of one or more of the above. Described embodiments may also be implemented in the form of a bitstream or other sequence of signal values electrically or optically transmitted through a medium, stored magnetic-field variations in a magnetic recording medium, etc., generated using a method and/or an apparatus as recited in the claims.
For example, when the program code is loaded into and executed by a machine, such as the computer of
In some embodiments, the storage medium may be a physical or logical device. In some embodiments, a storage medium may include physical or logical devices. In some embodiments, a storage medium may be mapped across multiple physical and/or logical devices. In some embodiments, storage medium may exist in a virtualized environment. In some embodiments, a processor may be a virtual or physical embodiment. In some embodiments, a logic may be executed across one or more physical or virtual processors.
For purposes of illustrating the present embodiment, the disclosed embodiments are described as embodied in a specific configuration and using special logical arrangements, but one skilled in the art will appreciate that the device is not limited to the specific configuration but rather only by the claims included with this specification.
Various elements, which are described in the context of a single embodiment, may also be provided separately or in any suitable subcombination. It will be further understood that various changes in the details, materials, and arrangements of the parts that have been described and illustrated herein may be made by those skilled in the art without departing from the scope of the following claims.
