Patent Grant
12066567
The present disclosure is related to automotive detection systems such as automotive radar systems and, in particular, to an apparatus and method for detecting and correcting for blockage of an automotive radar sensor.
In automotive radar systems, it is desirable to detect when the radar sensor is blocked by debris, such as dirt, snow, ice, etc. Sensor blockage or radar blockage attenuates the transmitted and received signal such that objects in the field of view are no longer detectable. It is also important to alert the driver when the sensor is blocked so that the driver does not rely on the radar system while a sensor is blocked, and so that the driver can intervene and clear the debris from the sensor to restore performance of the system.
Declaring a sensor blockage based on the absence of radar signal processing detections is a relatively straightforward means of determining sensor blockage with minimal additional processing time or resources. One drawback of this approach is that it is difficult to distinguish the blocked case from the case in which there are relatively few or no objects large enough to create detections in the field of view of a sensor that is not blocked and is functioning properly. This situation can occur, for example, when the automobile in which the system is operating is passing through a desert or along a bridge or causeway surrounded by water.
According to one aspect, a radar sensor is provided. The radar sensor includes a memory storing a model defining a relationship between a condition of the radar sensor and a plurality of features of radar detections, the model being generated by a machine learning approach in which, during a training operation, a plurality of training radar detections are received under known states of the condition of the radar sensor, the model storing values of the plurality of features associated with the known states of the condition of the radar sensor. The radar sensor further includes a radar detector for transmitting radar signals into a region, detecting reflected returning radar signals from the region, and converting the reflected returning radar signals into digital data signals. A processor receives the digital data signals and processes the digital data signals to generate actual radar detections, each of the actual radar detections being characterized by a plurality of the features of radar detections. The processor applies values of the features of the actual radar detections to the model to determine the state of the condition of the radar sensor from the values of the features of the actual radar detections.
In some exemplary embodiments, the radar system is an automotive radar system.
In some exemplary embodiments, the condition is blockage of the radar sensor. The state of the condition can be is blocked, partially blocked or unblocked.
In some exemplary embodiments, the machine learning approach comprises at least one of a neural network approach, a logistic regression approach and/or a bagged trees approach.
In some exemplary embodiments, the model identifies a subset of features associated with the training radar detections which are useful in determining the state of the condition of the radar sensor; and the processor applies the identified features of the actual radar detections to the model to determine the state of the condition of the radar sensor. The subset of features can be selected using analysis of histograms of features associated with the training radar detections. The subset of features can be selected using a bagged trees analysis of features associated with the training radar detections.
According to another aspect, a method for detecting a condition in a radar sensor is provided. The method includes: storing in a memory a model defining a relationship between the condition of the radar sensor and a plurality of features of radar detections, the model being generated by a machine learning approach in which, during a training operation, a plurality of training radar detections are received under known states of the condition of the radar sensor, the model storing values of the plurality of features associated with the known states of the condition of the radar sensor; transmitting radar signals into a region; detecting reflected returning radar signals from the region; converting the reflected returning radar signals into digital data signals; receiving the digital data signals with a processor; and processing the digital data signals with the processor to generate actual radar detections, each of the actual radar detections being characterized by a plurality of the features of radar detections, the processor applying values of the features of the actual radar detections to the model to determine the state of the condition of the radar sensor from the values of the features of the actual radar detections.
In some exemplary embodiments, the radar system is an automotive radar system.
In some exemplary embodiments, the condition is blockage of the radar sensor. The state of the condition can be is blocked, partially blocked or unblocked.
In some exemplary embodiments, the machine learning approach comprises at least one of a neural network approach, a logistic regression approach and/or a bagged trees approach.
In some exemplary embodiments, the model identifies a subset of features associated with the training radar detections which are useful in determining the state of the condition of the radar sensor; and the processor applies the identified features of the actual radar detections to the model to determine the state of the condition of the radar sensor. The subset of features can be selected using analysis of histograms of features associated with the training radar detections. The subset of features can be selected using a bagged trees analysis of features associated with the training radar detections.
The present disclosure is further described in the detailed description which follows, in reference to the noted plurality of drawings by way of non-limiting examples of embodiments of the present disclosure, in which like reference numerals represent similar parts throughout the several views of the drawings.
Radar sensor module 12A also receives returning radar signals at radar receive circuitry 22A via receive antenna 18A. Radar receive circuitry 22A generally includes any circuitry required to process the signals received via receive antenna 18A, such as pulse shaping/timing circuitry, receive trigger circuitry, RF switch circuitry, or any other appropriate receive circuitry used by the radar system. The received radar signals are processed by radar receive circuitry 22A to generate processed receive signals, which are forwarded to a mixer 28A, which mixes the processed receive signals with an RF signal from RF signal generator 24A. The resulting difference signals may be further filtered as required by filtering circuitry 32A to generate baseband signals, which are digitized by analog-to-digital converter circuitry (ADC) 34A to generate receive signals. These digitized baseband receive signals are processed by a processor, such as a digital signal processor (DSP) 36A, to generate target object detections related to objects in the region being monitored by detection system 10A. In some exemplary embodiments, the DSP 36A can perform any and/or all of the processing tasks required to implement the sensor blockage detection described herein according to the exemplary embodiments.
Radar sensor module 12B also receives returning radar signals at radar receive circuitry 22B via receive antenna 18B. Radar receive circuitry 22B generally includes any circuitry required to process the signals received via receive antenna 18B, such as pulse shaping circuitry, receive trigger circuitry, RF switch circuitry, or any other appropriate receive circuitry used by the radar system. In some exemplary embodiments, the received signals processed by radar receive circuitry 22B are forwarded to phase shifter circuitry 26B, which generates two signals having a predetermined phase difference. These two signals, referred to as an inphase (I) signal and a quadrature (Q) signal, are mixed with an RF signal from RF signal generator 24B by mixers 28B and 30B, respectively. The resulting difference signals are further filtered as required by filtering circuitry 32B to generate baseband I and Q signals, labeled “I” and “Q” in
In some exemplary embodiments, processor 36B can perform processing such as a fast Fourier Transform (FFT) to generate a plurality of Doppler range bins, which include range, bearing and velocity information for detections during multiple sweeps, i.e., frequency-modulated (FM) chirps, of the radar illumination signal transmitted into the region around the vehicle being analyzed. In some exemplary embodiments, radar sweeps are performed at a rate of approximately 12 Hz. It will be understood that other sweep rates can be used.
Referring to
It will be understood that, according to the present disclosure, detection system 10 can have many configurations, each including different numbers and locations of sensor modules 12. For example, detection system 10 can include one or more forward-looking sensor modules 12, one or more rear-looking sensor modules 12, and/or one or more side-looking sensor modules 12. Data gathered by sensor modules 12 can be processed by one or more processors, e.g., ECUs, to carry out the various functions implemented by detection system 10. These functions can include, but are not limited to, at least one or any combination of any subset of: a blind spot detection feature, a rear cross traffic detection feature, a lane change assistance feature, a trailer detection feature, a safe door opening feature an adaptive cruise control function, an autonomous braking function, and the sensor blockage determination described herein in detail. Also, the one or more processors, e.g., ECUs, can perform any and/or all of the processing tasks required to implement the sensor blockage detection described herein according to the exemplary embodiments.
According to the present disclosure, blockage of a radar sensor module is determined based on analysis of particular features of object detections which are determined to be closely correlated with the presence and/or absence of sensor blockage. A feature can be derived from detection data, track data, or other sensor parameters. A sensor could have hundreds of defined features. Most features are not related to sensor blockage, and, therefore, their presence or absence, or their nominal value, associated with a detection is of no help in determining whether the sensor may be blocked. Additionally, even though a feature may be relatively correlated with sensor blockage and may, therefore, have some value in a blockage determination, due to limitations on processing resources and/or processing time constraints, it is desirable to limit the number of features used in a blockage determination to only the most useful features, i.e., the features most closely correlated with sensor blockage.
By way of exemplary illustration, one of the features associated with object detections is sensor temperature. According to the present disclosure, it has been shown that temperature of the sensor is uncorrelated with sensor blockage. That is, it has been shown that there is little or no relationship between the temperature of the sensor and the blockage condition of the sensor. Stated more simply, it has been shown that temperature of the sensor has little to nothing to do with whether the sensor is blocked. Accordingly, sensor temperature is not a feature used in making the blockage determination of the present disclosure. On the other hand, number of detections has been found to be a very useful feature associated with sensor blockage. This logically follows since, if the number of detections were to start decreasing or drop to zero, under certain circumstances, that could be very closely correlated with a blockage situation. Accordingly, number of detections is a feature that is considered useful in generating a blockage determination.
As described above, the radar system generates a large number of output features, most of which are not relevant to the blockage issue, such as, for example, sensor temperature. In contrast, “useful” or “important” features are correlated with the blockage state. Processing of all features could be overly taxing of the processing and memory capabilities of the sensor. It may also be too time consuming to be useful. Therefore, according to the disclosure, the most useful features are selected in step S104 of
Regarding the first-stage histogram process, truth data is gathered under multiple sensor known blockage states for each feature to generate a histogram for each feature. The truth data is acquired in various environments when the radar sensor is truly clear or intentionally blocked. For each feature, the histogram associated with the feature is then used to evaluate the correlation of the feature with the blockage state. In generating the histogram for a feature from the truth data, multiple values for the feature can be generated for each of the predetermined blockage states. On one particular embodiment, three blockage states are used, and multiple feature values for each blockage state are generated and used to develop the histogram. In one embodiment, the three states are a “no block” or “clear” state, a 50% blocked state, and an 80% blocked state.
In some exemplary embodiments, to compile the truth data, ten hours of drive data were collected using a with an automotive radar system, such as that manufactured and sold by Applicant. Microwave foam was used to block the radar, the selected foam having two-way attenuation of 30 dB (radar range reduced to 42% of full range) or 15 dB (radar range reduced to 18% of full range). The truth data had three possibilities: unblocked, partially blocked (15 dB attenuation) and blocked (30 dB attenuation). Long drives were broken up into 15-second windows, so that all detections acquired during 15 seconds were combined into a structure representing one data point. The data were split into test and training datasets randomly, with 60% of the data used for training and 40% used for testing. The data were split so that data from any single drive was all put into the test or train datasets, instead of being divided between test and train datasets. This was done because data points from a single drive could be highly correlated, so that the test and training sets would not be independent observations.
According to the disclosure, each histogram is used to represent the distribution of feature data. In some exemplary embodiments, the total area of a histogram used for probability density is normalized to 1. In generating each histogram, the entire range of possible values for the associated feature is divided into a series of equal intervals or “bins”. During the truth data gathering, the number of values that fall into each bin is counted. In the illustrative exemplary embodiments, the histograms for three cases, i.e., no block, 50% block and 80% block, are plotted for each feature variable. The histograms are then evaluated to determine the usefulness of the associated feature in the blockage determination. This evaluation can be performed by viewing the histograms and manually concluding usefulness. Alternatively, the evaluation can be performed mathematically, such as by a user or by a processor.
One observation related to usefulness and the histogram process is that the more spread the three density functions are over the full array of bins for feature values, the more useful the feature.
After the first-stage histogram process will follow the second-stage bagged-tree machine learning process. Under the bagged trees machine learning process, several comparatively smaller trees (models) are trained, instead of a complicated full decision tree. In training, random back sampling is used from the whole train sets to train a decision tree. Under random back sampling, sampled data is put back into the training data pool and could be re-sampled. In subsequent testing, data not selected in training is referred to as “out-of-bag” (OOB) data. OOB data is used to calculate the model error. In using the bagged trees process in connection with feature importance, the calculated error using OOB data for each tree in the testing stage is referred to as: e1, e2, e3, . . . , eN, where N is the number of trees (models). For each tree, random noise is added to each feature of its OOB data, to then obtain the model error: s1, s2, . . . , sN. The importance factor of each feature is then computed as:
One premise of this computation is that the more important a feature is, the more severe performance loss is when the feature is disregarded.
Referring to
Referring to
Referring to
According to one particular exemplary embodiment, a total of 31 potential features are identified by application of the analysis of the histograms for over 150 features. According to some exemplary embodiments, after this first-stage feature selection using histograms generated from truth data is performed, the set of “potentially selected” features, e.g., 31 features, then undergoes second-stage feature selection, using a machine learning process, such as, for example, the well-known bagged trees process. The bagged trees process is used to further down-select the features in consideration of computational cost, and possibly other factors. The bagged trees process is used to calculate each feature's importance in the blockage determination and, therefore, is helpful in down-selecting the features.
Referring to
Hence, as a result of this feature selection process, in these particular illustrative exemplary embodiments, the four most useful features for the blockage determination are selected. In one particular embodiment, the four feature variables are named as Filtered Energy, Average Clutter Range, Maximum Clutter Range, and Number of Detections. In summary, 31 potential features are identified as a result of the feature histogram analysis, and the bagged trees machine learning process is used to calculate each feature's importance in the blockage determination. In the illustrative exemplary embodiment described herein, four features are selected based on their behavior in both the histogram analysis and the bagged trees analysis. It should be noted that the feature selection can also be adjusted by users inserting their own judgment to determine features which do or do not belong in the final process. For example, features related to angular sensor alignment, such as AzimCorrCurve_deg, mentioned above, may not be included because they rely on blockage, and circular logic could result.
According to the present disclosure, using the feature selection process described above with histogram and bagged trees, multiple features are selected for use in the sensor blockage determination, based on their correlation with sensor blockage. In some exemplary embodiments, four features are selected, the four features being number of detections, total detection energy, maximum detection range and average detection range. Because these four features are so closely correlated with sensor blockage, according to the disclosure, during operation of the radar system, these four features are monitored, and their status or values are used to generate a blockage or non-blockage determination.
According to the present disclosure, a subset of the total number of features is selected/computed from detection data generated by the radar system. This subset of features is determined to be the features most closely correlated to sensor blockage and, therefore, the most useful features to be used in the blockage determination. As noted above, in some exemplary embodiments, this subset of features includes four features, namely, TotalDetections, FilteredEnergy, MaxClutterRange, and AvgClutterRange. Each detection has an associated magnitude, or power. By summing the power of all detections over one cycle, FilteredEnergy, or total detection energy, is obtained. Each detection has a range associated with it. The maximum range (MaxClutterRange) is the maximum range among all detections, in a single cycle. The average range (AvgClutterRange) is the average range of all detections, in a single cycle. In an alternative embodiment, the maximum range and average range could be computed over a time window, for example, of a duration of approximately 15 seconds.
After the features are identified as described above, the features are combined based on a machine learning approach, which is used to train a machine learning architecture to learn the relationship between the selected features and the blockage state. Any of several known machine learning architectures can be used, such as, for example, neural network, recurrent neural network, logistic regression, decision tree, bagged trees (random forest), and other such approaches. Under logistic regression, the machine learning architecture determines a method for combining the features into a logistic function. Essentially, this gives each feature a weight and an offset. By looking at the weight of each feature, its importance to the function can be observed. For example, if all features have equal weights, it can be observed that they are all equally important.
In summary of the approach of the disclosure using feature selection, according to these exemplary embodiments, all features derived from detections, tracks, and other sensor outputs, i.e., approximately 150 features, are plotted on histograms based on truth data, i.e., known sensor blockage state, to identify their importance to a sensor blockage determination. Based on this histogram analysis, a subset of features, e.g., 31 features, are identified as being potential features for final selection in the final classification process. Next, using machine learning, such as, for example, a Bagged Trees approach, in possible combination with human engineering judgment to include or exclude features, the final selected features, e.g., four features, are identified for used during subsequent classification of sensor blockage condition based on actual detections. As described above, a large set of data is acquired, under various blockage conditions, so that the “truth” data is known. Part of this data is used as training data, to determine the classifier parameters. The rest of the data can be used as test data, to verify the classifier performance. Different classifiers that were considered have different architectures. They all have the commonality that they can be specified by equations, and by a set of parameters, which are determined by the training described above.
Another machine learning technique used according to the present disclosure is tree bagging or “bagged trees.” To describe tree bagging, first we review the decision tree. For example, we consider if blockage will be predicated on two normalized features, A and B, which could be, for exemplary illustration purposes, number of detections and maximum detection range. The first decision in the tree could be: Is A less than or greater than some threshold. If A is greater, go right on the tree, otherwise go left. Then the next decision could be, is B less than or greater than another threshold. Given a set of features, A and B, we can construct a decision tree. Decision trees can be computed by using the concept of information gain, such that for each node, the feature that can provide the most information is chosen as the decision variable. Using multiple decision trees has been found to give a better result than using any single decision tree. In tree bagging, each tree will be generated with a subset of the training data. Then the decision from many trees can be computed by using a voting algorithm over all of the trees.
According to the present disclosure, the Logistic Regression machine learning approach used for the blockage determination according to some exemplary embodiments is described in detail. The input to the algorithm is a set of independent variables [x1, x2, . . . , xn] where n is arbitrary. In some particular exemplary embodiments, for the blockage algorithm, four independent variables are used. It will be understood that other numbers of independent variables can be used.
Optionally, not necessarily, each variable is normalized so that it ranges from 0 to 1. Consider the variable MaxClutterRange, which may range from 0 to 170 m, if the maximum sensor range is 170 m. It can be normalized by dividing by 170. Next, consider the variable AvgClutterRange, which typically ranges from 0 to 50 m. It can be normalized by dividing by 50. It is possible that AvgClutterRange may exceed 50 m, and in this case, the normalized variable can be set to 1 (it should not exceed 1). In some exemplary embodiments, the normalization parameters will be set manually by examining the variable ranges, and the parameters can be set so that they do not affect the output result. This normalization is not critical to the computation.
Table 1 lists the four selected feature variable names, their definitions, their normalization factors and their variable names used in the following computations. As shown in Table 1, the four selected input variables are renamed as the four variables [x1, x2, . . . , xn], where each variable shall vary between 0 and 1. Table 2 shows typical values for the blocked and unblocked (clear) case. The normalization and renaming of variables can be considered pre-processing steps.
The equation for logistic model is given in (1).
On the right side of (1), the four variables x1, . . . x4 are the inputs to the model. The β parameters β0, . . . β4. are what the machine learning will optimize. The probability P(x) is the probability that the sensor is blocked. If P(x)=1, then the sensor is definitely CLEAR. If P(x)=0, then the sensor is BLOCKED. If P(x)=0.5, a definite determination is not made.
In a more compact form,
It can be observed that if β1=1 and x1 is large, then P(x)=0 because u will be large. This would correspond to the case where there are many detections and the sensor is clear. But if x1=0, we can see that u will be small and then the probability could approach 1, indicating that there are few detections, and the sensor is blocked.
For further illustration, actual realistic numbers can be plugged into the algorithm. Several hours of drive data in the CLEAR and BLOCKED conditions, in different environs, were acquired. This gave us a table like Table 2, with tens of thousands of entries. We used 50% of the data for training and 50% for testing. Then the computer can find the optimal parameters [β0, β1, . . . β5] such that the estimated CLEAR/BLOCK values will most closely match the true CLEAR/BLOCK values.
From our fit, we find that
Because all x values are normalized, the magnitude of the coefficient may indicate the importance of the corresponding parameter. The coefficients indicate that the MaxClutterRange variable was less important than other variables. This is consistent with our initial finding that MaxClutterRange was not an important variable.
Initially, we will interpret P>0.5 as that the sensor is CLEAR and P<0.5 as that the sensor is BLOCKED We can apply (2) and (3) using the coefficients of (4). For the first row of Table 2, with [x1, x2, . . . , xn]=[4E-5,0.0017,0.0076,0.303]. Then we compute a blockage probability P=0.0032, indicating that the sensor is BLOCKED. Next we try the second row of Table 2, with [x1, x2, . . . , xn]=[0.24,0.37,0.54,0.71]. In this case the model gives P=1 indicating that the sensor is CLEAR. These results both gave the correct answer.
The model can be applied to the test dataset.
According to the present disclosure, a subset of the total number of features is selected/computed from detection data generated by the radar system. This subset of features is determined to be the features most closely correlated to sensor blockage and, therefore, the most useful features to be used in the blockage determination. As noted above, in some exemplary embodiments, this subset of features includes four features, namely, number of detections, total detection energy, maximum detection range and average detection range. Once the features are selected, in some exemplary embodiments, a machine learning architecture is trained to learn the relationship between the features and the blockage state. In another approach, instead of performing feature selection to identify the optimal subset of useful features, all of the detections for each cycle are input to a machine learning process/architecture. In either case, the machine learning architecture can be any of a neural network architecture, a recurrent neural network architecture, a logistical regression architecture, a decision tree architecture or a bagged trees (random forest) architecture. In the case of the former approach, in which the selected subset of potential features is used, in one particular exemplary embodiment, logistical regression is used. In the latter case, in which all detections are processed without performing feature selection, in one particular embodiment, neural network machine learning is used. It will be understood that any machine learning approach can be applied to either technique.
As noted above, in an alternative approach, instead of performing feature selection to identify a subset of useful features, all of the detections for each cycle are input to a machine learning process/architecture.
With regard to the neural networks approach identified in
As noted above, with reference to
According to some exemplary embodiments, the image is designed to have three layers. In the first layer, we have a grid of size 40×40, where each grid cell represents a spatial region of length 4 meters on a side. Then, in each grid cell we enter the number of detections in that spatial region, during the 15-second window. On the second layer, we sum the energy over the spatial region, during the 15-second window. Finally, on the third layer, we include the values of four features used in the feature-based classifier described above in detail. By setting the central values of an image to the four values, we could input the values as part of an image, which made it easier to use in the neural network. Representative images are shown in
The neural network architecture according to some exemplary embodiments is shown in Table 3. It starts with a convolutional network and then a fully connected network. Dropout layers are used to prevent overfitting. The layers indicated by N can vary depending on whether we try to classify N=3 or N=2 levels of blockage. The neural network of Table 4 has 50,000 fit parameters, which is significantly more than the logistic model or the bagged tree model. It is preferable to use the minimal number of fit parameters for simplicity, to reduce memory usage, and to minimize the risk of overfitting.
Whereas many alterations and modifications of the disclosure will become apparent to a person of ordinary skill in the art after having read the foregoing description, it is to be understood that the particular embodiments shown and described by way of illustration are in no way intended to be considered limiting. Further, the subject matter has been described with reference to particular embodiments, but variations within the spirit and scope of the disclosure will occur to those skilled in the art. It is noted that the foregoing examples have been provided merely for the purpose of explanation and are in no way to be construed as limiting of the present disclosure.
While the present inventive concept has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present inventive concept as defined by the following claims.
This application is a continuation of U.S. patent application Ser. No. 16/257,817, filed on Jan. 25, 2019, the entire contents of which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6771207 | Lang | Aug 2004 | B1 |
| 7973701 | Lohmeier | Jul 2011 | B2 |
| 11280883 | Fetterman | Mar 2022 | B2 |
| 20090243912 | Lohmeier et al. | Oct 2009 | A1 |
| 20150309165 | Elwart | Oct 2015 | A1 |
| 20160320471 | Preussner | Nov 2016 | A1 |
| 20170059695 | Fetterman | Mar 2017 | A1 |
| 20170269196 | Millar et al. | Sep 2017 | A1 |
| 20180024569 | Branson | Jan 2018 | A1 |
| 20180143299 | Chabaud | May 2018 | A1 |
| 20180285319 | Nieuwenhuys | Oct 2018 | A1 |
| 20180301002 | Turcott | Oct 2018 | A1 |
| 20190025404 | Farmer | Jan 2019 | A1 |
| 20190072646 | Zeleny | Mar 2019 | A1 |
| 20190385025 | McMichael | Dec 2019 | A1 |
| 20200097767 | Perry | Mar 2020 | A1 |
| 20200134358 | She | Apr 2020 | A1 |
| 20200175332 | Karlinsky | Jun 2020 | A1 |
| 20200210887 | Jain | Jul 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 108027421 | May 2018 | CN |
| 4223346 | Jan 1994 | DE |
| 102005059902 | Jun 2007 | DE |
| 102017108348 | Jun 2018 | DE |
| 102017211816 | Jan 2019 | DE |
| 3290946 | Aug 2019 | EP |
| 20180133745 | Dec 2018 | KR |
| Entry |
|---|
| International Search Report and Written Opinion for International Application No. PCT/US2019/065675 dated Feb. 28, 2020. |
| Number | Date | Country | |
|---|---|---|---|
| 20220163631 A1 | May 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16257817 | Jan 2019 | US |
| Child | 17670015 | US |
