The present disclosure generally relates to vehicles, and more particularly relates to methods and radar systems for vehicles.
Certain vehicles today utilize radar systems. For example, certain vehicles utilize radar systems to detect other vehicles, pedestrians, or other objects on a road in which the vehicle is travelling. Radar systems may be used in this manner, for example, in implementing automatic braking systems, adaptive cruise control, and avoidance features, among other vehicle features. Certain vehicle radar systems, called multiple input, multiple output (MIMO) radar systems, have multiple transmitters and receivers. While radar systems are generally useful for such vehicle features, in certain situations existing radar systems may have certain limitations.
Accordingly, it is desirable to provide improved techniques for radar system performance in vehicles, for example for classification of objects using MIMO radar systems. It is also desirable to provide methods, systems, and vehicles utilizing such techniques. Furthermore, other desirable features and characteristics of the present invention will be apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background.
In accordance with an exemplary embodiment, a method is provided for controlling a radar system of a vehicle, the radar system having a plurality of receivers. The method comprises receiving radar signals pertaining to an object via each of the plurality of receivers, generating a plurality of feature vectors based on the radar signals, and generating a three dimensional representation of the object using the plurality of feature vectors.
In accordance with an exemplary embodiment, a radar control system for a vehicle is provided. The radar control system comprises one or more transmitters, a plurality of receivers, and a processor. The one or more transmitters are configured to transmit radar signals. The plurality of receivers are configured to receive return radar signals after the transmitted radar signals are deflected from an object proximate the vehicle. The processor is coupled to the plurality of receivers, and is configured to generate a plurality of feature vectors based on the radar signals and generate a three dimensional representation of the object using the plurality of feature vectors.
The present disclosure will hereinafter be described in conjunction with the following drawing figures, wherein like numerals denote like elements, and wherein:
The following detailed description is merely exemplary in nature and is not intended to limit the disclosure or the application and uses thereof. Furthermore, there is no intention to be bound by any theory presented in the preceding background or the following detailed description. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic, and/or processor device, individually or in any combination, including without limitation: application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.
In the depicted embodiment, the vehicle 10 also includes a chassis 112, a body 114, four wheels 116, an electronic control system 118, a steering system 150, and a braking system 160. The body 114 is arranged on the chassis 112 and substantially encloses the other components of the vehicle 10. The body 114 and the chassis 112 may jointly form a frame. The wheels 116 are each rotationally coupled to the chassis 112 near a respective corner of the body 114.
In the exemplary embodiment illustrated in
Still referring to
The steering system 150 is mounted on the chassis 112, and controls steering of the wheels 116. The steering system 150 includes a steering wheel and a steering column (not depicted). The steering wheel receives inputs from a driver of the vehicle 10. The steering column results in desired steering angles for the wheels 116 via the drive shafts 134 based on the inputs from the driver.
The braking system 160 is mounted on the chassis 112, and provides braking for the vehicle 10. The braking system 160 receives inputs from the driver via a brake pedal (not depicted), and provides appropriate braking via brake units (also not depicted). The driver also provides inputs via an accelerator pedal (not depicted) as to a desired speed or acceleration of the vehicle 10, as well as various other inputs for various vehicle devices and/or systems, such as one or more vehicle radios, other entertainment or infotainment systems, environmental control systems, lightning units, navigation systems, and the like (not depicted in
Also as depicted in
The radar control system 12 is mounted on the chassis 112. As mentioned above, the radar control system 12 classifies objects based upon a three dimensional representation of the objects using received radar signals of the radar system 103. In one example, the radar control system 12 provides these functions in accordance with the method 400 described further below in connection with
While the radar control system 12, the radar system 103, and the controller 104 are depicted as being part of the same system, it will be appreciated that in certain embodiments these features may comprise two or more systems. In addition, in various embodiments the radar control system 12 may comprise all or part of, and/or may be coupled to, various other vehicle devices and systems, such as, among others, the actuator assembly 120, and/or the electronic control system 118.
With reference to
As depicted in
With reference to
The radar system 103 generates the transmittal radar signals via the signal generator(s) 302. The transmittal radar signals are filtered via the filter(s) 304, amplified via the amplifier(s) 306, and transmitted from the radar system 103 (and from the vehicle 10 to which the radar system 103 belongs, also referred to herein as the “host vehicle”) via the antenna(e) 308. The transmitting radar signals subsequently contact other vehicles and/or other objects on or alongside the road on which the host vehicle 10 is travelling. After contacting the other vehicles and/or other objects, the radar signals are reflected, and travel from the other vehicles and/or other objects in various directions, including some signals returning toward the host vehicle 10. The radar signals returning to the host vehicle 10 (also referred to herein as received radar signals) are received by the antenna(e) 310, amplified by the amplifier(s) 312, mixed by the mixer(s) 314, and digitized by the sampler(s)/digitizer(s) 316.
Returning to
The processing unit 226 processes the information obtained by the receivers 222 for classification of objects based upon a three dimensional representation of the objects using received radar signals of the radar system 103. The processing unit 226 of the illustrated embodiment is capable of executing one or more programs (i.e., running software) to perform various tasks instructions encoded in the program(s). The processing unit 226 may include one or more microprocessors, microcontrollers, application specific integrated circuits (ASICs), or other suitable device as realized by those skilled in the art, such as, by way of example, electronic control component, processing logic, and/or processor device, individually or in any combination, including without limitation: application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.
In certain embodiments, the radar system 103 may include multiple memories 224 and/or processing units 226, working together or separately, as is also realized by those skilled in the art. In addition, it is noted that in certain embodiments, the functions of the memory 224, and/or the processing unit 226 may be performed in whole or in part by one or more other memories, interfaces, and/or processors disposed outside the radar system 103, such as the memory 242 and the processor 240 of the controller 104 described further below.
As depicted in
As depicted in
As depicted in
The memory 242 can be any type of suitable memory. This would include the various types of dynamic random access memory (DRAM) such as SDRAM, the various types of static RAM (SRAM), and the various types of non-volatile memory (PROM, EPROM, and flash). In certain examples, the memory 242 is located on and/or co-located on the same computer chip as the processor 240. In the depicted embodiment, the memory 242 stores the above-referenced program 250 along with one or more stored values 252 (such as, by way of example, information from the received radar signals and the spectrograms therefrom).
The bus 248 serves to transmit programs, data, status and other information or signals between the various components of the computer system 232. The interface 244 allows communication to the computer system 232, for example from a system driver and/or another computer system, and can be implemented using any suitable method and apparatus. The interface 244 can include one or more network interfaces to communicate with other systems or components. In one embodiment, the interface 244 includes a transceiver. The interface 244 may also include one or more network interfaces to communicate with technicians, and/or one or more storage interfaces to connect to storage apparatuses, such as the storage device 246.
The storage device 246 can be any suitable type of storage apparatus, including direct access storage devices such as hard disk drives, flash systems, floppy disk drives and optical disk drives. In one exemplary embodiment, the storage device 246 comprises a program product from which memory 242 can receive a program 250 that executes one or more embodiments of one or more processes of the present disclosure, such as the method 400 (and any sub-processes thereof) described further below in connection with
The bus 248 can be any suitable physical or logical means of connecting computer systems and components. This includes, but is not limited to, direct hard-wired connections, fiber optics, infrared and wireless bus technologies. During operation, the program 250 is stored in the memory 242 and executed by the processor 240.
It will be appreciated that while this exemplary embodiment is described in the context of a fully functioning computer system, those skilled in the art will recognize that the mechanisms of the present disclosure are capable of being distributed as a program product with one or more types of non-transitory computer-readable signal bearing media used to store the program and the instructions thereof and carry out the distribution thereof, such as a non-transitory computer readable medium bearing the program and containing computer instructions stored therein for causing a computer processor (such as the processor 240) to perform and execute the program. Such a program product may take a variety of forms, and the present disclosure applies equally regardless of the particular type of computer-readable signal bearing media used to carry out the distribution. Examples of signal bearing media include: recordable media such as floppy disks, hard drives, memory cards and optical disks, and transmission media such as digital and analog communication links. It will similarly be appreciated that the computer system 232 may also otherwise differ from the embodiment depicted in
As depicted in
After the radar signals are reflected from objects on or around the road, return radar signals are received by the radar system 103 at 406 of
The received radar signals are filtered at 408. In one embodiment, the received radar signals of 406 are passed through a filter bank stored in the memory 242 of
An object is identified in proximity to the vehicle at 410. Similar to the discussion above, as used throughout this Application, an object may comprise, among other possible examples, another vehicle, a pedestrian, a tree, a rock, debris, a guard rail or another road component, and so on, in proximity to the host vehicle 10. In one embodiment, the object is identified based on the received radar signals of 406 by a processor, such as the processing unit 226 and/or the processor 240 of
A location of the object of 410 is determined at 412. In addition, an azimuth angle for the object with respect to the host vehicle 10 is determined at 414, an elevation angle for the object with respect to the host vehicle 10 is determined at 416, and a range is determined for the object with respect to the host vehicle 10 at 418. In one embodiment, the location, azimuth angle, elevation angle, and range of 410-418 are determined for the object of 410 based on the received radar signals of 406 by a processor, such as the processing unit 226 and/or the processor 240 of
The spatially distributed radar signals are processed at 420. In one embodiment, the radar signals are associated to a three dimensional (3D) array at 420. In one embodiment, the three dimensional representation comprises a union of the plurality of feature vectors (or patches) of 420 over a three dimensional array. Also in one embodiment, the array of 422 is constructed such that the array has a first dimension based on the azimuth angle of 414, a second dimension based on the elevation angle of 416, a third dimension based on the range of 418, and a center that is based on the location of the object of 412. In other embodiments, the radar signals may be associated to a two dimensional (2D) array. In yet another embodiment, the radar signals may be associated to different spatial positions. In one embodiment 420 is performed by a processor, such as the processing unit 226 and/or the processor 240 of
Compressive sensing feature extraction is performed at 422. In one embodiment, during 420 features are extracted from the received radar signals using compressive sensing techniques. As used in this Application, compressive sensing techniques comprise techniques for finding radar signal decomposition dictionary in which signals are expressed compactly, for example as a technique for smart feature extraction. In one embodiment, 422 includes a signal processing technique for efficiently acquiring and reconstructing the radar signal by finding solutions to undetermined linear systems. In one such embodiment, a least squares mathematical solution may be utilized. In one embodiment, a plurality of feature vectors are generated at 422 based on the received radar signals using the compressive sensing techniques. In one such embodiment, a separate feature vector is generated for radar signals received from each of the plurality of receivers 222 of
The object is classified at 424. In one embodiment, the object is classified at 424 based on the three dimensional representation of 420 and the compressive sensing feature extraction of 422. In one embodiment, the classification comprises a predefined category or type of object (e.g., a pedestrian, another vehicle, a wall, and so). In another embodiment, the classification pertains to whether the object is of any concern (e.g. for possible impact). In certain embodiments, shape recognition may be performed as part of 424. In addition, in one embodiment, the classification of the object at 424 is performed by a processor, such as the processing unit 226 and/or the processor 240 of
In one embodiment, the classification of 424 consists of a training stage (e.g., prior to a current vehicle ignition cycle) and then a real time classification (e.g., during a current vehicle ignition cycle). In one embodiment, during the training stage a smart dictionary is built per object (class) category, and then the per class dictionaries are merged. Subsequently, each signal is decomposed in a new merged dictionary and energy signatures are built with the components being a sum of the absolute decomposition coefficients per specific “object” dictionaries. The learned dictionary is stored in a memory, such as the memory 224 and/or the memory 244 of
In one embodiment, a sparse dictionary learning is used based on a temporal gradient that captures a Doppler frequency shift with respect to radar signals deflected from the object (also referred to herein as the “target”). In one embodiment, the relatively short time interval the Doppler shift is proportional to the relative changes in the object's position. The sparse dictionary learning-based feature extraction reduces the data dimensionality to a small number, C, of basic target's directions of motion, whose combination is used to represent all other possible directions. Thus the proposed direction of motion estimation process can be presented in two stages. In the first stage the set of the C sparse dictionaries is learned from the training data. In the second stage any radar measurement that strongly depends on the target's direction of motion is decomposed in these dictionaries. These two stages are described in greater detail below.
First, in the dictionary learning phase, Let Λ={(X1, θ1), (X2, θ2), . . . , (XC, θC)} be a dictionary training data set, where an X×N matrix Xc=[x1(θc), x2(θc), . . . , xN(θc)], c=1, . . . , C is the collection of the X×1 slow-time radar echos received from the radar control system from N spatial cells when observing the target moving with direction θc. Each column of Xc is split into U overlapping frames of the size K, thus forming the K×U data sample matrices1 Yci, i=1, . . . , N. The training data for the dictionary c contains the radar echoes obtained from all spatial cells of interest (cells that contain the target) and has the following form:
Yc2KN×U=[R{Yc1};I{Yc1};R{Yc2};I{Yc2}; . . . ,R{YcN};I{YcN}] (Equation 1),
where R{•} and I{•} denote the real and the imaginary parts of the argument. Each column vector ycm, ∀m=1, . . . , U of the matrix Yc (the mth training sample for the dictionary c) consists of the radar echoes received from the N spatial cells of interest when observing the target moving with cth basic direction, therefore adding spatial information about the observed extended target to the training data.
The column vectors in Yc can be represented using the following linear model:
ycm=Dcαcm+ncm (Equation 2),
where ncm is the 2KN×1 additive noise vector with the limited energy, ∥ncm∥22<ε, that models additive noise and the deviation from the model, Dc is the 2KN×J possibly overcomplete (J>2KN) dictionary with J atoms, and αcm is the J×1 sparse vector of coefficients indicating atoms of Dc that represent data vector ycm. The dictionary Dc and the corresponding vectors of the sparse coefficients αcm, m=1, . . . , U can be learned from the training data by solving the following optimization problem:
({hacek over (D)}c,{hacek over (A)}c)=arg minDc,Ac½∥{hacek over (D)}c{hacek over (A)}c−Yc∥2F+ξΣUm=1∥αcm∥1 (Equation 3),
where ∥.∥ is the matrix Frobenious norm, and the J×U matrix Ac=[αc1, αc2, . . . , αcu] contains the sparse decomposition coefficients of the columns of the training data matrix Yc. Minimization of the first summand in Equation 3 decreases the error between the original data and its representation, while minimization of the second summand preserves the sparsity of the obtained solution. The coefficient ξ controls the trade-off between the reconstruction error and sparsity. The optimization problem in Equation 3 can be numerically solved using modern convex optimization techniques, for example the SPArse Modeling Software (SPAMS) toolbox. \
Because Micro-Doppler signatures for different targets' motion directions may have similarities, in one embodiment a non-class-specific dictionary is constructed, which contains characteristics of the C basic directions:
D2KN×JC=[D1,D2, . . . ,DC]. (Equation 4)
In this example, every measurement is represented as the combination of the selected basic directions of motion, while the corresponding decomposition coefficients are used as the features for classification or regression. Accordingly, in one embodiment, the learned dictionaries are used to represent as many data variations as possible.
In the above-referenced second stage of this example, the signature vectors are generated using the dictionary D for features extraction. In one embodiment, Let Λt={(X11, θ1), . . . , (X1Ft, θ1), . . . , (XCt1, θCt), . . . , (XCtFt, θCt)} be a regression training data set, where each one of the Ft data blocks Xctf, f=1, . . . , Ft is an Xt×N matrix that contains a slow-time radar echoes received from N spatial cells while observing target moving at direction θct.
Also in this example, TF defines the target observation period required for the decision on the target motion direction. For the pulse repetition period Tr, the target observation time TF and the dimensionality of the regression training data vector Xt are related in the following way: Xt=TF/Tr. In order to represent more directions of motion in the regression training data without increasing the number of dictionaries C, we assume that _t contains the radar data from the larger number of different directions than Λt (i.e. ΛtεΛt).
Each of the N columns of Xctf is split into Ut overlapping frames of the size K to form K×Ut matrices Yctfi, i=1, . . . , N. Similarly to Equation 1 above, these matrices are combined into an 2KN×Ut sample matrix Yctf. The columns of Yctf can be represented by the dictionary D by solving the following convex optimization problem:
{hacek over (A)}ctf=arg minActf½∥DActf−Yctf∥2F+ξUtj=1∥αctfj∥1 (Equation 5),
where Actf=[αctf1, αctf2, . . . αctfUt] is a JC×Ut matrix of corresponding sparse decompositions. The JC×1 vector αcfj=[(αctfj)1, . . . , (αctfj)J, . . . , (αctfj)JC]T, which is the sparse representation of the jth data sample from Yctf in the merged dictionary D, contains the decomposition coefficients of the ctth target's direction in the basis constructed from the C basic directions. The contribution of the cth basic direction to the decomposition of the data matrix Yctf can be obtained by the summation of the absolute values of all decomposition coefficients that correspond to the basic direction (c) over Ut data samples:
(βctf)c=ΣUtj=1ΣcJi=(c−1)J+1∥(αctfj)i∥2. (Equation 6)
The vector βctf=[(βctf)1, (βctf)2, . . . , (βctf)C]T can be considered as the energy signature of the data samples Yctf, where each entry of the βctf represents the energy contributed by the corresponding basic direction of motion. Using the signature vectors as features reduces the dimensionality of the data from Xt to the number of basic directions C. In addition, the signature vectors capture information about relations between different directions of motion. In one embodiment, the summation in Equation 6 over relatively small number of samples Ut in Yctf is expected to provide significantly higher robustness of the energy signature. After the signature vectors are extracted from Ft training data blocks for each one of the Ct different directions the following regression training data set can be constructed: Γt={(B1, θ1), (B2, θ2), . . . , (BCt, θCt)}, where Bct=[βct1, βct2, . . . , βctFt], ct=1, . . . , Ct. In one embodiment, the sparse-learning-based feature extraction from the radar micro-Doppler data and the energy signatures can be used for various types of classification of the object, such as the object's motion direction estimation, pedestrian activities classification, and ground moving targets recognition.
In various other embodiments, other classification techniques may be used. For example, in one embodiment the object of 410 is classified at 424 based upon the three dimensional representation of 422 and a circular regression model. For example, in one such embodiment, the data from the feature vectors of the three dimensional representation are defined on a circle (with respect to the sin and cosine functions), and circular regression models are applied to overcome any discontinuity issues. In various embodiments, the objects may be classified at 424 using the energy signatures of 422 by using any number of different techniques, such as, by way of example, support vector machine (SVM), mathematical linked pair (MLP), and other techniques, such as those discussed above.
The objects may be tracked over time at 426, for example by tracking changes in movement and/or position of the objects using the received radar signals of 406, the location determined at 412, and the classification of 424. In addition, in one embodiment, the tracking of the object at 426 is performed by a processor, such as the processing unit 226 and/or the processor 240 of
Vehicle actions may be initiated as appropriate at 428 based upon the classification and/or tracking. In addition, in one embodiment, the actions of 428 are performed by a processor, such as the processing unit 226 and/or the processor 240 of
By way of further example, in one embodiment, if the host vehicle 10 is determined to be in contact (or soon to be in contact) with the object, then the action(s) at 428 may further depend upon the classification of 424 as to the type of the object. For example, if the object is classified at 424 as being a pedestrian, then a first set of actions may be taken at 428 to reduce the stiffness of the host vehicle 10, for example by opening a hood of the host vehicle 10 for protection of the pedestrian. Conversely, if the object is classified at 424 as being a brick wall, then a second set of actions may instead be taken at 428 to increase the stiffness of the host vehicle 10 for protection of the occupants of the host vehicle 10.
In various embodiments, the method 400 may terminate at 430 when the action is complete, or when further use of the radar system and/or the method 400 is no longer required (e.g. when the ignition is turned off and/or the current vehicle drive and/or ignition cycle terminates).
With reference to
A two dimensional representation 620 of the same object 506 (in this example, a pedestrian) is provided in
Methods and systems are provided herein for classifying objects for radar systems of vehicles. The disclosed methods and systems provide for the classification of objects using based upon a three dimensional representation of the objects using received radar signals of the radar system 103.
It will be appreciated that the disclosed methods, systems, and vehicles may vary from those depicted in the Figures and described herein. For example, the vehicle 10, the radar control system 12, the radar system 103, the controller 104, and/or various components thereof may vary from that depicted in
While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist. It should also be appreciated that the exemplary embodiment or exemplary embodiments are only examples, and are not intended to limit the scope, applicability, or configuration of the disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing the exemplary embodiment or exemplary embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the scope of the appended claims and the legal equivalents thereof.
Number | Name | Date | Kind |
---|---|---|---|
4833469 | David | May 1989 | A |
5467072 | Michael | Nov 1995 | A |
5610620 | Stites et al. | Mar 1997 | A |
5751240 | Fujita | May 1998 | A |
5767793 | Agravante | Jun 1998 | A |
5973648 | Lindenmeier et al. | Oct 1999 | A |
6674394 | Zoratti | Jan 2004 | B1 |
6680689 | Zoratti | Jan 2004 | B1 |
7053845 | Holloway et al. | May 2006 | B1 |
7224309 | Shimomura | May 2007 | B2 |
7868817 | Meyers | Jan 2011 | B2 |
20030151541 | Oswald | Aug 2003 | A1 |
20040117090 | Samukawa | Jun 2004 | A1 |
20060262007 | Bonthron | Nov 2006 | A1 |
20090040501 | Matsuo | Feb 2009 | A1 |
20090184865 | Valo | Jul 2009 | A1 |
20090322871 | Ji | Dec 2009 | A1 |
20100109938 | Oswald | May 2010 | A1 |
20110102242 | Takeya | May 2011 | A1 |
20120035846 | Sakamoto | Feb 2012 | A1 |
20120194377 | Yukumatsu | Aug 2012 | A1 |
20120313810 | Nogueira-Nine | Dec 2012 | A1 |
20140022113 | Nogueira-Nine | Jan 2014 | A1 |
20140049420 | Lehning | Feb 2014 | A1 |
20140077989 | Healy, Jr. | Mar 2014 | A1 |
20140176679 | Lehning | Jun 2014 | A1 |
20150198711 | Zeng | Jul 2015 | A1 |
Number | Date | Country |
---|---|---|
101581780 | Nov 2009 | CN |
102879777 | Jan 2013 | CN |
102944876 | Feb 2013 | CN |
Entry |
---|
USPTO, Notice of Allowance and Fee(s) Due for U.S. Appl. No. 12/886,322 mailed Feb. 27, 2015. |
USPTO, Response to Designated New Ground of Rejection Pursuant to 37 C.F.R. Section 1.111 and Request to Reopen Prosecution Pursuant to 37 CFR Section 41.40 and MPEP Section 1207.03(B) for U.S. Appl. No. 12/886,322 mailed Aug. 4, 2014. |
USPTO, Decision on Petition to Designate New Grounds of Rejection and Reopen Prosecution for U.S. Appl. No. 12/886,322 mailed Jun. 12, 2014. |
USPTO, Petition to Designate a New Ground of Rejection and to Reopen Prosecution under 37 C.F.R. Sections 1.181 and 41.40 and MPEP Section 2107.03(b) for U.S. Appl. No. 12/886,322 mailed Apr. 14, 2014 (087.0022). |
USPTO, Examiner's Answer for U.S. Appl. No. 12/886,322 mailed Feb. 14, 2014. |
State Intellectual Property Office of the People's Republic of China, Office Action in Chinese Patent Application No. 201510383757.8 dated Mar. 20, 2017. |
Amit Kumar Mishra, et al., “Information sensing for radar target classification using compressive sensing,” IRS 2012, 19th International Radar Symposium, May 23-25, Warsaw, Poland, pp. 326-330. |
Ming-Hua Xue, et al., “Research on Three-Dimensional Imaging Algorithm of Radar Target,” Radar Science and Technology, vol. 11, No. 1, Feb. 2013, pp. 65-70. |
Number | Date | Country | |
---|---|---|---|
20160003939 A1 | Jan 2016 | US |