METHOD OF TARGET FEATURE EXTRACTION AND MULTI-TARGET TRACKING BASED ON 4D MILLIMETER WAVE RADAR

Abstract

A method of target feature extraction and multi-target tracking based on 4D millimeter wave radar includes acquiring the point cloud data output from the radar and preprocess the point cloud data to eliminate the outlier value; processing the points acquired after preprocessing by Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering; processing each of the acquired clusters for target feature extraction; predicting the existing trajectory is predicted, and the prediction results are globally correlated with the clustering results; based on the correlation clustering results, the corresponding multi-target tracking motion state processing is executed to obtain the transverse pendulum angular velocity; and based on the target motion state, the target size information as well as the heading angle information are updated.

Description

FIELD OF TECHNOLOGY

The present invention relates to the technical field of computer vision, in particular to the field of target feature extraction and tracking, and specifically refers to a method, a device, a processor and a computer-readable storage medium thereof for realizing target feature extraction and multi-target tracing based on 4D millimeter wave radar.

DESCRIPTION OF RELATED ARTS

Millimeter wave radar is less affected by environmental factors such as rain, snow, fog, etc., and has all-weather, all-day operating characteristic. Conventional millimeter wave radar detection of targets is usually a single point or a number of strongly reflecting points, although it is possible to measure the velocity information of the target, it is still less information than what can be provided by cameras and lidar. 4D millimeter-wave radar has enhanced angular resolution in both horizontal and pitch directions compared to conventional radar, enabling richer information about the target to be detected. As a result of richer target information, one of the manifestations of which is a greater number of reflected points detected for a single target, conventional millimeter-wave radar point-track-based target tracking methods are difficult to apply directly to 4D millimeter-wave radar and do not fully utilize the information provided by 4D millimeter-wave radar. Among them, the relevant techniques that have been disclosed in the field all suffer from different technical defects, as follows: Application number: CN202111612665.4 the invention patent application proposes a multi-target tracking and breath detection method and device based on millimeter wave radar. In the multi-target tracking method provided by the invention, a radar point cloud is directly associated to an existing trajectory and the existing trajectory is updated; Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering is performed on the remaining unassociated radar point clouds then only the center of the clusters is taken as the start of the new trajectory. The invention does not fully utilize the velocity observation information of the target provided by the millimeter wave radar, and also does not further extract richer target features from the clustered point cloud.

Application number: CN202210473036.6 the invention patent application proposes a multi-target tracking method based on millimeter wave radar. In the multi-target tracking method provided by the invention, the gaussian likelihood function values are calculated by computing the residual and covariance matrices between the radar clustered targets and the trajectory predictions to compute the probability of interconnection. In fact, the radar clustering targets are not in one-to-one correspondence with the state parameters of the trajectory prediction values, and the calculation of the likelihood function requires either the establishment of a target observation model or feature extraction of the radar clustering results to match the trajectory state parameters, neither of which is accounted for in this application. In addition, the application utilizes a probabilistic data correlation filter for Doppler in the trajectory updating step, but the Doppler information measured by the millimeter wave radar is the radial velocity component of the target, and there is a large error in updating the target's kinematic state by directly utilizing this information.

Application number: CN201910661302.6 the invention patent application proposes a multi-target tracking method. In the multi-target tracking method provided by the invention, feature extraction was performed after clustering the radar point cloud data. The invention takes the average coordinates of the same class of points as an estimate of the target coordinates; the absolute value of the maximum difference between the x-axis and y-axis of the same class of points and the average coordinates as the long and short axes as an estimate of the target's dimensions; and the average value of the velocity of the same class of points as an estimate of the target's velocity. The radar target feature extraction method provided by the invention is only applicable to targets that move in a straight line along the x-axis or y-axis, and when the target is deflected, its estimated target size is large and lacks heading information; when the direction of the target's motion is not relative to the radial motion of the radar, the mean value of the velocity at the same class of points only reflects the radial component of the target's velocity, and there is a large error in updating the target's velocity directly with this.

In summary, based on the relevant technical content disclosed in the prior art, there are technical functional limitations that cannot simultaneously satisfy the comprehensive assessment of precision tracking and observation correlation reliability.

SUMMARY OF THE INVENTION

It is an object of the present invention to overcome the drawbacks of the above-mentioned prior art, and to provide a method, device, processor, and computer-readable storage medium of target feature extraction and multi-target tracking based on 4D millimeter wave radar.

In order to achieve the above objectives, the method, device, processor, and computer-readable storage medium of target feature extraction and multi-target tracking based on 4D millimeter wave radar are as follows:

The method of target feature extraction and multi-target tracking based on 4D millimeter wave radar, the main feature of which is that the said method comprises following steps:

• • (1) Acquire the point cloud data output from the radar and preprocess the point cloud data to eliminate the outlier value;
  • (2) The points acquired after preprocessing are processed by DBSCAN clustering;
  • (3) Each of the acquired clusters is processed for target feature extraction;
  • (4) The existing trajectory is predicted, and the prediction results are globally correlated with the clustering results;
  • (5) Based on the correlation clustering results, the corresponding multi-target tracking motion state processing is executed to obtain the transverse pendulum angular velocity;
  • (6) Based on the target motion state, the target size information as well as the heading angle information are updated.

Preferably, the said step (1) is specifically:

Acquiring point cloud data output from the radar and performing data preprocessing of said point cloud data to remove points with abnormal velocities, remove points with abnormal heights, and filter points outside the region of interest to remove outlier values.

Preferably, the said step (2) is specifically:

• • the DBSCAN clustering process is performed based on the coordinate position of each point in the point cloud information and the relative radial velocity as:

$\{\begin{array}{c}❘x_i-x_j❘<0.5m\\ ❘y_i-y_j❘<0.5m\\ ❘z_i-z_j❘<2.m\\ ❘{\mathrm{vr}}_i-{\mathrm{vr}}_j❘<2.m/s\end{array}$

• • where i, j denote the indexes of the two points involved in the discrimination, x, y, z denote the coordinate position of each point respectively, and vr denotes the relative radial velocity.

Preferably, the said step (3) specifically comprises the following steps:

• • (3.1) calculate the clustered mean radial velocity vr, the x-mean x and the y-mean y;
  • (3.2) the maximum and minimum values of the clustered points in the z direction are counted, and the difference of the maximum value minus the minimum value is used as the cluster height feature;
  • (3.3) subtract x, y from the x, y coordinates of each point under the same clustering to get x_i, y_i respectively;
  • (3.4) traversing the range 0 to 90° at 1° intervals and rotating all the resulting x_i, y_i at each angle, determine the rotated x_max, y_max, x_min and y_min, and compute the loss function using the following equation:

$\mathrm{Loss}=\sum _{i=1}^N\min \left(x_i-x_{\min },x_{\max }-x_i,y_i-y_{\min },y_{\max }-y_i\right)$

• • the minimum angular value of angle and the corresponding x_max, y_max, x_min and y_min of the said loss function are also recorded;
  • (3.5) use the larger of (x_max−x_min) and (y_max−y_min) as the clustering length feature and the smaller as the clustering width feature, and use the mean x_mean of x_mean, x_min and the mean y_mean of y_max, y_min after rotating by—angle, and then add x, y to get the value as the clustering center coordinates (x_c, y_c);
  • (3.6) if (x_max−x_min)<(y_max−y_min), angle increases by 90 degrees, and the angle is more than 90 degrees with the direction of the average radial velocity vr, then the angle is increased by another 180 degrees and the final angle value is taken as the heading angle feature for clustering;
  • (3.7) according to the said clustering center coordinates, clustering length features, clustering width features, clustering height features and heading angle features obtained as described above, a clustered rectangular target bounding box is constructed for subsequent target feature extraction processing.

Preferably, the said step (4) specifically comprises the following steps:

• • (4.1) the motion states are taken as x, v_x, a_x, y, v_y, a_y and the existing trajectory prediction is processed using linear Kalman filtering as:

$\left[\begin{array}{c}{x}_{\mathrm{pred}}\\ v_{\mathrm{xpred}}\\ a_{\mathrm{xpred}}\\ y_{\mathrm{pred}}\\ v_{\mathrm{ypred}}\\ a_{\mathrm{ypred}}\end{array}\right]=\left[\begin{array}{cccccc}1& T& 0.5T^2& 0& 0& 0\\ 0& 1& T& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 1& T& 0.5T^2\\ 0& 0& 0& 0& 1& T\\ 0& 0& 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ v_x\\ a_x\\ y\\ v_y\\ a_y\end{array}\right]$

• • where v_x is the x-direction velocity component, a_x is the x-direction acceleration component, v_y is the y-direction velocity component, a_y is the y-direction acceleration component, x_pred is the predicted value of the target's x-direction position, v_xpred is the predicted value of the x-direction velocity component, a_xpred is the predicted value of the x-direction acceleration component, y_pred is the predicted value of the target's y-direction position, v_ypred is the predicted value of the y-direction velocity component, a_ypred is the predicted value of the y-direction acceleration component, and Tis the time interval between the neighboring two frames of radar point cloud;
  • (4.2) calculate the diagonal length dist_cri_1 using the length and width of the existing trajectory:

$\mathrm{dist\_cri}\_1=\sqrt{\begin{array}{c}\mathrm{the}\mathrm{length}\mathrm{of}\mathrm{the}\mathrm{existing}{\mathrm{trajectory}}^2+\\ \mathrm{the}\mathrm{width}\mathrm{of}\mathrm{the}\mathrm{existing}{\mathrm{trajectory}}^2\end{array}}$

• • the diagonal length dist_cri_2 is calculated using the clustered length and width features:

$\mathrm{dist\_cri}\_2=\sqrt{\mathrm{clustered}\mathrm{length}{\mathrm{features}}^2+\mathrm{clustered}\mathrm{width}{\mathrm{features}}^2}$

• • and weighted by weights of 0.6 and 0.4 to calculate the upper limit of the association distance dist_cri

$\mathrm{dist\_cri}=0.6\times \mathrm{dist\_cri}\_1+0.4\times \mathrm{dist\_cri}\_2$

• • calculate the Euclidean distance dist between the predicted position of the existing trajectory (x_pred, y_pred) and the clustering center:

$\mathrm{dist}=\sqrt{{\left(x_c-x_{\mathrm{pred}}\right)}^2+{\left(y_c-y_{\mathrm{pred}}\right)}^2}$

• • and calculate the distance correlation score according to the following equation:

$\mathrm{distance\_score}=\{\begin{array}{cc}0,& \mathrm{dist}>\mathrm{dist\_cri}\\ \left(1-\frac{\mathrm{dist}}{\mathrm{dist\_cri}}\right)\times 100,& \mathrm{dist}\le \mathrm{dist\_cri}\end{array}$

• • (4.3) based on the existing trajectory prediction position, the clustering position, the size, and the heading angle, the coordinates of the vertices of each vertex of the said clustered rectangular target bounding box and the area of the rectangular box are calculated, and the size correlation score is calculated according to the following equation:

$\mathrm{area\_score}=\{\begin{array}{cc}100,& \begin{array}{c}\mathrm{An}\mathrm{overlap}\mathrm{between}\\ \mathrm{the}\mathrm{existing}\mathrm{trajectories}\\ \mathrm{and}\mathrm{the}\mathrm{clustering}\mathrm{results}\end{array}\\ \left(1-\frac{\begin{array}{c}\min (\mathrm{existing}\\ \mathrm{trajectory}\mathrm{area},\\ \mathrm{cluster}\mathrm{area})\end{array}}{\begin{array}{c}\max (\mathrm{existing}\\ \mathrm{trajectory}\mathrm{area},\\ \mathrm{cluster}\mathrm{area})\end{array}}\right)\times 100,& \mathrm{otherwise}\end{array}$

• • (4.4) based on the predicted positions, predicted velocities and pendulum angular velocities of the existing trajectories, the predicted radial velocities of each point of the to-be-associated clusters are calculated by the following equation:

$v_{\mathrm{ri}}=\left(v_{\mathrm{xpred}}-\omega y_{\mathrm{pred}}\right)\cos {\phi }_i+\left(v_{\mathrm{ypred}}+\omega x_{\mathrm{pred}}\right)\sin {\phi }_i$

• • where v_ri denotes the predicted radial velocity, ω denotes the target transverse swing angular velocity, and φ_i denotes the radial velocity direction of each point in the clustered point cloud;
  • the average predicted radial velocity is acquired by averaging the predicted radial velocities at each point within the same cluster, and the difference is made with the measured average radial velocity of the cluster, get the difference delta_vr, and calculate the speed correlation score by using the following equation:

$\mathrm{velocity\_score}=\{\begin{array}{cc}0,& ❘\mathrm{delta\_vr}❘>\mathrm{vr\_cri}\\ \left(1-\frac{❘\mathrm{delta\_vr}❘}{\mathrm{vr\_cri}}\right)\times 100,& ❘\mathrm{delta\_vr}❘\le \mathrm{vr\_cri}\end{array}$

• • where vr_cri is the preset threshold;
  • (4.5) the distance correlation score, size correlation score, and speed correlation score acquired from the above calculations are weighted and summed to obtain the final correlation score of the trajectory with the clusters;
  • (4.6) construct an association matrix based on the final association scores of each cluster with each existing trajectory, including: the number of rows of the matrix is equal to the number of existing trajectories, the number of columns of the matrix is the number of current clusters, the value of the i-th row and j-th column of the matrix is the association score of the i-th existing track with the j-th cluster, and the global association matching is performed based on the association matrix.

Preferably, the said step (5) comprises:

• • compute observation model of trajectories using the following equation:

$\left(\begin{array}{c}{x}_c\\ y_c\\ \stackrel{\_}{v_r}\end{array}\right)=\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0\\ 0& \stackrel{\_}{\cos \phi }& 0& 0& \stackrel{\_}{\sin \phi }& 0\end{array}\right]\left(\begin{array}{c}x\\ \mathrm{vx}\\ \mathrm{ax}\\ y\\ \mathrm{vy}\\ \mathrm{ay}\end{array}\right)$

where x_c, y_c are the clustering centers, f-r is the mean radial velocity of the clustered point cloud, cos φ is the average cosine of the mean azimuthal observations of the clustered point cloud, sin φ is the average sinusoidal of the mean azimuthal observations of the clustered point cloud, x, vx, ax, y, vy and ay are the motion states of the trajectory.

Preferably, the said step (5) specifically comprises the following steps:

• • (5.1) Determine whether the corresponding clusters are successfully associated with each existing trajectory, if yes, go to step (5.2), otherwise, go to step (5.4);
  • (5.2) For the associated pre-existing trajectories, the predicted observations x_cpred, y_cpred and v_rpred are computed for the trajectories based on the trajectory predicted motion state and the said trajectory observation model:

$\left(\begin{array}{c}{x}_{\mathrm{cpred}}\\ y_{\mathrm{cpred}}\\ \stackrel{\_}{v_{\mathrm{rpred}}}\end{array}\right)=\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0\\ 0& \stackrel{\_}{\cos \phi }& 0& 0& \stackrel{\_}{\sin \phi }& 0\end{array}\right]\left(\begin{array}{c}{x}_{\mathrm{pred}}\\ v_{\mathrm{xpred}}\\ a_{\mathrm{xpred}}\\ y_{\mathrm{pred}}\\ v_{\mathrm{ypred}}\\ a_{\mathrm{ypred}}\end{array}\right)$

• • (5.3) The corresponding clustered eigenvalues under this clustering are subtracted from the predicted observations to get the residuals, and multiply this residual by the Kalman gain matrix K and add the trajectory predicted motion state to get the updated result of the already associated trajectory motion state:

$\left(\begin{array}{c}{x}_{\mathrm{update}}\\ v_{\mathrm{xupdate}}\\ a_{\mathrm{xupdate}}\\ y_{\mathrm{update}}\\ v_{\mathrm{yupdate}}\\ a_{\mathrm{yupdate}}\end{array}\right)=K\left(\left(\begin{array}{c}{x}_c\\ y_c\\ \stackrel{\_}{v_r}\end{array}\right)-\left(\begin{array}{c}{x}_{\mathrm{cpred}}\\ y_{\mathrm{cpred}}\\ \stackrel{\_}{v_{\mathrm{rpred}}}\end{array}\right)\right)+\left(\begin{array}{c}{x}_{\mathrm{pred}}\\ v_{\mathrm{xpred}}\\ a_{\mathrm{xpred}}\\ y_{\mathrm{pred}}\\ v_{\mathrm{ypred}}\\ a_{\mathrm{ypred}}\end{array}\right)$

• • where x_update, y_xupdate, a_xupdate, y_update, v_yupdate and a_yupdate are the updated trajectory motion states;
  • (5.4) eliminate the determination of the trajectory that fails to be associated, if its consecutive cumulative number of unassociated times is less than a preset threshold, the trajectory prediction result will be used as the update result to continue the trajectory extrapolation, otherwise the trajectory will be deleted directly;
  • (5.5) A new trajectory is created for the clusters with failed associations, and the motion state parameters of the new trajectory are expressed using the following equation:

$\left[\begin{array}{c}x\\ v_x\\ a_x\\ y\\ v_y\\ a_y\end{array}\right]=\left[\begin{array}{c}{x}_c\\ \frac{\stackrel{\_}{v_r}}{\cos \left(\mathrm{orient}-\mathrm{vrAngle}\right)}\times \cos \left(\mathrm{orient}\right)\\ 0\\ y_c\\ \frac{\stackrel{\_}{v_r}}{\cos \left(\mathrm{orient}-\mathrm{vrAngle}\right)}\times \sin \left(\mathrm{orient}\right)\\ 0\end{array}\right]$

• • where orient is the clustered heading angle feature, vrAngle is the mean direction angle of the radial velocity of the point cloud, x_c, y_c are the clustering centers, and v_r is the mean radial velocity of the clustered point cloud;
  • (5.6) Based on the currently acquired update results of the associated trajectory motion state, the trajectory extrapolation results, and the calculation results of the new trajectory motion state, the corresponding transverse pendulum angular velocity ω is calculated by the following equation:

$❘\omega ❘=\frac{\sqrt{a_x^2+a_y^2}}{\sqrt{v_x^2+v_y^2}}$

• • where the positive and negative values of v_x·a_y−v_y·a_x, the result of the cross-multiplication of the velocity vector and the acceleration vector, are taken as the positive and negative values of the transverse pendulum angular velocity.

Preferably, the said step (5.5) specifically comprises the following steps:

• • (5.5.1) Using the center of clustering as the new track start position;
  • (5.5.2) Using clustered heading angle features as velocity directions for new trajectories, the clustered mean radial velocities are back-projected into the heading angle direction to obtain the velocity of the new trajectory;
  • (5.5.3) Take the new trajectory acceleration to zero;
  • (5.5.4) Take said clustered length feature as the length of the new trajectory, take said clustered width feature as the width of the new trajectory, and take the clustered heading angle feature as the heading angle of the new trajectory.

Preferably, the said step (6) specifically comprises the following steps:

• • (6.1) The trajectory length is weighted and averaged with the clustered length features to get the trajectory length update results, and the trajectory width is weighted and averaged with the clustered width features to get the trajectory width update results;
  • (6.2) Based on the velocity vector and acceleration vector of the trajectory motion state, the acceleration component perpendicular to the direction of the velocity vector is computed, the magnitude of the transverse pendulum angular velocity of the target is then obtained by dividing the mode of the vertical acceleration component by the mode of the velocity vector, and the direction of the cross product of the velocity vector and the acceleration vector is the direction of the transverse pendulum angular velocity;
  • (6.3) The velocity heading angle is calculated based on the trajectory speed, and the clustered heading angle features, velocity heading angle, and trajectory heading angle are weighted and averaged to obtain the trajectory heading angle update results.

The device of target feature extraction and multi-target tracking based on 4D millimeter wave radar, the main feature of which is that the said device comprises:

• • processor, configured to execute computer-executable instructions;
  • memory, storing one or more computer-executable instructions, when the said computer-executable instructions are executed by the said processor, various steps for realizing the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar as claimed in above-described.

The processor of target feature extraction and multi-target tracking based on 4D millimeter wave radar, the main feature of which is that the said processor being configured to execute computer-executable instructions, when the said processor being configured to execute computer-executable instructions, various steps for realizing the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar as claimed in above-described.

The computer-readable storage medium, the main feature of which is that the computer program stored on it, and the said computer program may be executed by a processor to implement the various steps for realizing the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar as claimed in above-described.

With the use of this method, device, processor, and computer-readable storage medium of target feature extraction and multi-target tracking based on 4D millimeter wave radar, compared with the traditional radar multi-target tracking method, it obtains information such as target size and heading angle in addition to position from the radar clustered point cloud, and fully combines this information to improve the reliability of the association between trajectory and observation in multi-target tracking; and for 4D millimeter-wave radar, a new trajectory observation model is provided, through which individual observation points of the same target can be fully utilized to update in concert; In addition to this, the present invention introduces a target transverse pendulum angular velocity as a means to enable more stable tracking of turning maneuvering targets.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flowchart of a method for realizing target feature extraction and multi-target tracking based on a 4D millimeter wave radar of the present invention.

FIG. 2 shows the effect of feature extraction on point cloud clustering and representation into a rectangular bounding box of the present invention.

FIG. 3 shows a schematic diagram of the effect of target tracking of a motor vehicle in a specific embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In order to be able to understand the technical content of the present invention more clearly, is further exemplified by the following detailed description of embodiments.

Before describing in detail the embodiments according to the present invention, it should be noted that, in the following, the terms “including”, “comprising” or any other variant are intended to cover non-exclusive inclusion, so that a processes, methods, goods, or equipment comprising a set of elements contains more than just those elements, and it also contains other elements that are not explicitly listed or that are inherent to such processes, methods, goods, or equipment.

Referring to FIG. 1, the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar, wherein the said method comprises following steps:

• • (1) Acquire the point cloud data output from the radar and preprocess the point cloud data to eliminate the outlier value;
  • (2) The points acquired after preprocessing are processed by DBSCAN clustering;
  • (3) Each of the acquired clusters is processed for target feature extraction;
  • (4) The existing trajectory is predicted, and the prediction results are globally correlated with the clustering results;
  • (5) Based on the correlation clustering results, the corresponding multi-target tracking motion state processing is executed to obtain the transverse pendulum angular velocity;
  • (6) Based on the target motion state, the target size information as well as the heading angle information are updated.

As a preferred embodiment of the present invention, the said step (1) is specifically: Acquiring point cloud data output from the radar and performing data preprocessing of said point cloud data to remove points with abnormal velocities, remove points with abnormal heights, and filter points outside the region of interest to remove outlier values.

As a preferred embodiment of the present invention, the said step (2) is specifically: the DBSCAN clustering process is performed based on the coordinate position of each point in the point cloud information and the relative radial velocity as:

$\{\begin{array}{c}❘x_i-x_j❘<0.5m\\ ❘y_i-y_j❘<0.5m\\ ❘z_i-z_j❘<2.m\\ ❘{\mathrm{vr}}_i-{\mathrm{vr}}_j❘<2.m/s\end{array}$

• • where i, j denote the indexes of the two points involved in the discrimination, x, y, z denote the coordinate position of each point respectively, and vr denotes the relative radial velocity;

As a preferred embodiment of the present invention, the said step (3) specifically comprises the following steps:

• • (3.1) calculate the clustered mean radial velocity vr, the x-mean x and the y-mean y;
  • (3.2) the maximum and minimum values of the clustered points in the z direction are counted, and the difference of the maximum value minus the minimum value is used as the cluster height feature;
  • (3.3) subtract x, y from the x, y coordinates of each point under the same clustering to get x_i, y_i respectively;
  • (3.4) traversing the range 0 to 90° at 1° intervals and rotating all the resulting x_i, y_i at each angle, determine the rotated x_max, y_max, x_min and y_min, and compute the loss function using the following equation:

$\mathrm{Loss}=\sum _{i=1}^N\min \left(x_i-x_{\min },x_{\max }-x_i,y_i-y_{\min },y_{\max }-y_i\right)$

• • the minimum angular value of angle and the corresponding x_max, y_max, x_min and y_min of the said loss function are also recorded;
  • (3.5) use the larger of (x_max−x_min) and (y_max−y_min) as the clustering length feature and the smaller as the clustering width feature, and use the mean x_mean of x_mean, x_min and the mean y_mean of y_max, y_min after rotating by—angle, and then add x, y to get the value as the clustering center coordinates (x_c, y_c);
  • (3.6) if (x_max−x_min)<(y_max−y_min), angle increases by 90 degrees, and the angle is more than 90 degrees with the direction of the average radial velocity vr, then the angle is increased by another 180 degrees and the final angle value is taken as the heading angle feature for clustering;
  • (3.7) according to the said clustering center coordinates, clustering length features, clustering width features, clustering height features and heading angle features obtained as described above, a clustered rectangular target bounding box is constructed for subsequent target feature extraction processing;

As a preferred embodiment of the present invention, the said step (4) specifically comprises the following steps:

• • (4.1) the motion states are taken as x, v_x, a_x, y, v_y, a_y and the existing trajectory prediction is processed using linear Kalman filtering as:

$\left[\begin{array}{c}{x}_{\mathrm{pred}}\\ v_{\mathrm{xpred}}\\ a_{\mathrm{xpred}}\\ y_{\mathrm{pred}}\\ v_{\mathrm{ypred}}\\ a_{\mathrm{ypred}}\end{array}\right]=\left[\begin{array}{cccccc}1& T& 0.5T^2& 0& 0& 0\\ 0& 1& T& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 1& T& 0.5T^2\\ 0& 0& 0& 0& 1& T\\ 0& 0& 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ v_x\\ a_x\\ y\\ v_y\\ a_y\end{array}\right]$

• • where v_x is the x-direction velocity component, a_x is the x-direction acceleration component, v_y is the y-direction velocity component, a_y is the y-direction acceleration component, x_pred is the predicted value of the target's x-direction position, v_xpred is the predicted value of the x-direction velocity component, a_xpred is the predicted value of the x-direction acceleration component, y_pred is the predicted value of the target's y-direction position, v_ypred is the predicted value of the y-direction velocity component, a_ypred is the predicted value of the y-direction acceleration component, and Tis the time interval between the neighboring two frames of radar point cloud;
  • (4.2) calculate the diagonal length dist_cri_1 using the length and width of the existing trajectory:

$\mathrm{dist\_cri}\_1=\sqrt{\begin{array}{c}\mathrm{the}\mathrm{length}\mathrm{of}\mathrm{the}\mathrm{existing}{\mathrm{trajectory}}^2+\\ \mathrm{the}\mathrm{width}\mathrm{of}\mathrm{the}\mathrm{existing}{\mathrm{trajectory}}^2\end{array}}$

• • the diagonal length dist_cri_2 is calculated using the clustered length and width features:

$\mathrm{dist\_cri}\_2=\sqrt{\mathrm{clustered}\mathrm{length}{\mathrm{features}}^2+\mathrm{clustered}\mathrm{width}{\mathrm{features}}^2}$

• • and weighted by weights of 0.6 and 0.4 to calculate the upper limit of the association distance dist_cri

$\mathrm{dist\_cri}=0.6\times \mathrm{dist\_cri}\_1+0.4\times \mathrm{dist\_cri}\_2$

• • calculate the Euclidean distance dist between the predicted position of the existing trajectory (x_pred, y_pred) and the clustering center:

$\mathrm{dist}=\sqrt{{\left(x_c-x_{\mathrm{pred}}\right)}^2+{\left(y_c-y_{\mathrm{pred}}\right)}^2}$

• • and calculate the distance correlation score according to the following equation:

$\mathrm{distance\_score}=\{\begin{array}{cc}0,& \mathrm{dist}>\mathrm{dist\_cri}\\ \left(1-\frac{\mathrm{dist}}{\mathrm{dist\_cri}}\right)\times 100,& \mathrm{dist}\le \mathrm{dist\_cri}\end{array}$

• • (4.3) based on the existing trajectory prediction position, the clustering position, the size, and the heading angle, the coordinates of the vertices of each vertex of the said clustered rectangular target bounding box and the area of the rectangular box are calculated, and the size correlation score is calculated according to the following equation:

$\mathrm{area\_score}=\{\begin{array}{cc}100,& \begin{array}{c}\mathrm{An}\mathrm{overlap}\mathrm{between}\\ \mathrm{the}\mathrm{existing}\mathrm{trajectories}\\ \mathrm{and}\mathrm{the}\mathrm{clustering}\mathrm{results}\end{array}\\ \left(1-\frac{\begin{array}{c}\min (\mathrm{existing}\\ \mathrm{trajectory}\mathrm{area},\\ \mathrm{cluster}\mathrm{area})\end{array}}{\begin{array}{c}\max (\mathrm{existing}\\ \mathrm{trajectory}\mathrm{area},\\ \mathrm{cluster}\mathrm{area})\end{array}}\right)\times 100,& \mathrm{otherwise}\end{array}$

• • (4.4) based on the predicted positions, predicted velocities and pendulum angular velocities of the existing trajectories, the predicted radial velocities of each point of the to-be-associated clusters are calculated by the following equation:

$v_{\mathrm{ri}}=\left(v_{\mathrm{xpred}}-\omega y_{\mathrm{pred}}\right)\cos {\phi }_i+\left(v_{\mathrm{ypred}}+\omega x_{\mathrm{pred}}\right)\sin {\phi }_i$

• • where v_ri denotes the predicted radial velocity, ω denotes the target transverse swing angular velocity, and φi denotes the radial velocity direction of each point in the clustered point cloud;
  • the average predicted radial velocity is acquired by averaging the predicted radial velocities at each point within the same cluster, and the difference is made with the measured average radial velocity of the cluster, get the difference delta_vr, and calculate the speed correlation score by using the following equation:

$\mathrm{velocity\_score}=\{\begin{array}{cc}0,& ❘\mathrm{delta\_vr}❘>\mathrm{vr\_cri}\\ \left(1-\frac{❘\mathrm{delta\_vr}❘}{\mathrm{vr\_cri}}\right)\times 100,& ❘\mathrm{delta\_vr}❘\le \mathrm{vr\_cri}\end{array}$

• • where vr_cri is the preset threshold;
  • (4.5) the distance correlation score, size correlation score, and speed correlation score acquired from the above calculations are weighted and summed to obtain the final correlation score of the trajectory with the clusters;
  • (4.6) construct an association matrix based on the final association scores of each cluster with each existing trajectory, including: the number of rows of the matrix is equal to the number of existing trajectories, the number of columns of the matrix is the number of current clusters, the value of the i-th row and j-th column of the matrix is the association score of the i-th existing track with the j-th cluster, and the global association matching is performed based on the association matrix.

As a preferred embodiment of the present invention, the said step (5) comprises:

• • compute observation model of trajectories using the following equation:

$\left(\begin{array}{c}{x}_c\\ y_c\\ \stackrel{\_}{v_r}\end{array}\right)=\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0\\ 0& \stackrel{\_}{\cos \phi }& 0& 0& \stackrel{\_}{\sin \phi }& 0\end{array}\right]\left(\begin{array}{c}x\\ \mathrm{vx}\\ \mathrm{ax}\\ y\\ \mathrm{vy}\\ \mathrm{ay}\end{array}\right)$

• • where x_c, y_c are the clustering centers, v_r is the mean radial velocity of the clustered point cloud, cos φ is the average cosine of the mean azimuthal observations of the clustered point cloud, sin φ is the average sinusoidal of the mean azimuthal observations of the clustered point cloud, x, vx, ax, y, vy and ay are the motion states of the trajectory.

As a preferred embodiment of the present invention, the said step (5) specifically comprises the following steps:

• • (5.1) Determine whether the corresponding clusters are successfully associated with each existing trajectory, if yes, go to step (5.2), otherwise, go to step (5.4);
  • (5.2) For the associated pre-existing trajectories, the predicted observations x_cpred, y_cpred and v_rpred are computed for the trajectories based on the trajectory predicted motion state and the said trajectory observation model:

$\left(\begin{array}{c}{x}_{\mathrm{cpred}}\\ y_{\mathrm{cpred}}\\ \stackrel{\_}{v_{\mathrm{rpred}}}\end{array}\right)=\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0\\ 0& \stackrel{\_}{\cos \phi }& 0& 0& \stackrel{\_}{\sin \phi }& 0\end{array}\right]\left(\begin{array}{c}{x}_{\mathrm{pred}}\\ v_{\mathrm{xpred}}\\ a_{\mathrm{xpred}}\\ y_{\mathrm{pred}}\\ v_{\mathrm{ypred}}\\ a_{\mathrm{ypred}}\end{array}\right)$

• • (5.3) The corresponding clustered eigenvalues under this clustering are subtracted from the predicted observations to get the residuals, and multiply this residual by the Kalman gain matrix K and add the trajectory predicted motion state to get the updated result of the already associated trajectory motion state:

$\left(\begin{array}{c}{x}_{\mathrm{update}}\\ v_{\mathrm{xupdate}}\\ a_{\mathrm{xupdate}}\\ y_{\mathrm{update}}\\ v_{\mathrm{yupdate}}\\ a_{\mathrm{yupdate}}\end{array}\right)=K\left(\left(\begin{array}{c}{x}_c\\ y_c\\ \stackrel{\_}{v_r}\end{array}\right)-\left(\begin{array}{c}{x}_{\mathrm{cpred}}\\ y_{\mathrm{cpred}}\\ \stackrel{\_}{v_{\mathrm{rpred}}}\end{array}\right)\right)+\left(\begin{array}{c}{x}_{\mathrm{pred}}\\ v_{\mathrm{xpred}}\\ a_{\mathrm{xpred}}\\ y_{\mathrm{pred}}\\ v_{\mathrm{ypred}}\\ a_{\mathrm{ypred}}\end{array}\right)$

• • where x_update, v_xupdate, a_xupdate, y_update, v_yupdate and a_yupdate are the updated trajectory motion states;
  • (5.4) eliminate the determination of the trajectory that fails to be associated, if its consecutive cumulative number of unassociated times is less than the preset threshold, the trajectory prediction result will be used as the update result to continue the trajectory extrapolation, otherwise the trajectory will be deleted directly;
  • (5.5) A new trajectory is created for the clusters with failed associations, and the motion state parameters of the new trajectory are expressed using the following equation:

$\left[\begin{array}{c}x\\ v_x\\ a_x\\ y\\ v_y\\ a_y\end{array}\right]=\left[\begin{array}{c}{x}_c\\ \frac{\stackrel{\_}{v_r}}{\cos \left(\mathrm{orient}-\mathrm{vrAngle}\right)}\times \cos \left(\mathrm{orient}\right)\\ 0\\ y_c\\ \frac{\stackrel{\_}{v_r}}{\cos \left(\mathrm{orient}-\mathrm{vrAngle}\right)}\times \sin \left(\mathrm{orient}\right)\\ 0\end{array}\right]$

• • where orient is the clustered heading angle feature, vrAngle is the mean direction angle of the radial velocity of the point cloud, x_c, y_c are the clustering centers, and v_r is the mean radial velocity of the clustered point cloud;
  • (5.6) Based on the currently acquired update results of the associated trajectory motion state, the trajectory extrapolation results, and the calculation results of the new trajectory motion state, the corresponding transverse pendulum angular velocity a is calculated by the following equation:

$❘\omega ❘=\frac{\sqrt{a_x^2+a_{y^2}}}{\sqrt{v_x^2+v_y^2}}$

• • where the positive and negative values of v_x·a_y−v_y·a_x, the result of the cross-multiplication of the velocity vector and the acceleration vector, are taken as the positive and negative values of the transverse pendulum angular velocity.

As a preferred embodiment of the present invention, the said step (5.5) specifically comprises the following steps:

• • (5.5.1) Using the center of clustering as the new track start position;
  • (5.5.2) Using clustered heading angle features as velocity directions for new trajectories, the clustered mean radial velocities are back-projected into the heading angle direction to obtain the velocity of the new trajectory;
  • (5.5.3) Take the new trajectory acceleration to zero;
  • (5.5.4) Take said clustered length feature as the length of the new trajectory, take said clustered width feature as the width of the new trajectory, and take the clustered heading angle feature as the heading angle of the new trajectory.

As a preferred embodiment of the present invention, the said step (6) specifically comprises the following steps:

• • (6.1) The trajectory length is weighted and averaged with the clustered length features to get the trajectory length update results, and the trajectory width is weighted and averaged with the clustered width features to get the trajectory width update results;
  • (6.2) Based on the velocity vector and acceleration vector of the trajectory motion state, the acceleration component perpendicular to the direction of the velocity vector is computed, the magnitude of the transverse pendulum angular velocity of the target is then obtained by dividing the mode of the vertical acceleration component by the mode of the velocity vector, and the direction of the cross product of the velocity vector and the acceleration vector is the direction of the transverse pendulum angular velocity;
  • (6.3) The velocity heading angle is calculated based on the trajectory speed, and the clustered heading angle features, velocity heading angle, and trajectory heading angle are weighted and averaged to obtain the trajectory heading angle update results.

In practical application, this target feature extraction method provided by the present invention mainly includes the following steps: 1. calculate the clustered mean radial velocity, mean x-value and mean y-value; 2. determine the extreme deviation of the clustered points in the z-axis direction to get the clustering height characteristics; 3. will be within the same class of x, y coordinates of each point minus x, y mean, traversing the range of 0 to 90 degrees, for each angle, in the xoy plane will be subtracted from the mean value of the point rotated to the corresponding angular value, and to determine the maximum value of the rotated x value x_max, the maximum value of y value y_max, the minimum x value of x_min and the minimum value of y value y_min, the calculation of the loss function is as follows:

$\mathrm{Loss}=\sum _{i=1}^N\min \left(x_i-x_{\min },x_{\max }-x_i,y_i-y_{\min },y_{\max }-y_i\right)$

• • where x_i, y_i denotes the coordinates of the points after subtracting the mean and rotating, N denotes the number of points in the same category, and min(denotes taking the smallest of the four. Obtain the angle value angle that minimizes the above loss function and the corresponding x_max, y_max, x_min and y_min. 4. Use the larger of (x_max−x_min) and (y_max−y_min) as the clustering length feature and the smaller as the clustering width feature. The mean x_mean of x_max, x_min and the mean y_mean of y_max, y_min are used as the cluster center coordinates after rotating—angle and adding the mean x, y of the point cloud. 5. If (x_max−x_min)<(y_max−y_min), angle increases by 90 degrees; if the angle between the direction of angle and the direction of the average radial velocity (the average velocity is greater than zero, then pointing to the center of the clusters from the origin, and vice versa, pointing to the origin from the center of the clusters) is more than 90 degrees, the angle is increased by another 180 degrees. The final angle value is taken as the heading angle feature of the clusters.

In practical application, the multi-target tracking method provided by the present invention mainly comprises the following steps: 1. acquire the point cloud data output from the radar, preprocess the point cloud and eliminate the outlier value; 2. clustering based on position, relative radial velocity, reflection intensity, etc. in the point cloud information and target feature extraction using the method provided by the present invention; 3. predict the current existing trajectory, and match the prediction results with the clustering results; 4. for trajectories on the association, update, and for unassociated tracks, extrapolate or extinguish; 5. generate new trajectories for unassociated clustering results; 6. for the trajectory currently being tracked, the target size information is updated according to its associated clustering results, the transverse pendulum angular velocity is calculated according to the target motion state, and the heading angle is updated according to the clustering results and the target motion state.

The matching association of the trajectory prediction results with the clustering results of step 3 comprises the following steps: 3.1. calculate the upper limit of the association distance dist_cri based on the size of the clusters and the size of the existing trajectories; calculate the Euclidean distance dist between the predicted position of the trajectory and the clustering center; calculate the distance correlation score using dist and dist_cri. 3.2 calculation of size correlation scores based on clustered positions, dimensions, and heading angles with those of existing trajectories. 3.3 predicted radial velocities for each point of the cluster to be correlated are calculated based on the predicted positions, predicted velocities, and transverse pendulum angular velocity of the existing trajectories:

$v_{\mathrm{ri}}=\left(v_x-\omega y\right)\cos {\phi }_i+\left(v_y+\omega x\right)\sin {\phi }_i$

• • where v_ri denotes the predicted radial velocity, v_x, v_y denote the velocity of trajectory prediction, ω denotes the target transverse swing angular velocity, and φ_i denotes the radial velocity direction of each point in the clustered point cloud; the average predicted radial velocity is acquired by averaging the predicted radial velocities at each point within the same cluster, and the difference is made with the measured average radial velocity of the cluster, get the difference delta_vr, to calculate the speed correlation score. 3.4 the distance correlation score, size correlation score, and speed correlation score acquired are weighted and summed to obtain the final correlation score of the trajectory with the clusters; 3.5 construct an association matrix based on the final association scores of each cluster with each existing trajectory, perform a global association match.

The correlated trajectory update in step 4 first requires a trajectory observation model. Aiming at the characteristic that a 4D millimeter wave radar can observe a point cloud of the target and only the radial velocity of the target relative to the radar, the present invention provides an observation model as follows:

$\left(\begin{array}{c}{x}_c\\ y_c\\ \stackrel{\_}{v_r}\end{array}\right)=\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0\\ 0& \stackrel{\_}{\cos \phi }& 0& 0& \stackrel{\_}{\sin \phi }& 0\end{array}\right]\left(\begin{array}{c}x\\ \mathrm{vx}\\ \mathrm{ax}\\ y\\ \mathrm{vy}\\ \mathrm{ay}\end{array}\right)$

• • where x_c, y_c are the clustering centers, f-r is the mean radial velocity of the clustered point cloud, cos φ is the average cosine of the mean azimuthal observations of the clustered point cloud, sin φ is the average sinusoidal of the mean azimuthal observations of the clustered point cloud. In the above equation, the trajectory motion statex, v_x, a_x, y, v_y, a_y is taken as an example, representing the x-direction position, x-direction velocity component, x-direction acceleration component, y-direction position, y-direction velocity component and y-direction acceleration component of the target respectively. The motion state of the actual trajectory is not limited to this form, but can be converted to it to make use of the observational models described above. Step 4 comprises the following specific steps: 4.1. For the existing trajectory that has been associated, calculate the predicted observation value of the trajectory based on the predicted motion state of the trajectory and the observation model described above; 4.2. Subtract the corresponding clustered eigenvalues from the predicted observations to obtain the residuals. 4.3. Multiply the residuals by the filter gain and add the trajectory predicted motion state to get the updated result of the already associated trajectory motion state. 4.4 For an existing track that has not been associated, if its cumulative number of consecutive unassociated times is less than the preset threshold, the trajectory prediction result will be used as the update result to continue extrapolation, otherwise the trajectory will be deleted directly.

Step 5 Generating a new trajectory comprises the following specific steps: 5.1 Using the center of clustering as the new trajectory start position; 5.2 Using clustered heading angle features as velocity directions for new trajectories, the clustered mean radial velocities are back-projected into the heading angle direction to obtain the velocity of the new trajectory; 5.3 Take the new trajectory acceleration to zero; 5.4 Take the clustered length features and width features are used as the length and width of the new trajectory respectively, and heading angle features are used as the starting heading angle of the new trajectory.

Step 6 comprises the following specific steps: 6.1 The trajectory sizes were weighted and averaged with the clustered sizes to obtain the trajectory size update results; 6.2 Based on the velocity vector and acceleration vector of the trajectory motion state, the acceleration component perpendicular to the direction of the velocity vector is computed, the magnitude of the transverse pendulum angular velocity of the target is then obtained by dividing the mode of the vertical acceleration component by the mode of the velocity vector, and the direction of the cross product of the velocity vector and the acceleration vector is the direction of the transverse pendulum angular velocity; 6.3 The velocity heading angle is calculated based on the trajectory speed, and the clustered heading angle features, velocity heading angle, and trajectory heading angle are weighted and averaged to obtain the trajectory heading angle update results.

As a specific embodiment of the present invention, this method of the present invention for realizing target feature extraction and multi-target tracking based on 4D millimeter wave radar specifically comprises the following steps:

• • Step 1: After acquiring a radar point cloud, carry out point cloud pre-processing, including deleting points with abnormal velocity, deleting points with abnormal height, filtering points outside the region of interest, and the like.
  • Step 2: Clustering the pre-processed points. DBSCAN clustering is used in this embodiment, the minimum number of neighborhoods of the core points is taken as 3, and the discriminative condition of the neighborhoods is:

$\{\begin{array}{c}❘x_i-x_j❘<0.5m\\ ❘y_i-y_j❘<0.5m\\ ❘z_i-z_j❘<2.m\\ ❘{\mathrm{vr}}_i-{\mathrm{vr}}_j❘<2.m/s\end{array}$

• • where i, j denote the indexes of the two points involved in the discrimination, x, y, z and vr denote the coordinate position of each point respectively and the relative radial velocity.
  • Step 3: For each cluster, feature extraction is performed.
  • 3.1 Calculate the clustered mean radial velocity vr, the x-mean x and the y-mean y;
  • 3.2 Determine the extreme deviation of the clustered points in the z-axis direction to obtain the cluster height feature;
  • 3.3 Subtract x, y from the x, y coordinates of each point under the same clustering to get x_i, y_i respectively; traversing the range 0 to 90° at 1° intervals and rotating all the resulting x_i, y_i at each angle, determine the rotated x_max, y_max, x_min and y_min, and compute the loss function:

$\mathrm{Loss}=\sum _{i=1}^N\min \left(x_i-x_{\min },x_{\max }-x_i,y_i-y_{\min },y_{\max }-y_i\right)$

• • the minimum angular value of angle and the corresponding x_max, y_max, x_min and y_min of the said loss function are also recorded;
  • 3.4 Use the larger of (x_max−x_min) and (y_max−y_min) as the clustering length feature and the smaller as the clustering width feature, and use the mean x_mean of x_mean, x_min and the mean y_mean of y_max, y_min after rotating by—angle, and then add x, y to get the value as the clustering center coordinates;
  • 3.5 Calculate clustered heading angle features. if (x_max−x_min)<(y_max−y_min), angle increases by 90 degrees, if the angle between the direction of angle and the direction of the mean radial velocity (mean velocity greater than zero, pointing from the origin to the center of the cluster, and vice versa, pointing from the center of the cluster to the origin) exceeds 90 degrees, the angle is increased by another 180 degrees and the final angle value is taken as the heading angle feature for clustering.

In this embodiment, a rectangular bounding box representing the target is constructed based on the clustering features (center coordinates, length, width, height, heading angle), with the effect shown in FIG. 2.

Step 4: Motion state prediction for existing trajectory. Here the motion state is taken as (x, v_x, a_x, y, v_y, a_y), then the prediction can be done using linear Kalman filter prediction process:

$\left[\begin{array}{c}{x}_{\mathrm{pred}}\\ v_{\mathrm{xpred}}\\ a_{\mathrm{xpred}}\\ y_{\mathrm{pred}}\\ v_{\mathrm{ypred}}\\ a_{\mathrm{ypred}}\end{array}\right]=\left[\begin{array}{c}\begin{array}{ccccccc}1& T& 0.5T^2& & 0& 0& 0\\ 0& 1& T& & 0& 0& 0\\ 0& 0& 1& & 0& 0& 0\end{array}\\ \begin{array}{ccccccc}0& 0& 0& & 1& T& 0.5T^2\\ 0& 0& 0& & 0& 1& T\\ 0& 0& 0& & 0& 0& 1\end{array}\end{array}\right]\left[\begin{array}{c}x\\ v_x\\ a_x\\ y\\ v_y\\ a_y\end{array}\right]$

Step 5: Correlate the existing trajectory prediction results with the clustering.

• • 5.1 Using the length and width of the existing trajectory, to calculate the diagonal length, the upper bound dist_cri of the association distance is calculated by weights of 0.6 and 0.4; calculate the Euclidean distance between the predicted position of the existing trajectory (x_pred, y_pred) with the clustering center, and calculate the distance correlation score:

$\mathrm{distance\_score}=\{\begin{array}{cc}0,& \mathrm{dist}>\mathrm{dist\_cri}\\ \left(1-\frac{\mathrm{dist}}{\mathrm{dist\_cri}}\right)\times 100,& \mathrm{dist}\le \mathrm{dist\_cri}\end{array}$

• • 5.2 The coordinates of the vertices of the rectangular box and the area of the rectangular box are calculated based on the position, size, and heading angle of the existing trajectories and clusters, and the size correlation score is calculated:

$\mathrm{area\_score}=\{\begin{array}{cc}100,& \begin{array}{c}\mathrm{An}\mathrm{overlap}\mathrm{between}\mathrm{the}\\ \mathrm{existing}\mathrm{trajectories}\\ \mathrm{and}\mathrm{the}\mathrm{clustering}\mathrm{results}\end{array}\\ \left(1-\frac{\begin{array}{c}\min (\mathrm{existing}\mathrm{trajectory}\mathrm{area},\\ \mathrm{cluster}\mathrm{area})\end{array}}{\begin{array}{c}\max (\mathrm{existing}\mathrm{trajectory}\mathrm{area},\\ \mathrm{cluster}\mathrm{area})\end{array}}\right)\times 100,& \mathrm{otherwise}\end{array}$

• • 5.3 The predicted mean radial velocity was calculated and subtracted from the clustered point cloud mean radial velocity to obtain the difference delta_vr and calculate the velocity correlation score:

$\mathrm{velocity\_score}=\{\begin{array}{cc}0,& ❘\mathrm{delta\_vr}❘>\mathrm{vr\_cri}\\ \left(1-\frac{❘\mathrm{delta\_vr}❘}{\mathrm{vr\_cri}}\right)\times 100,& ❘\mathrm{delta\_vr}❘\le \mathrm{vr\_cri}\end{array}$

• • where vr_cri is the preset threshold, 2.5 m/s is taken in this embodiment.
  • 5.4 The distance correlation score, size correlation score and speed correlation score are weighted and summed to calculate a final score. The weights in this embodiment are 0.4, 0.3, and 0.3, respectively.
  • 5.5 A final score for each existing trajectory with each cluster is calculated and a score matrix is constructed for matching. In this embodiment, each element of the score matrix is subtracted from 100 to convert the score matrix to a cost matrix.
  • 5.6 Subtract each row of the cost matrix from the smallest element of that row, and subsequently subtract each column of the matrix from the smallest element of that column.
  • 5.7 Thread all the zeros in the new matrix with the least number of rows and columns and check if the allocation is currently optimal. If the row and column lines do not thread all the elements of the matrix, go to step 5.8, otherwise go to step 5.9.
  • 5.8 Find the smallest element of the elements where the rows and columns are not threaded together, subtract the smallest element from the remaining elements, add the smallest element to the element that corresponds to the intersection of the rows and columns, and go back to step 5.7.
  • 5.9 Find the 0 element corresponding to each row and the 0 element corresponding to the column, and find the optimal allocation based on the 0 element.
  • Step 6: Kalman filtering is applied to the successfully associated trajectories to update the motion state.
  • Step 7: eliminate determination for trajectories with failed associations. If the number of extrapolated frames of the existing trajectory has reached a preset threshold, which is taken to be 3 frames in this embodiment, the trajectory is deleted, otherwise the prediction result of the existing trajectory is used as the extrapolation result.
  • Step 8: Establish a new trajectory based on the clustering of correlation failures. The motion state parameters of the new trajectory are:

$\left[\begin{array}{c}x\\ v_x\\ a_x\\ y\\ v_y\\ a_y\end{array}\right]=\left[\begin{array}{c}{x}_c\\ \frac{\stackrel{\_}{v_r}}{\cos \left(\mathrm{orient}-\mathrm{vrAngle}\right)}\times \cos \left(\mathrm{orient}\right)\\ 0\\ y_c\\ \frac{\stackrel{\_}{v_r}}{\cos \left(\mathrm{orient}-\mathrm{vrAngle}\right)}\times \sin \left(\mathrm{orient}\right)\\ 0\end{array}\right]$

• • where orient is the clustered heading angle feature, vrAngle is the mean direction angle of the radial velocity of the point cloud. The length and width of the new trajectory are taken from the length feature and width feature of the cluster, and the heading angle of the new trajectory is taken from the heading angle feature of the cluster.
  • Step 9: Calculate the transverse pendulum angular velocity for the current trajectory (including existing trajectory updates, extrapolation results and new trajectories). The magnitude of the transverse pendulum angular velocity a is:

$❘\omega ❘=\frac{\sqrt{a_x^2+a_{y^2}}}{\sqrt{v_x^2+v_y^2}}$

• • the cross-multiplication of the velocity vector and the acceleration vector v_x·a_y−v_y·a_x is the plus or minus of the transverse pendulum angular velocity.
  • Step 10: Update the current trajectory with sizes and heading angles. The trajectory sizes are weighted and averaged with the clustered sizes, and the weights in this embodiment are 0.7 and 0.3, respectively, to obtain the trajectory sizes update results:

$\mathrm{Trajectory}\mathrm{length}\mathrm{update}\mathrm{results}=0.7\times \mathrm{Original}\mathrm{trajectory}\mathrm{length}+0.3\times \mathrm{Cluster}\mathrm{length}\mathrm{features}$ $\mathrm{Trajectory}\mathrm{width}\mathrm{update}\mathrm{results}=0.7\times \mathrm{Original}\mathrm{trajectory}\mathrm{width}+0.3\times \mathrm{Cluster}\mathrm{width}\mathrm{features}$

The velocity heading angle is calculated based on the trajectory speed, and the clustered heading angle, the velocity heading angle, and the trajectory heading angle are weighted and averaged, with the weights in this embodiment being 0.4, 0.3, and 0.3, respectively, to obtain the trajectory heading angle update result:

$\mathrm{Trajectory}\mathrm{heading}\mathrm{angle}\mathrm{update}\mathrm{result}=0.4\times \mathrm{velocity}\mathrm{heading}\mathrm{angle}+0.3\times \mathrm{clustered}\mathrm{heading}\mathrm{angle}+0.3\times \mathrm{Original}\mathrm{trajectory}\mathrm{heading}\mathrm{angle}$

The device of target feature extraction and multi-target tracking based on 4D millimeter wave radar, wherein the said device comprises:

• • processor, configured to execute computer-executable instructions;
  • memory, storing one or more computer-executable instructions, when the said computer-executable instructions are executed by the said processor, various steps for realizing the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar as claimed in above-described.

The processor of target feature extraction and multi-target tracking based on 4D millimeter wave radar, wherein the said processor being configured to execute computer-executable instructions, when the said processor being configured to execute computer-executable instructions, various steps for realizing the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar as claimed in above-described.

The computer-readable storage medium, wherein the computer program stored on it, and the said computer program may be executed by a processor to implement the various steps for realizing the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar as claimed in above-described.

Any process or method description depicted in the flowchart or otherwise described herein may be understood to represent a module, fragment, or portion of code comprising one or more executable instructions for implementing the steps of a particular logical function or process, and that the scope of the preferred embodiments of the present invention includes additional implementations, which may be, in no particular order as shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order, according to the functions involved, should be understood by those skilled in the art to which embodiments of the present invention belong.

It should be understood that various parts of the invention may be implemented with hardware, software, firmware, or combinations thereof. In the above embodiments, a plurality of steps or methods may be implemented with software or firmware stored in memory and executed by a suitable instruction execution device.

One of ordinary skill in the art can appreciate that all or some of the steps carried out to realize the method of the above embodiments can be accomplished by instructing the associated hardware by means of a program, which can be stored in a computer-readable storage medium that, when executed, comprises one of the steps of the method embodiments or a combination thereof.

The storage media mentioned above may be read-only memories, disks or CD, etc.

In the description of this specification, reference to the terms “an embodiment”, “some embodiments”, “example”, “specific example”, or “embodiment” means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Moreover, specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as a limitation of the present invention, and that one of ordinary skill in the art may make changes, modifications, substitutions, and variations of the above embodiments within the scope of the present invention.

With the use of this method, device, processor, and computer-readable storage medium of target feature extraction and multi-target tracking based on 4D millimeter wave radar, compared with the traditional radar multi-target tracking method, it obtains information such as target size and heading angle in addition to position from the radar clustered point cloud, and fully combines this information to improve the reliability of the association between trajectory and observation in multi-target tracking; and for 4D millimeter-wave radar, a new trajectory observation model is provided, through which individual observation points of the same target can be fully utilized to update in concert; In addition to this, the present invention introduces a target transverse pendulum angular velocity as a means to enable more stable tracking of turning maneuvering targets.

In this specification, the present invention has been described with the reference to its specific embodiments. However, it is obvious still may be made without departing from the spirit and scope of the present invention, various modifications and transformation. Accordingly, the specification and drawings should be considered as illustrative rather than restrictive.

Claims

1. A method of target feature extraction and multi-target tracking based on 4D millimeter wave radar, comprising the following steps: (1) acquiring point cloud data output from the radar and preprocess the point cloud data to eliminate an outlier value;(2) processing points acquired after preprocessing by Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering;(3) processing each of acquired clusters for target feature extraction;(4) predicting an existing trajectory, and globally correlating prediction results with clustering results;(5) based on correlation clustering results, executing a corresponding multi-target tracking motion state processing to obtain a transverse pendulum angular velocity;(6) based on a target motion state, updating target size information as well as the heading angle information;wherein step (2) comprises: performing the DBSCAN clustering process based on a coordinate position of each point in the point cloud data and a relative radial velocity as:

2. The method of target feature extraction and multi-target tracking based on 4D millimeter wave radar according to claim 1, wherein said step (1) comprises: acquiring point cloud data output from the radar and performing data preprocessing of said point cloud data to remove points with abnormal velocities, remove points with abnormal heights, and filter points outside a region of interest to remove outlier values.

3. The method of target feature extraction and multi-target tracking based on 4D millimeter wave radar according to claim 1, wherein said step (5) comprises: computing observation model of trajectories using the following equation:

4. The method of target feature extraction and multi-target tracking based on 4D millimeter wave radar according to claim 3, wherein said step (5) comprises the following steps: (5.1) determining whether corresponding clusters are successfully associated with each existing trajectory, if yes, go to step (5.2), otherwise, go to step (5.4);(5.2) for associated pre-existing trajectories, predicted observations xcpred, ycpred and vrpred are computed for trajectories based on a trajectory predicted motion state and a trajectory observation model:

5. The method of target feature extraction and multi-target tracking based on 4D millimeter wave radar according to claim 4, wherein said step (5.5) comprises the following steps: (5.5.1) using a center of clustering as a new trajectory start position;(5.5.2) using clustered heading angle features as velocity directions for new trajectories, wherein clustered mean radial velocities are back-projected into a heading angle direction to obtain a velocity of a new trajectory;(5.5.3) taking a new trajectory acceleration to zero;(5.5.4) taking a clustered length feature as a length of the new trajectory, taking a clustered width feature as a width of the new trajectory, and taking a clustered heading angle feature as a heading angle of the new trajectory.

6. The method of target feature extraction and multi-target tracking based on 4D millimeter wave radar according to claim 4, wherein said step (6) comprises the following steps: (6.1) a trajectory length is weighted and averaged with clustered length features to get trajectory length update results, and a trajectory width is weighted and averaged with clustered width features to get trajectory width update results;(6.2) based on a velocity vector and an acceleration vector of a trajectory motion state, an acceleration component perpendicular to a direction of the velocity vector is computed, a magnitude of a transverse pendulum angular velocity of the target is then obtained by dividing a mode of a vertical acceleration component by a mode of the velocity vector, and a direction of a cross product of the velocity vector and the acceleration vector is the direction of the transverse pendulum angular velocity;(6.3) a velocity heading angle is calculated based on a trajectory speed, and clustered heading angle features, velocity heading angle, and trajectory heading angle are weighted and averaged to obtain trajectory heading angle update results.

7. A device of target feature extraction and multi-target tracking based on 4D millimeter wave radar, comprising: a processor, configured to execute computer-executable instructions;a memory, storing one or more computer-executable instructions, when executed by said processor, for performing the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar as claimed in claim 1.

8. A processor of target feature extraction and multi-target tracking based on 4D millimeter wave radar, configured to execute computer-executable instructions to perform the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar as claimed in claim 1.

9. A non-transitory computer-readable storage medium, storing a computer program to be executed by a processor to implement the method of target feature extraction and multi-target tracking based on 4D millimeter wave radar as claimed in claim 1.