Fusion Algorithm for Vidar Traffic Surveillance System

Abstract

This invention is related to a fusion algorithm for a video-Doppler-radar (Vidar) traffic surveillance system comprising of (1) a robust matching algorithm which iteratively matches the information from a video camera and multiple Doppler radars corresponding to a same moving vehicle, and (2) a stochastic algorithm which fuses the matched information from the video camera and Doppler radars to derive the vehicle velocity and range information.

Description

TECHNICAL FIELD

This invention relates to a fusion algorithm for a Vidar traffic surveillance system.

BACKGROUND OF THE INVENTION

A traditional radar based traffic surveillance system uses a Doppler radar for vehicle speed monitoring which measures a vehicle speed at line-of-sight (LOS). In FIG. 1, the speed of an approaching (or a leaving) vehicle is calculated in terms of Doppler frequency f_D by

$\begin{array}{cc}{v}_t=\frac{f_D}{K\cos \left({\phi }_t\right)}& \left(1\right)\end{array}$

where K is a Doppler frequency conversion constant and φ_t is called the Doppler cone angle or simply the Doppler angle. Although a Doppler radar based system has an advantage of a long detection range, there are several difficulties associated with the traditional radar based system, including (1) the Doppler radar beam angle is too large to precisely locate vehicles within the radar beam; (2) the angle between the vehicle moving direction and the LOS, is unknown and therefore, needs to be small enough for a reasonable speed estimation accuracy; (3) since all velocity vectors on the equal-Doppler cone in FIG. 1 will generate a same speed, the Doppler radar cannot differentiate the vehicles with a same speed but different directions defined by the same equal-Doppler cone. Therefore, no precise target location information can be derived in a traditional Doppler radar based traffic surveillance system.

Video Camera Based Traffic Surveillance Systems

A video camera based traffic surveillance system uses a video camera to capture a traffic scene and relies on computer vision techniques to indirectly calculate vehicle speeds. Precise vehicle locations can be identified. However, since no direct speed measurements are available and the camera has a finite number of pixels, the video camera based traffic surveillance system can be used only in a short distance application.

Video-Doppler-Radar (Vidar) Traffic Surveillance Systems

A video-Doppler-radar (Vidar) traffic surveillance system combines both the Doppler radar based system and the video based system into a unique system to preserve the advantages of both systems and overcome the shortcomings of both systems. A patent application on Vidar traffic surveillance system has been filed by the first author, Patent Application No. 12266227.

A Vidar traffic surveillance system may include a first movable Doppler radar to generate a first radar beam along the direction of a first motion ray, a second movable Doppler radar to generate a second radar beam along the direction of a second motion ray, a third fixed Doppler radar to generate a third radar beam along a direction ray, a video camera to serve as an information fusion platform by intersecting the first and second radar motion rays with the camera virtual image plane, a data processing device to process Doppler radar and video information, a tracking device to continuously point the surveillance system to the moving vehicle, and a recording device to continuously record the complete information of the moving vehicle.

Robustly matching information from a video camera and multiple Doppler radars is a prerequisite for information fusion in a Vidar traffic surveillance system. However, because of the different modalities of video and Doppler radar sensors, matching information from a video camera and Doppler radars is very difficult. Due to the special video-radar geometry introduced in Vidar, correctly matching between a video sequence and Doppler signals is possible. This invention describes a robust algorithm to match video signals and Doppler radar signals, and an algorithm to fuse the matched video and Doppler radar signals.

SUMMARY

A fusion algorithm for a Vidar traffic surveillance system may include the following steps: (1) deriving Doppler angles from a video sequence; (2) generating estimated Doppler signals from estimated Doppler angles; (3) matching estimated Doppler signals to the measured Doppler signals of two moving Doppler radars; (4) finding the best match between the estimated and measured Doppler signals; (5) forming a three-scan, range-Doppler geometry from the stationary Doppler radar and estimated Doppler angles; (6) matching video signals to stationary Doppler radar signals; (7) fusing the matched video and Doppler radar signals to generate moving vehicle velocity and range information.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may be understood by reference to the following description taken in conjunction with the accompanying drawings, in which, like reference numerals identify like elements, and in which:

FIG. 1 illustrates the fundamental of a Doppler radar for speed measuring;

FIG. 2 illustrates the functional flow chart of the fusion algorithm;

FIG. 3 illustrates the layout of the Vidar sensor suite;

FIG. 4 illustrates the sensing geometry of the Vidar traffic surveillance system; and

FIG. 5 illustrates a three-scan geometry for fusing video and Doppler radar signals.

DETAILED DESCRIPTION

The functional flow chart of the algorithm is shown in FIG. 2. In the following, we will provide detailed description of the invention.

BRIEF DESCRIPTION OF VIDAR SENSOR SUITE

FIG. 3 shows the layout of the Vidar sensor suite 201 where 202—a first moving Doppler radar, 203—a second moving Doppler radar, 204—a fixed or stationary Doppler radar, 205—a fixed or stationary video camera, 206—a data processing device, such as a computer, laptop, personal computer, PDA or other such device, and 207—a data recording device, such as a hard drive, a flash drive or other such device. FIG. 3 also indicates the sensing geometry where 208—the camera virtual image plane of the video camera 205, 212—a first moving Doppler radar motion ray, 213—a second moving Doppler radar motion ray, 214—a radar direction ray connecting the Vidar device apparatus 201 to a moving vehicle 215, 209—the intersection of the first Doppler radar motion ray 212 with the virtual image plane 208, 210—the intersection of the second Doppler radar motion ray 213 with the virtual image plane 208, 211—the intersection of the ray connecting the Vidar apparatus 201 and the moving vehicle 215 with the virtual image plane 208, and 215 a moving vehicle. The first and second Doppler radars 202, 203 in the Vidar apparatus 201 may be moved in such a way that the vehicle 215 is located in one side of both moving radar motion rays 212 and 213 with sufficiently large angles θ_τ1 and θ_τ2. The first and second Doppler radars 202, 203 in the Vidar apparatus 201 may be extended or retracted or moved side to side as illustrated in FIG. 3 by a motor (not shown) which may be a DC or stepper motor or other movement device and may be moved on sliding tracks (not shown). An optical encoder (not shown) may be mounted on the shaft of the motor, so the sliding speeds of the Doppler radars (υ_τ1 and υ_τ2 in FIG. 3) may be predetermined. The sliding track orientation angles (θ_τ1 and θ_τ2 in FIG. 3) may also be predetermined. Using a calibration method, the intersections 209 and 210 of the first and second motion rays 212, 213 with the virtual image planes 208 may be predetermined as well.

Derive Doppler Angles from a Video Sequence

The objective of this step (step 105 in FIG. 2) is to derive Doppler angle pairs, {θ_τ1k, θ_τ2k} as indicated in FIG. 4 where the subscript k is suppressed, from an image sequence. Assume the vehicle location on the image is q_k=[u_k, υ_k], as shown in FIG. 4. The vector from O to q_k may be defined as Oq_k=[u_k, υ_k, f] where f is the camera focal length, and the vectors from O to C₁ and C₂ may be given by: OC₁=[u_c1, υ_c1, f] and OC₂=[u_c2, υ_c2, f]. The Doppler angles may be estimated in step 105 by

$\begin{array}{cc}{\hat{\theta }}_{r_{1k}}={\cos }^{-1}\frac{{\underset{\_}{\mathrm{Oq}}}_k·{\underset{\_}{\mathrm{OC}}}_1}{{\underset{\_}{\mathrm{Oq}}}_k{\underset{\_}{\mathrm{OC}}}_1}& \left(2\right)\\ =\frac{u_ku_{c1}+v_kv_{c1}+f^2}{\sqrt{u_k^2+v_k^2+f^2}\sqrt{u_{c1}^2+v_{c1}^2+f^2}}\mathrm{and}& \left(3\right)\\ {\hat{\theta }}_{r_{2k}}={\cos }^{-1}\frac{{\underset{\_}{\mathrm{Oq}}}_k·{\underset{\_}{\mathrm{OC}}}_2}{{\underset{\_}{\mathrm{Oq}}}_k{\underset{\_}{\mathrm{OC}}}_2}& \left(4\right)\\ =\frac{u_ku_{c2}+v_kv_{c2}+f^2}{\sqrt{u_k^2+v_k^2+f^2}\sqrt{u_{c2}^2+v_{c2}^2+f^2}}.& \left(5\right)\end{array}$

Match Video Signals to Moving Radar Signals

Referring to FIG. 4, the Doppler angles may be related to the Doppler signals of the moving Doppler radars. For the first moving Doppler radar, the following holds

f _{D k} ¹ =K ₁υ_τ1k cos(θ_τ1k)+K₁υ_tk cos(φ_k)  (6)

where υ_tk cos(φ_k) may be provided by the stationary Doppler radar via

f _{D k} ³ =K ₃υ_tk cos(φ_k).  (7)

Since the motion of the first moving Doppler radar is known as

υ_τ1k=a₁ cos(ωt_k+φ₁),  (8)

we have

$\begin{array}{cc}{f}_{D_k}^1=K_1a_1\cos \left({\theta }_{r_{1k}}\right)\cos \left(\omega t_k+{\psi }_1\right)+\frac{K_1}{K_3}f_{D_k}^3& \left(9\right)\\ =A_{1k}\cos \left(\omega t_k+{\psi }_1\right)+B_{1k}f_{D_k}^3\mathrm{where}& \left(10\right)\\ A_{1k}=K_1a_1\cos \left({\theta }_{r_{1k}}\right)\mathrm{and}B_{1k}=\frac{K_1}{K_3}.& \left(11\right)\end{array}$

A similar equation may be derived for the second moving Doppler radar

$\begin{array}{cc}{f}_{D_k}^2=A_{2k}\cos \left(\omega t_k+{\psi }_2\right)+B_2f_{D_k}^3\mathrm{where}& \left(12\right)\\ A_{2k}=K_2a_2\cos \left({\theta }_{r_{2k}}\right)\mathrm{and}B_{2k}=\frac{K_2}{K_3}& \left(13\right)\end{array}$

and a₁, a₂, K₁, K₂, K₃, φ₁, and φ₂ are all known from calibration. Given Doppler angle estimates, {circumflex over (θ)}_τ1k and {circumflex over (θ)}_τ1k, we have

 _1k =K ₁ a ₁ cos({circumflex over (θ)}_τ1k) and Â_2k=K₂a₂ cos({circumflex over (θ)}_τ2k).  (14)

Within a predefined time window, cosine signals (Doppler signals) may be generated as

 _1k cos(ωt+φ₁) and Â_2k cos(ωt+φ₂),t_k−L≦t≦t_k  (15)

where L is the window length. It is straightforward to match estimated cosine signals to the measured Doppler signals in a single vehicle case using a least square method which is performed in step 106 of FIG. 2. For a multiple vehicles case, a multiple hypothesis test may be needed.

From the video camera, multiple pairs of Doppler angles are estimated:

$\left\{{\hat{\theta }}_{r_{1_k}}^i,{\hat{\theta }}_{r_{2_k}}^i\right\},i=1,\dots ,N$

where N is the number of vehicles, which in turn generate multiple cosine signals as

 _1k ⁱ cos(ωt+φ₁) and Â_2kⁱ cos(ωt+φ₂),i=1, . . . , N.

Using multiple hypothesis testing, the moving radar Doppler data set {D₁ⁱ, D₂ⁱ} corresponding to

$\left\{{\hat{\theta }}_{r_{1_k}}^i,{\hat{\theta }}_{r_{2_k}}^i\right\}$

may be identified (also in step 106 of FIG. 2), from which a set of new estimates may be derived:

${\stackrel{\widehat{\_}}{A}}_{1k}^i\cos \left(\omega t+{\varphi }_1\right)+{\stackrel{\widehat{\_}}{f}}_{1_{D_k}}^{3i}\mathrm{and}{\stackrel{\widehat{\_}}{A}}_{2k}^i\cos \left(\omega t+{\psi }_2\right)+{\stackrel{\widehat{\_}}{f}}_{2_{D_k}}^{3i},i=1,\dots ,N.$

Combing

${\stackrel{\widehat{\_}}{f}}_{1_{D_k}}^{3i},{\stackrel{\widehat{\_}}{f}}_{2_{D_k}}^{3i}\mathrm{and}{\stackrel{\widehat{\_}}{f}}_{D_k}^{3i}$

from three Doppler radars, a more accurate Doppler frequency of the ith vehicle may be determined.

Match Video Signals to Stationary Radar Signals

When two vehicles are close to each other, Doppler angles alone cannot set them apart. The stationary Doppler radar signals should provide additional information about their speeds. In general, it is relatively more accurate for a camera to measure an angle than derive a velocity. On the other hand, it is relatively more accurate for a Doppler radar to measure a velocity than derive an angle. The contribution of this invention is to robustly tie together the angle information from a video camera and the Doppler (velocity) information from a Doppler radar. In this invention, we will match angle rates from video signals to stationary Doppler radar signals via a unique three-scan geometry.

A three-scan geometry is shown in FIG. 5, where

$\begin{array}{cc}{\mathrm{\Delta \theta }}_k^i={\cos }^{-1}\frac{{\underset{\_}{\mathrm{Oq}}}_k^i·{\underset{\_}{\mathrm{Oq}}}_{k+1}^i}{{\underset{\_}{\mathrm{Oq}}}_k^i{\underset{\_}{\mathrm{Oq}}}_{k+1}^i}\mathrm{and}\text{}{\mathrm{\Delta \theta }}_{k+1}^i={\cos }^{-1}\frac{{\underset{\_}{\mathrm{Oq}}}_{k+1}^i·{\underset{\_}{\mathrm{Oq}}}_{k+2}^i}{{\underset{\_}{\mathrm{Oq}}}_{k+1}^i{\underset{\_}{\mathrm{Oq}}}_{k+2}^i}& \left(16\right)\end{array}$

where Oq_kⁱ=[u_kⁱ, τ_kⁱ, f] and Oq_k+1ⁱ=[u_k+1ⁱ, υ_k+1ⁱ, f] are the locations of the ith vehicle on the image plane. Assume a constant velocity model, i.e., υ_tkⁱ=υ_tk+1ⁱ=|{dot over (X)}_kⁱ|. Also assume that Doppler frequencies f³ⁱ_Dk and f³ⁱ_Dk+1 are provided by the stationary Doppler radar. We then have

$\begin{array}{cc}{\Delta }_k^i=T\frac{f_{D_k}^{3^i}}{K_3}\mathrm{and}{\Delta }_{k+1}^i=T\frac{f_{D_{k+1}}^{3^i}}{K_3}.& \left(17\right)\end{array}$

Using the cosine law, we have the constrained equation for the three-scan geometry (step 107 in FIG. 2) as

(a¹+Δ_kⁱ)²+(aⁱ)²−2(aⁱ+Δ_k^j)aⁱ cos(Δθ_kⁱ)=(aⁱ−Δ_k+1ⁱ)²+(aⁱ)²−2(aⁱ−Δ_k+1ⁱ)aⁱ cos(Δθ_k+1ⁱ).  (18)

Solving the following equation for

(aⁱ)²[2 cos(Δθ_k+1ⁱ)−2 cos(Δθ_kⁱ)]+aⁱ[2Δ_kⁱ+2Δ_k+1ⁱ−2Δ_kⁱ cos(Δθ_kⁱ)−2Δ_k+1ⁱ cos(Δθ_k+1ⁱ)]+(Δ_kⁱ)²−(Δ_k+1ⁱ)²=0  (19)

we may find the range from the Vidar device to the vehicle which is performed in step 108 of FIG. 2. Similarly,

$\begin{array}{cc}{b}^i=a^i-T\frac{f_{D_{k+1}}^{3^i}}{K_3}\mathrm{and}c^i=a^i+T\frac{f_{D_k}^{3^i}}{K_3}.& \left(20\right)\end{array}$

The criterion for matching video signals to stationary Doppler radar signals becomes validating the following equation. Given an arbitrary Doppler signal pair from the stationary Doppler radar, say f^3j_Dk and f^3j_Dk+1, if it matches the video signals, the following equation should be satisfied

(aⁱ)²[2 cos(Δθ_k+1ⁱ)−2 cos(Δθ_kⁱ)]+aⁱ[2Δ_k^j+2Δ_k+1^j−2Δ_k^j cos(Δθ_kⁱ)−2Δ_k+1^j cos(Δθ_k+1ⁱ)]Δ(Δ_k^j)²−(Δ_k+1^j)²=0.  (21)

Fusion of Video and Doppler Signals

Once the matched video and Doppler radar signals are found, they are fed into a stochastic model for fusion, which is performed in step 109 of FIG. 2.

Assume the kinematics of the ith vehicle satisfy a stochastic constant velocity (CV) model

$\begin{array}{cc}{\left[\begin{array}{c}\underset{\_}{X}\\ \underset{\_}{\stackrel{.}{X}}\end{array}\right]}_{k+1}^i={\left[\begin{array}{cc}I& \mathrm{IT}\\ 0& I\end{array}\right]\left[\begin{array}{c}\underset{\_}{X}\\ \underset{\_}{\stackrel{.}{X}}\end{array}\right]}_k^i+\left[\begin{array}{c}\frac{1}{2}{\mathrm{IT}}^2\\ I\end{array}\right]{\underset{\_}{\rho }}_k^i,{\underset{\_}{\rho }}_k^i~N\left(\underset{\_}{0},Q_k^i\right)& \left(22\right)\end{array}$

where X_kⁱ=[xⁱ, y¹, zⁱ]_k is the ith vehicle's 3D coordinate. The positional measurement equation may be

$\begin{array}{cc}0=\left(\frac{{\Delta }^if_{D_k}^{13}}{{\Delta }^if_{D_k}^{23}}v_{r_{2\mathrm{xk}}}-v_{r_{1\mathrm{xk}}}\right)x_k^i+\left(\frac{{\Delta }^if_{D_k}^{13}}{{\Delta }^if_{D_k}^{23}}v_{r_{2\mathrm{yk}}}-v_{r_{1\mathrm{yk}}}\right)y_k^i+\left(\frac{{\Delta }^if_{D_k}^{13}}{{\Delta }^if_{D_k}^{23}}v_{r_{2\mathrm{zk}}}-v_{r_{1\mathrm{zk}}}\right)z_k^i& \left(23\right)\\ =\left[\begin{array}{c}\frac{{\Delta }^if_{D_k}^{13}}{{\Delta }^if_{D_k}^{23}}v_{r_{2\mathrm{xk}}}-v_{r_{1\mathrm{xk}}},\\ \frac{{\Delta }^if_{D_k}^{13}}{{\Delta }^if_{D_k}^{23}}v_{r_{2\mathrm{xk}}}-v_{r_{1\mathrm{xk}}},\frac{{\Delta }^if_{D_k}^{13}}{{\Delta }^if_{D_k}^{23}}v_{r_{2\mathrm{xk}}}-v_{r_{1\mathrm{xk}}}\end{array}\right]{\underset{\_}{X}}_k^i+{\underset{\_}{\gamma }}_{\mathrm{xk}}^i& \left(24\right)\\ ={\left({\underset{\_}{v}}_{r_{{12}_k}}^i\right)}^T{\underset{\_}{X}}_k^i+{\underset{\_}{\gamma }}_{x_k}^i{\underset{\_}{\gamma }}_{x_k}^i~N\left(\underset{\_}{0},R_{x_k}^i\right).& \left(25\right)\end{array}$

The velocity measurement equation may be established as

$\begin{array}{cc}{f}_{D_k}^{3^i}={\stackrel{\_}{u}}_k{\stackrel{.}{x}}_k^i+{\stackrel{\_}{v}}_k{\stackrel{.}{y}}_k^i+\stackrel{\_}{f}{\stackrel{.}{z}}_k^i+{\underset{\_}{\gamma }}_{{\stackrel{.}{x}}_k}^i& \left(26\right)\\ =\left[{\stackrel{\_}{u}}_k,{\stackrel{\_}{v}}_k,\stackrel{\_}{f}\right]{\underset{\_}{\stackrel{.}{X}}}_k^i+{\underset{\_}{\gamma }}_{{\stackrel{.}{x}}_k}^i& \left(27\right)\\ ={\stackrel{\_}{\underset{\_}{\mathrm{oq}}}}_k^T{\stackrel{.}{\underset{\_}{X}}}_k^i+{\underset{\_}{\gamma }}_{x_k}^i\mathrm{where}& \left(28\right)\\ {\stackrel{\_}{u}}_k=K_3\frac{-u_k}{\sqrt{u_k^2+v_k^2+f^2}},{\stackrel{\_}{v}}_k=K_3\frac{-v_k}{\sqrt{u_k^2+v_k^2+f^2}}\mathrm{and}\stackrel{\_}{f}=K_3\frac{-f}{\sqrt{u_k^2+v_k^2+f^2}}.& \left(29\right)\end{array}$

Eqs. (22), (25) and (28) form a stochastic system for vehicle information fusion and an extended Kalman filter may be used to estimate the position and velocity of the vehicle. For a CV model, minimum two scans may be needed and for a constant acceleration (CA) model minimum three scans may be needed to converge.

Claims

1. A method of fusing video signals and Doppler radar signals for estimating moving vehicle velocity and range information, comprising the steps of: a. matching said video signals to said Doppler radar signals; andb. fusing the matched said video signals and said radar signals to derive said velocity and range information of said vehicle.

2. A method of fusing video signals and Doppler radar signals as recited in claim 1, wherein the method estimates Doppler angles from said video signals.

3. A method of fusing video signals and Doppler radar signals as recited in claim 1, wherein the method estimates Doppler signals from said Doppler angles.

4. A method of fusing video signals and Doppler radar signals as recited in claim 1, wherein the method matches said Doppler signals to measured Doppler signals from moving Doppler radars.

5. A method of fusing video signals and Doppler radar signals as recited in claim 1, wherein the method forms a multiple scan geometry from said video signals and said Doppler radar signals.

6. A method of fusing video signals and Doppler radar signals as recited in claim 1, wherein the method matches said video signals to measured Doppler signals from stationary Doppler radar.

7. A method of fusing video signals and Doppler radar signals as recited in claim 1, wherein the method forms a stochastic model for said video signals and said Doppler radar signals.

8. A method of fusing video signals and Doppler radar signals as recited in claim 1, wherein the method estimates said the velocity and range information from said video signals and said Doppler radar signals using said stochastic model.