The present invention relates to the processing of video image sequences, and more particularly to a semi-automatic method for Kalman tracking of color objects within the video image sequence.
With the advent of digital television and the resulting large bandwidth requirements for baseband video signals, compression techniques become ever more important. The currently accepted standard for television compression that provides the most compression while still resulting in acceptable decoded images is the MPEG2 standard. This standard compresses an image using one of three types of compressed frames—an Independently compressed frame, a predictively compressed frame and a bi-directional predictively compressed frame. This standard operates on the images as a whole.
However the content of images may be composed of several objects, such as tennis players and a ball, in front of a background, such as spectators. It is posited that if the objects (tennis players and ball) are separated out from the background (spectators), then the objects may be compressed separately for each frame, but the background only needs to be compressed once since it is relatively static. To this effect many techniques have been proposed for separating objects from the background, as indicated in the recently published proposed MPEG7 standard.
Just separating the objects is not sufficient—the objects need to be tracked throughout a given sequence of images that make up a scene. What is desired is a method for tracking objects within a video image sequence.
Accordingly the present invention provides Kalman tracking of color objects within a video image sequence. Objects are separated on the basis of color using a color separator, and a user identifies an object or objects of interest. The object(s) are tracked using a Kalman prediction: algorithm to predict the location of the centroid of the object(s) in successive frames, with the location being subsequently measured using a mass density function and then filtered to provide a smooth value for centroid location and velocity. If one of the assumptions for the tracking algorithm fails, then an error recovery scheme is used based upon the assumption that failed, or the user is asked to re-initialize in the current frame.
The objects, advantages and other novel features of the present invention are apparent from the following detailed description when read in conjunction with the appended claims and attached drawing.
In performing semi-automatic tracking of colored objects in a given video image sequence, a user indicates in one or more key frames a set of one or more colored objects. The user also indicates other regions of significant size and different colors in the video image sequence. The objects are separated based upon color, and a tracking algorithm then tracks the movements of the indicated objects over time through the video image sequence. This tracking is achieved by associating a Kalman tracking model to each object. The basic algorithm is shown in FIG. 1.
An input video Image sequence is input to a color segmentation algorithm, such as that described in co-pending U.S. patent application Ser. No. 09/270,233 filed Mar. 15, 1999 by Anil Murching et al entitled “Histogram-Based Segmentation of Objects from a Video Signal Via Color Moments”. This algorithm uses a hierarchical approach using color moment vectors. The color segmentation algorithm segments the images in the input video image sequence Into regions/classes of uniform color properties. Then a Kalman tracking algorithm is applied to each of the segmented objects to produce object “tracks” from one frame to the next of the video image sequence.
As shown in
Kalman tracking applies a Newtonian motion model to the centroide of the objects of interest. As an example, the objective is to track object #K in
where (xk, Yk) are the location coordinates of the centroid for object #K, and (vxk, vyk) are the velocity components of object #K. The Newtonian motion model for all objects assumes that acceleration is a white-noise process. This motion model is well known in the art and may be found in the literature on Kalman filtering.
With this motion model a state-transition equation becomes:
ζk[n+1]=F ζk[n]+Gηks[n]
where F and G are vector constants and ηks[n] is a stationary, independent, white noise vector with mean: E{ηks[n]}=0.
A correlation vector bandwidth Rks=E{ηks[n]ηk[m]T}=|σxk2, 0; 0, σyk2|. The noise variances are estimated from the input video sequence.
Through tracking, the position of the centroid of the object #K in the next frame is measured, so:
Ψk[n+1]=Hζk[n+1]+ηko[n+1]
where ηk0[n] is the stationary, independent, observation noise vector with means equal to 0, and H is a vector constant. Again there is a correlation vector Rko with noise variances that are estimated.
In steady state tracking the object #K has been tracked to frame In and its position and velocity are known. From this point the first step is Kalman prediction. To locate the object #K in frame In+1
(Predicted)ζk′[n+1|n]=F(filtered)ζk″[n|n]
The first two entries in ζk′[n+1|n] give the predicted position of the centroid in frame In+1, Segment P×Q blocks of In+1 into the many colors and identify all the blocks that belong to color model #A—object #K has this color. Then starting from the predicted position, extract a connected set of P×Q blocks that all belong to the color model #A.
The set of connected blocks identified in the first step constitute the desired detection/tracking of the object of interest in frame In+1. The second step is to measure the centroid position, performed by:
105 xk[n+1]=ΣXkYk/ΣYk
105 yk[n+1]=ΣykYk/ΣYk
where Y is luminance data in frame In+1. Calculate the centroid position by using luminance as a “mass density” function. This improves the robustness of the tracking algorithm. Either of the color components may also be used as mass density functions.
Both the measurement and prediction steps are susceptible to noise, so a third step is to filter/smooth the state information. The familiar Kalman filtering equations are used:
ζk″[n+1|n]=ζk′[n+1|n]+Σk[n+1|n]HT(HΣk[n+1|n]HT+Rk0)−1*(Ψk[n+1]−Hζk′[n+1|n])
Σk[n+1|n+1]=Σk[n+1|n]−Σk[n+1|n]HT(HΣk[n+1|n]HT+Rk0)−1*HTΣk[n+1|n]
Σk[n+1|n]=F Σk[n|n]FT+G RksGT
From these equations the filtered/smoothed position and velocity of the centroid of object #K in frame In+1 are obtained. The same process is repeated for each succeeding frame.
For the initialization of the process the position of the centroid in frame I0, ζk″[0|0], is determined. The user “clicks” near the visually estimated geometric center of the object #K, and that point serves as the initial position. The initial velocity is set to zero. Then values for Rks, Rko and Σk[0|0] are determined experimentally and used to determine the centroid position. One such set is
Although the above equations ostensibly give the predicted position of the centroid of object #K in the new frame In+1, it is possible that these coordinates lie outside the image field of view. This is easily detected and is an indication to the user that the object of interest has exited the field of view, which is a perceptually significant event. In the algorithm above use the last known “good” position and attempt to delete the object In frame In+1 at that location. If successful, the algorithm continues. Otherwise the algorithm prompts the user to either (a) verify that the object has left the field of view, and hence stop tracking it, or (b) re-initialize at frame In+1 because the tracker model has broken down.
Sometimes, due to the geometric shape of the object or due to sudden changes in acceleration, the Kalman prediction points to a centroid location that is outside the boundary of the object #K, as shown in FIG. 5. This situation arises when the P×Q block that contains the predicted centroid position is classified by the color segmentation algorithm as belonging to a class other than color model #A. Again this situation is easily detected. To recover from this, search around a local neighborhood of the predicted centroid position. As shown in
The Kalman tracking algorithm is based upon the following assumptions: (I) objects of interest have regular shapes, i.e., cannot track spokes of a bicycle wheel as they are too “thin”; (ii) objects of interest have smooth color, i.e., no stripes or strange patterns; (iii) objects are moving “regularly”, i.e., not Brownian motion of gas molecules; and (iv) objects do not occlude each other. When both the out of field of view and outside object boundary error recovery schemes described above fail, then the Kalman tracker is said to have failed. At this point one of the above assumptions has failed. The options at this point are (I) detect all connected regions in frame In+1, that have color model #A, sort according to size/shape and try to locate the desired object #K among them, or (II) ask the user for help, i.e., prompt the user to re-initialize the tracking algorithm at frame In+1.
For option (I) the color segmentor outputs a segmentation map Sn+1. See FIG. 8. Each sample in Sn+1 represents a spatially corresponding P×Q block of frame In+1. The value of the sample “n” is {0, 1, . . . , Nu}, where {1, . . . , Nu} are the color models provided to the color segmentor and {0} represents “garbage”. The segmentation map is converted to a binary alpha map αn+1 by tagging all samples as Sn+1 that have the same color model as object #K. Thus pixels in αn+1 have a value 255 if their corresponding P×Q block in In+1 has the same color as object #K, and have a value of 0 otherwise. The alpha map is fed to a “grow connections” algorithm (
Thus the present invention provides for Kalman tracking of color objects in an input video image sequence by segmenting the image in the initial frame into a group of objects according to color, determining the position of the centroid of an object of interest and tracking the object through successive frames; and also provides some simple error recovery schemes if the object moves out of the field of view, the predicted centroid falls outside the boundaries of the object or the algorithm breaks down.
Number | Name | Date | Kind |
---|---|---|---|
5051751 | Gray | Sep 1991 | A |
5280530 | Trew et al. | Jan 1994 | A |
5293574 | Roehm et al. | Mar 1994 | A |
5469374 | Graham et al. | Nov 1995 | A |
5719951 | Shackleton et al. | Feb 1998 | A |
5768412 | Mitsuyama et al. | Jun 1998 | A |
5852792 | Nielson | Dec 1998 | A |
5933524 | Schuster et al. | Aug 1999 | A |
5960097 | Pfeiffer et al. | Sep 1999 | A |
6044165 | Perona et al. | Mar 2000 | A |
6148092 | Qian | Nov 2000 | A |
6167167 | Matsugu et al. | Dec 2000 | A |
6188776 | Covell et al. | Feb 2001 | B1 |
6198833 | Rangan et al. | Mar 2001 | B1 |
6246803 | Gauch | Jun 2001 | B1 |
6278460 | Myers et al. | Aug 2001 | B1 |
6381363 | Murching et al. | Apr 2002 | B1 |
6404900 | Qian et al. | Jun 2002 | B1 |
6526169 | Murching et al. | Feb 2003 | B1 |
6711278 | Gu et al. | Mar 2004 | B1 |