Object detection and tracking in video sequences may be important in applications such as content-based retrieval, natural human-computer interfaces, object-based video compression, and video surveillance. Classifiers which provide early rejection of non-object patterns may be used for object detection and tracking. In one approach, a number of classifiers may be arranged in a cascade. An input pattern may be evaluated by a first classifier trained to remove a certain percentage of non-object patterns while keeping all object patterns. Second and subsequent stage classifiers may be trained in the same manner. After N stages, the false alarm rate may drop very close to zero while maintaining a high hit rate.
From stage to stage a more complex classifier may be needed to achieve the goal. While the cascade approach has been successfully validated for frontal upright face detection, which tend to be very regular and similar, cascade classifiers may have difficulty handling visually more complex and diverse object classes such as multi-view faces and mouths.
A node in the tree may have depending nodes, which are lower in the hierarchy. The node may be referred to as a parent node, and the nodes depending from the parent node may be referred to as child nodes. The parent node may be a child node of another node higher in the tree-structure.
The tree classifier includes a root node 110 at the top of the tree. The root node distinguishes itself from other nodes by not having a parent. There may be splits 115 in the branches of the tree, where a parent has two or more child nodes. The different child nodes at a split may be specialized to classify different features of the input.
The classifiers may be used to filter input images to identify a specified object, e.g., a face. The classifiers may be boosted classifiers trained to have a high hit rate (e.g., 99.9%) and a moderate false positive (false alarm) rate (e.g., 50%). A classifier may be able to identify specified objects with extremely high accuracy and identify non-pattern images, e.g., images not including the specified object, about half of the time.
The classifiers may be trained using a boosting algorithm such as AdaBoost. Psuedocode 200 for AdaBoost is given in
Initially, all weights may be set equally, but on each round, the weights of incorrectly classified examples may be increased so that the weak learner is forced to focus on the hard examples in the training set. The weak learner's job may try to find a weak hypothesis ht: X→{−1, +1} appropriate for the distribution Dt. The goodness of a weak hypothesis is measured by its error:
The error may be measured with respect to the distribution Dt on which the weak learner was trained. In practice, the weak learner may be an algorithm that can use the weights Dt on the training examples. Alternatively, a subset of the training examples may be sampled according to Dt, and the unweighted, resampled examples can be used to train the weak learner.
Once the weak hypothesis ht has been received, AdaBoost may choose a parameter αt, which measures the importance that is assigned to ht. Generally, αt≧0 if εt≦½, and αt gets larger as εt gets smaller.
The distribution Dt may be updated using the update rule 205 shown in
The classifiers may be trained using a set of positive training samples (including the specified object) and a set of negative training samples (not including the specified object). The tree may be grown by training the classifiers using a recursive algorithm such that the tree will grow until a desired depth is achieved. The desired depth is either pre-specified or adaptively chosen based on the desired combination of hit and false alarm rate.
An exemplary algorithm for growing and training a tree classifier is described in the flowchart 300 in
At each node, the negative training samples may be specified or filtered by the parent node (block 307). A monolithic strong classifier at node S1 may be trained with positive (SPOS) and negative samples (block 310).
At each node level, a determination is made whether to keep the monolithic classifier or split the tree into different branches, each branch including a node with a classifier trained to filter a different subclass of the object of interest. The splitting criterion may be based on the minimal number of features, and hence the lowest computational complexity, needed to achieve a given training hit and false alarm rate ignoring the overall detection performance.
After the monolithic classifier is trained, the BestClassifierSet variable is set to identify the monolithic classifier (S1), and the BestNoOfFeatures is set to the number of features used by the monolithic classifier (block 312). Next, the computational complexity of two or more sets of specialized classifiers is determined.
A k-means clustering algorithm may be utilized to divide the positive samples into k subsets (block 315). The k positive subsets and the negative samples may be used to train k strong classifiers (block 320). If the total number of features used by these k classifiers (O(Sk1)+. . . +O(Skk)) is less than the total number of features used in the monolithic classifier (O(S1)), the k strong classifiers are considered to be computational more efficient than the monolithic classifier. If so, BestClassifierSet is set to identify this set of k specialized classifiers (Sk1, . . . , Skk) and BestNoOfFeatures is set to the total number of features used by the specialized classifiers (block 325). This process may be repeated up to Kmax.
The variable kbest is updated throughout the process. If kbest is “1”, then the monolithic classifier is selected for the node level, otherwise the set of specialized classifiers which uses the least total number of features is selected (block 330). The process is repeated in each of the branches of the split (block 335). The training process 303 may be recursively applied until a given target depth (Smax) of the tree is reached (block 340).
The system 600 may include a finite state machine with two states: detection and tracking. The system may begin with the detection state in which a face detector 605 followed by a tree classifier 610 for mouth detection is utilized to locate the face of a speaker as well as his/her mouth location. If the detections are successful in several successive frames, the state machine may enter the tracking state where only the tree classifier 610 is employed to detect the mouth in the region around the location predicted from previous detection or tracking results. If any detection failure occurs in the tracking state, the state machine may switch back to the detection state to recapture the object. The system 600 may also include a post-processing module 615 to smooth the raw mouth locations and conceal accidental detection failures.
In an embodiment, the face detector 605 may be a single cascade classifier, which may be powerful enough for detection of full, upright faces. The search area for the mouth with the tree classifier 610 may be reduced to the lower region of the detected face. To accommodate scale variations, a multi-scale search may be utilized within a constrained range estimated according to the face detection result.
In the tracking state, only the tree classifier 610 may be used to detect the mouth. A linear Kalman filter (LKF) 620 may be employed to predict the center of the search region in the next frame and correct the result in the current frame. The LKF 620 may address the general problem of estimating the state X of a discrete-time process that is governed by a linear stochastic difference equation
Xk+1=AXk+wk
with a measurement Z, which is
Zk=HXk+vk
The random variables wk and vk are assumed to be independent of each other and have normal probability distributions. In an embodiment, a Newton dynamics model may be employed, i.e.,
where Δt=0.4 based on a frame rate of 25 Hz. In practice, the search region in the next frame t+1 may centered around (xc, yc) obtained from the time update with a width and height of 40% larger than the detected mouth at time t.
The post-processing module 615 may be used to refine the trajectory of mouth in three phases. A linear interpolation may be employed to fill in the gaps in trajectory caused by detection failures. A median filter may then be used to eliminate incorrect detections under the assumption that outliers only occur individually. A Gaussian filter may then be used to suppress the jitter in the trajectory.
For training, 1,050 mouth images were extracted from the sequences of the “Client” subset of the XM2FDB database. These sample images were manually classified into two hundred-fifty images of speakers with beard and eight hundred without beard. By randomly mirroring, rotating, and re-scaling these images, six thousand positive training samples of speakers with beard and nine thousand without beard were generated. Negative training examples were randomly extracted from a set of approximately 16,500 face-free and mouth-free images.
Three mouth tracking systems were built and compared.
The third system was based on a tree classifier, such as that shown in
The three systems were tested on the “Imposter” subset of the XM2FDB database with 759 sequences recorded from 95 speakers using an Intel® Pentium® 4 computer with 1.7 GHz and 1 GB RAM.
Table 1 lists the accuracy and the average execution time per frame obtained by each system, together with the results obtained by the support vector machine (SVM) based system. The results indicate that the tree classifier is superior to the cascade classifier with respect to accuracy, while having the shortest execution time of all three systems. Only the detection accuracy for multiple specialized cascade classifiers was slightly better but at a significantly higher computational cost, e.g., about 45% more demanding. In addition, compared with the SVM based system, the tree classifier based system was about sixty-six and fifteen times faster in detection and tracking, respectively, while preserving at least the same accuracy.
A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. For example, blocks in the flowcharts may be skipped or performed out of order and still produce desirable results. Accordingly, other embodiments are within the scope of the following claims.
This application claims benefit of the priority of the U.S. Provisional Application No. 60/456,033 filed Mar. 17, 2003 and entitled “A Detector Tree of Boosted Classifiers for Real-Time Object Detection and Tracking.”
Number | Name | Date | Kind |
---|---|---|---|
5078952 | Gozani et al. | Jan 1992 | A |
5621861 | Hayashi et al. | Apr 1997 | A |
6445409 | Ito et al. | Sep 2002 | B1 |
6456993 | Freund | Sep 2002 | B1 |
6546379 | Hong et al. | Apr 2003 | B1 |
6556983 | Altschuler et al. | Apr 2003 | B1 |
6662170 | Dom et al. | Dec 2003 | B1 |
6801662 | Owechko et al. | Oct 2004 | B1 |
6823323 | Forman et al. | Nov 2004 | B2 |
6859455 | Yazdani et al. | Feb 2005 | B1 |
20010037324 | Agrawal et al. | Nov 2001 | A1 |
20030018475 | Basu et al. | Jan 2003 | A1 |
20030123456 | Denz et al. | Jul 2003 | A1 |
20030176931 | Pednault et al. | Sep 2003 | A1 |
20060034484 | Bahlmann et al. | Feb 2006 | A1 |
Number | Date | Country | |
---|---|---|---|
20040186816 A1 | Sep 2004 | US |
Number | Date | Country | |
---|---|---|---|
60456033 | Mar 2003 | US |