Patent Application
20090296989
This invention relates generally to methods for automatically detecting and tracking multiple objects and more particularly to methods for automatically detection and tracking pedestrians in crowds.
As is known in the art, multi-object detection and tracking, such as detection and tracking of pedestrians in a crowd, over extended periods of time is a challenging problem that has attracted significant attention in the context of video surveillance systems. The main technical challenges are in building practical and scalable systems whose computational complexity scales with the complexities in tracking due to the number of people, inter-object occlusions, appearance and shape/size similarities in persons (including similar geometry, clothing, and homogeneity in person facial appearance). See for example. U.S. Pat. No. 7,006,950, inventors Greiffenhagen, et al. issued Feb. 28, 2006 entitled “Statistical modeling and performance characterization of a real-time dual camera surveillance system”.
In accordance with the invention, a method is provided for automatic detection and tracking of people in a crowd from single or multiple video streams. In one embodiment, the method uses an electronic computing apparatus to process video signals using human-like reasoning approach involving staged computation of approximate guessing (indexing) and refinement of hypotheses about the number of persons, their locations and direction of motion. A final optimization stage refines the hypotheses generated to compute a quantitative estimate of the number of persons, their locations their tracks, along with specific attributes of the persons (including size, height, dynamic state or posture in 3-dimensions).
In one embodiment, a method is provided for automatically detecting and tracking each one of a plurality of people in a scene. The method includes: acquiring video frames from a video camera: extracting discriminative features distinguishing foreground from background in the acquired video frames; detecting changes in the extracted features to produce a change map; using the change map to generate a hypothesis for estimating an approximate number of people along with locations of the people uncertainties therein; and using the estimates, initializing tracking each one of the people to obtain partial tracks of each one of the people and using partial tracks to refine the estimate of the number of people, their individual locations and uncertainties.
In one embodiment, the generation of the hypothesis includes: (a) using the change map and/or the video frames to identify smaller hypotheses regions in the scene for further examination; (b) computing a summed-weighted score of occupancy of the identified smaller hypotheses regions; (c) using the score of occupancy to guess the number of people; (d) using contours for a plurality of identified smaller hypotheses regions to estimate another guess of the number of people and their locations for each smaller hypotheses regions; and (e) using an appearance based classifier that uses a plurality of appearance features integrated with a rule-based reasoning method to estimate number of people and their locations.
In one embodiment, a method is provided for automatically detecting and tracking of each one of a plurality of people in a scene. The method includes: obtaining video data of the objects in the scene using a video system; processing the data in computer apparatus using a indexing process to generate estimate hypotheses of the location and attributes of the objects within the scene; using person track estimates from past frames to predict a likely locations of persons; using the estimated hypotheses as input to construct space-time features used to detect self and mutual occlusion hypotheses: using the occlusion hypotheses to initialize a plurality of mean-shift trackers whose histogram feature representation is chosen adaptively to discriminate between the given person and the rest of the scene and whose kernels are adaptively set according to the occlusion hypotheses and posture predictions; obtaining a plurality of partial tracks using the plurality of mean-shift trackers that are robust under occlusions; and fusing the partial tracks along with person location predictions to obtain a refined estimate of number of people, their locations and postures.
In one embodiment, a method is provided for automatically detecting and tracking objects in a scene. The method includes obtaining video data of the objects in the scene using a video system. The data is processed in computer apparatus using a fast-indexing process to generate estimate hypotheses of the location and attributes of the objects within the scene. The estimate hypothesis is refined to generate statistical models of appearance and geometry of the objects being tracked. The generated models are then used for discriminative tracking using context driven adaptive detection and tracking processing.
In one embodiment, the uncertainty estimates are used to derive predictive distributions of expected locations of persons and enable the derivation of an occlusion hypothesis that is fed back for adaptive decisions on feature.
In one embodiment, the method: acquires video frames from a video camera; extracts discriminative features, e.g., histograms computed from the most discriminative color spaces, from the video frames; detects changes in the extracted features using background subtraction to produce a change map; using the change map to use a hypothesis to estimate of an approximate number of people along with uncertainty in user specified locations; and using the estimate, track people and update the hypotheses for a refinement of the estimation of people count and location.
The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.
Like reference symbols in the various drawings indicate like elements.
Referring now to
Referring to
The computer platform 201 also includes an operating system and micro instruction code. The various processes and functions described herein may either be part of the micro instruction code or part of the application program (or a combination thereof) which is executed via the operating system. In addition, various other peripheral devices may be connected to the computer platform such as an additional data storage device and a printing device.
It is to be further understood that, because some of the constituent system components and method steps depicted in the accompanying figures may be implemented in software, the actual connections between the system components (or the process steps) may differ depending upon the manner in which the present invention is programmed. Given the teachings of the present invention provided herein, one of ordinary skill in the related art will be able to contemplate these and similar implementations or configurations of the present invention.
Referring now to
The architecture is illustrated in
Referring again to
The fast indexing step is illustrated below in
Fast indexing is an efficient algorithm that quickly calculates the number of people in a region of interest (ROI) in an input video frame. In the example in
Referring to
Next, the process fuses all initial estimates using uncertainty weighted average (Step 725). Note that less certain guesses are weighted less and the update weights are computed a priori. The weights are stored as a look up table conditioning on the imaging conditions such as foreground/background color contrast and resolution. Next, the process adaptively adjusts kernel sizes for space-time slices (Step 730). Next the process detects the occurrence of occlusion among people and due to structures in the scene (Step 735), by using inference on the space-time slices. Next, the process initializes a mean-shift tracker, determining the number of kernels and kernel size and location based on walking direction of a person (a pedestrian walking parallel to the camera will need a more detailed model (thus more kernels) than a pedestrian walking toward or away from the camera due to larger range of swaying arms/legs apparent in the video frames), amount of occlusion (legs of a person should not be used for tracking if they are occluded) (Step 740). Next, the process utilizes the mean-shift tracker to track people, using the set of kernels determined in Step 740 and features determined in Step 700, and recover partial tracks (Step 745). Next, the process fuses multiple hypotheses for a refined estimation of number of people, location, 3D posture, et al. (Step 750), using fused partial tracks estimated by the fast indexing schemes (Step 725), partial tracks provided by the mean-shift tracker, and partial tracks predicted by the motion prediction process (Step 760), Next the process updates a global state, number of people, locations, and postures, et al., for past estimations (Step 755). Next, the process predicts object state using past estimations and velocity estimation (Step 760)
It is noted that the change detection map is used to derive the approximate number of people along with uncertainty in user specified zones (see U.S. Pat. No. 7,457,436 patent by Paragios et al., “Real-time crowd density estimation from video”) (Step 705). Further, the approximate number of people and their locations in (Step 710) can be further estimated through the procedure outlined in criterion [see L. Dong, V. Parameswaran, V. Ramesh, I. Zoghlami: Fast Crowd Segmentation Using Shape Indexing, ICCV 2007] and [United States Patent, Dong et al. Fast Crowd Segmentation Using Shape Indexing, U.S. Patent Application Publication No. 2009/0034793, assigned to the same assignee as the present invention, the entire subject matter thereof being incorporated herein by reference] if the estimated number of people along with uncertainty in a given zone is less than 6. Other alternative schemes may be used to estimate the number of people and their locations (for example: an appearance based head detection scheme may give a rough order of magnitude of number of people along with uncertainty in a given region). ( Step 720).
Reference is also made to U. S. Provisional Patent Application Serial No. 61/107,707 filed Oct. 23, 2008 entitled “A General-View, Self-Calibrating Crowd-Size Estimator” inventors Vasudev Parameswaran and Visvanathan Ramesh, the entire subject matter thereof being incorporated herein by reference. As described therein:
It is assumed that (1) the scene is static, (2) the camera's height above the ground is greater than the maximum height of a human, (3) a background maintenance and change detection module supplies a binary change detection image C as input to our crowd size estimator, (4) the size of humans as a function of pixel position (i.e. partial scene geometry) is provided. These are reasonable assumptions in a typical surveillance scenario. For the purposes of exposition, we make two simplifications: (1) that humans can be modeled as rectangles (the method can be extended in a straightforward manner to work with more detailed models of humans), (2) vertical lines in the 3D world project to vertical lines in the image (the general case can be derived in a similar manner). Denote the width by w(y) and height by h(y). We work with the y-axis pointing downwards as in an image. Let the width and height of the image be W and H respectively. We propose that the crowd size S can be expressed as a weighted area of C:
Here S is a cumulative guess of the number of people in a scene, utilizing a weighted sum of partial evidences provided by each foreground pixel in the change detection image (change map) C. S is subsequently called a score. N. Paragios and V. Ramesh. “A MRF-based approach for real-time subway monitoring.” Proc. IEEE CVPR, 2001 choose θ(i)=1/(w(i)h(i)). Although this is approximately position invariant and a reasonable weight function, in this work, we derive a weight function that incorporates position invariance explicitly. Assume that there is one person in the scene such that the rectangle modeling the person has its top left corner at (.x,y). In this case we seek a function θ(.) such that:
Here f(y) is the y coordinate of the person's foot f(y)−y=h(f(y)). Let the y coordinate of the horizon be yv which can be obtained by solving h(y)=0. The smallest y coordinate for a person's head we consider is given b y:
y
0=max(0, yv+ε) (3)
Let ymax be the maximal head position above which the feet are below the image. Equation (2) applies for positions y0≦y≦ymax. For y>ymax the weighted sum is adjusted to the fraction of the visible height of the person. We thus have (H=y0+1) equations in as many unknowns and the linear system of equations can be solved to yield θ(.). Although this is in principle correct, the equations do not enforce smoothness of θ(y) and hence the resulting weight function is not typically smooth. We could remedy this problem using regularization (e.g. Tikhonov regularization, or ridge regularization) but we found the following method quite effective in our case. We first define the cumulative sum function
Hence, equation (2) can be written as
This is a recurrence relation in F. We arbitrarily set F(H)=1 and obtain F at sparse locations: y={H, H−h(H), . . . , y0}. Next we interpolate F using a cubic spline and finally obtain as follows:
θ(y)=F(y)−F(y−1) (6)
Denote the true number of people in a scene as N. For N>1, S obtained using the weight function above will exactly equal N if the people do not occlude each other. However, if there are occlusions, S will not be unique, but can be described by a probability distribution function (PDF) P(S|N). The entropy of P(S|N) will depend upon the camera angle and be lowest for a top-down view. We estimate this PDF by simulating N humans in various configurations and degrees of overlap and calculating S using the resulting binary image and the scene-specific weight function we calculated in section 3. Note that this process allows the inclusion of more detailed human body models and specific perturbation processes to the binary image. For example, if the sensor noise characteristics are available we could incorporate them into the simulation process. Similarly, structured perturbations such as shadows and reflections can also be introduced into the simulation process, allowing the scaling up to more complex viewing conditions. The essential output of the simulation process is an estimate. Let the maximum number of people in the scene be Nmax. The simulation process produces P(S|i),1≦i≦Nmax. At runtime, Bayes rule is used to find the posterior:
We further reduce the computational burden at run time by storing the posterior (rather than the likelihood) in a look up table. Hence, all that needs to be done at run-time is the calculation of S and a lookup into the table to obtain P(N|S). We also approximate P(N|S) as a normal distribution and simply store the mean and standard deviation in the table. This has been found to work quite well in practice.
These guesses of number of people can be combined in a sequential (the application of the algorithms may follow one another depending on the accuracy of the estimation known a priori) or a parallel fusion scheme that weight the guessed results where the weights are related to the uncertainties. Location information is fused in similar manner. (Step 725)
This estimated number of persons may be combined with predictive information from the past state of the global tracker which gives the prior distribution of number of people and their locations in the scene. (Steps 755, 760)
For each object hypotheses, space-time projections are computed with kernels that are chosen as function of object hypotheses and their locations as described in U.S. patent Application Publication No. 2008/0100473 entitled “Spatial-temporal Image Analysis in Vehicle Detection Systems” inventors Gao et al. published May 1, 2008 assigned to the same assignee as the present invention, the entire subject matter thereof being incorporated herein by reference. (Step 730)
The space time projections are used to diagnose occlusion states (occluded versus not-occluded) so that they can provide evidence to the mean-shift tracker for termination of tracks. (Step 735)
Given the object hypotheses and their locations, kernels for tracking the object hypotheses are adapted so that their likelihood of tracking will be maximized. (Step 740)
Mean-shift trackers [D. Comaniciu, V. Ramesh, P. Meer: Real-Time Tracking of Non-Rigid Objects Using Mean Shift. CVPR 2000: 2142-2149 (Best Paper Award, D. Comaniciu, V. Ramesh, P. Meer: The Variable Bandwidth Mean Shift and Data-Driven Scale Selection. ICCV 2001: 438-445, V. Parameswaran, V. Ramesh, I. Zoghlami: Tunable Kernels for Tracking. CVPR (2) 2006: 2179-2186, also US Patent Application Publication No, 2007/0183630, Tunable kernels for tracking, with inventors V. Parameswaran, V. Ramesh and I. Zoghlami, assigned to the same assignee as the present invention, the entire subject matter thereof being incorporated herein by reference] are initialized whose histograms are constructed by using the adaptive kernels. The histogram feature space will be adaptively constructed based on the most discriminative color space that discriminates the object from the neighboring objects and the background. (Step 745)
Partial tracks of objects along with their histograms and past locations are estimated and maintained in a tracking hypotheses list. When occlusion is predicted for a given object ID, the mean shift tracker is suspended. Re-initialization of the track is done by a search process that looks for a match of the object histogram based on the distribution of predicted location of the occluded object. (Step 750)
Object locations, appearance attributes and partial tracks of objects are fused (using a multiple hypotheses tracking and fusion framework) with the predictions from the past. (Steps 755,760)
The estimates can be improved through use of multiple cameras via head position estimation (through triangulation) when the objects are in mid to close range.
Furthermore, posture analysis techniques can be used to estimate 3D gait and posture and improve the choice of kernels for tracking.
Here, the method used three steps: a fast indexing step to estimate hypotheses for person (group) locations and attributes, a context driven adaptive detection and tracking step that refines these hypotheses, builds (or refines) statistical models of appearance, geometry of persons being tracked and utilizes these models for discriminative tracking, and an online uncertainty estimation step for person locations and attributes. The uncertainty estimates are utilized to derive predictive distributions of expected locations of persons in the subsequent frame and will enable the derivation of occlusion hypotheses that will be fed back to adaptive decisions on feature representations useful for robust tracking. The fast indexing step is illustrated in
The advantages of using such a fast indexing (divide-and-conquer) approach include: 1) by spending a small amount of computational power on easy cases, the overall system can perform most efficiently; 2) by focusing more computational power on more difficult cases, the system can achieve best accuracy; 3) Detectors and trackers can be tailored toward each specific case such that their overall performance can be more easily optimized; 4) Scenario dependent performance can be evaluated more accurately by theory and by experiments, thus providing a better understanding of the vision system performance bound under different conditions. Note that in the general scenarios, groups of pedestrians may split or merge thus making simpler or more difficult cases. When splitting or merging happens, the number of people in the new groups can be inferred from the original groups or the fast indexing algorithm can be called upon again.
An important tool for understanding and fine-tuning the detectors and trackers is performance characterization, i.e., a mapping from the tuning parameters to the tracker/detector success rate [see V. Ramesh, R. M. Haralick: Random Perturbation Models and Performance Evaluation of Vision Algorithms. CVPR 1992: 521-27, V. Ramesh, Performance Characterization of Image Understanding Algorithms, Ph.D. Dissertation, University of Washington, Seattle, March 1995, M. Greiffenhagen, V. Ramesh, D. Comaniciu, H. Niemann: Statistical Modeling and Performance Characterization of a Real-Time Dual Camera Surveillance System. CVPR 2000: 2335-2342].
Here, recorded videos are used with ground-truth for performance characterization of pedestrian detection and tracking sub-modules at different levels of difficulties. An experimental protocol that describes the data collection process, systems analysis and performance measurement process is devised to evaluate system performance and empirically determine tuning parameters for detection and tracking sub-modules. For instance, we can gather data with various crowded settings, different illumination conditions (e.g. different times of day), object or group attributes with similar size, appearance (clothing), and facial features to effectively determine the limits of our tracking system. Quantitative results such as probability of correct tracking, duration of persistent tracks, as functions of the various factors will be used to guide fine tuning of each sub-module and fusion of these sub-systems. A systematic sampling of the possible space of videos is performed so that the behavior of the tracking system under various factors can validate as outlined in the Table I below:
In the following sections we will describe in more details our technical approach for each of the modules that are utilized in our proposed tracking framework.
Fast indexing is an efficient algorithm that quickly calculates the number of people in a region of interest (ROI) in an input video frame. In the example in
The basic principle is to quickly map the contour of the group of people into the number of pedestrians. The algorithm uses discrete Fourier transform (DFT) on the 50 uniformly sampled points on the contour. Magnitudes of the DFT coefficients are used as indices to retrieve the number of people (and candidate configurations) in a blob using a k-nearest neighbor (k-NN) approach.
Referring to
After indexing, sub-classes of more specific problems can be defined. Group size and other prior knowledge regarding the environment, e.g., scene geometry and camera configurations, are used to design a set of pedestrian detectors and trackers that perform best in each sub-problem.
For level-1 scenarios, a basic blob tracker is utilized that is based on a combination of mean-shift tracker [D. Comaniciu, V. Ramesh, P. Meer: Real-Time Tracking of Non-Rigid Objects Using Mean Shift. CVPR 2000: 2142-2149 (Best Paper Award, D. Comaniciu, V. Ramesh, P. Meer: The Variable Bandwidth Mean Shift and Data-Driven Scale Selection. ICCV 2001: 438-445, V. Parameswaran, V. Ramesh, I. Zoghlami: Tunable Kernels for Tracking. CVPR (2) 2006: 2179-2186] and the background-subtraction results, given the prior information that there is only one person in the blob.
In level-2 cases, we adopt algorithms described in [L. Dong, V. Parameswaran, V. Ramesh, I. Zoghlami: Fast Crowd Segmentation Using Shape Indexing, ICCV 2007, V. Parameswaran, V. Ramesh, I. Zohlami: Tunable Kernels for Tracking. CVPR (2) 2006: 2179-2186]. The detection algorithm [L. Dong, V. Parameswaran, V. Ramesh, I. Zoghlami: Fast Crowd Segmentation Using Shape Indexing, ICCV 2007] involves two steps: 1) fast hypothesis generation using a discrete Fourier transform-based indexing method; 2) optimal people configuration searching using Markov-Chain Monte-Carlo (MCMC) sampling. A robust, real-time multiple pedestrian tracker is provided by 1) using advanced kernel methods [V. Parameswaran, V. Ramesh, I. Zoghlami: Tunable Kernels for Tracking. CVPR (2) 2006: 2179-2186]; 2) using occlusion reasoning of pedestrians inferred from the world state; the group of pedestrians are ordered from least to most likely to be occluded. Pedestrians that are less likely to be occluded will be tracked first. After they are tracked, more heavily occluded pedestrians are tracked using the configuration of tracked people and the possible occlusion introduced by them; 3) using the stereo cameras to track a pedestrian when he/she is covered by both cameras of the stereo system.
Solving the level-3 cases is a combination of utilizing active sensor planning, discriminative feature selection and optimal kernel weight calculation. In this case, a PTZ camera is actively involved in the detection/tracking process. The following steps are used for level-3 tracking and detection: 1) the overview camera detects entrance of crowd. 2) The PTZ camera is summoned to have a close-up view of the crowd. 3) A face detector [B. Xie, D. Comaniciu, V. Ramesh, M. Simon, T. E. Boult: Component Fusion for Face Detection in the Presence of Heteroscedastic Noise. DAGM-Symposium 2003, B. Xie, V. Ramesh, Y. Zhu, T. Boult: On Channel Reliability Measure Training for Multi-Camera Face Recognition. WACV 2007] or head/shoulder detector is used to detect the people in the scene. Detection results from multiple frames are combined for best accuracy. 4) A registration algorithm is used to match the PTZ view to the overview panorama. Utilizing the registration, the detected people and their locations can be transferred from the PTZ camera to the overview cameras. 5) The overview cameras track and book keep detected pedestrians. To track people in a crowd, the vision system must utilize prior scene geometry knowledge (camera calibration parameters, 3D scene models) and current world state to adaptively determine the best kernels and the most discriminative features for tracking.
A pedestrian may dress in clothes that have similar colors to the background. This will be a difficult case for both the background subtraction algorithm and the mean shift tracker. It is necessary to select the right set of discriminative features for reliable tracking. Siemens will incorporate the most advanced discriminative trackers [S. Avidan: Ensemble Tracking. IEEE Transactions on Pattern Analvsis and Machine Intelligence (PAMI) Vol. 29(2), pp 261-271, 2007, R. Collins and Y. Liu: On-Line Selection of Discriminative Tracking Features, ICCV'03, S. Lim, L. S. Davis and A. Mittal: Task Scheduling in Large Camera Networks. ACCV (1) 2007: 397-407] in this system.
People in a group may dress in similarly colored clothes. This case will cause tremendous difficulty for color-based trackers such as the mean-shift tracker. In order to track the group of people, we need to 1) find the most discriminative features for tracking. This can be achieved by using feature selection in the color space [R. Collins and Y. Liu: On-Line Selection of Discriminative Tracking Features, ICCV'03] or by using classification-based methods [S. Avidan: Ensemble Tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 29(2), pp 261-271, 2007, S. Lim, L. S. Davis and A. Mittal: Task Scheduling in Large Camera Networks. ACCV (1) 2007: 397-407]. If the ambiguities can be resolved by finer resolution images, an active sensor planning approach can be utilized; 2) use the dynamics and physical constraints of the group. There are certain physical rules that a group of pedestrians need to obey. For example, one pedestrian cannot interpenetrate another, and from frame to frame the acceleration and velocity of a pedestrian cannot change dramatically (in a matter of 33 milliseconds).
A number of embodiments of the invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. Accordingly, other embodiments are within the scope of the following claims.
This application claims priority from U.S. Provisional application No. 61/058,234 filed Jun. 3, 2008, the entire subject matter thereof being incorporated herein by reference and from U.S. Provisional application No. 61/107,707 filed Oct. 23, 2008 the entire subject matter thereof being incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61058234 | Jun 2008 | US | |
| 61107707 | Oct 2008 | US |
