Patent Application
20070086622
Visual tracking of specified objects (i.e., target objects) is an area of computer vision that has many useful applications. For example, visual tracking may be used in video surveillance, human-computer interfaces, digital video editing, and the like. In general, visual tracking tracks a target object during a given observation, such as a video sequence. Unfortunately, visual tracking techniques have difficulty tracking target objects in several situations.
For example, one situation where visual tracking techniques have difficulty tracking the target object occurs when the target object experiences sudden motion, such as from an unexpected dynamic change of the target object itself or from an abrupt motion of the camera. Another situation where visual tracking techniques have difficulty occurs when a similar looking object is in close proximity to the target object. In this situation, tracking techniques have difficulty distinguishing which of the two objects is the target object. Visual tracking is also difficult when occlusion occurs and the target object is partially or completely hidden by another object.
In overview, most tracking techniques use recursive estimation to estimate a location of a target object at a current time t based on observations up to the time t. In a Bayesian framework, the tracking problem is commonly formulated as a recursive estimation of a time-evolving posterior distribution P(xt|Y1:t) of state xt given all the observations Y1:t, such that:
Recursive estimation has two major advantages: 1) efficient computation; and 2) natural fit with real-time or on-line tracking applications.
However, many real world applications fit in the category of offline tracking, such as event statistics in video surveillance, object-based video compression, home video editing, video annotation, visual motion capture, and the like. Therefore, the recursive approach has also been applied to offline visual tracking. When this is done, the long input video sequence is typically first decomposed into short sequences by specifying one or more keyframes. The specified keyframes can be any of the frames within the video sequence. Each keyframe contains an object template which designates the object to be tracked (i.e., the target object). Visual tracking using these decomposed short sequences is commonly referred to as keyframe-based tracking. The recursive approach is then applied to each of the short sequences in either the forward or backward direction. This approach, however, typically fails somewhere in the middle of the sequence. When this occurs, another keyframe is added at the failed location.
While adding new keyframes improves the outcome of the visual tracking, adding new keyframes in a trial-and-error manner is prohibitively time consuming. Thus, there is a continual need to improve upon the tracking techniques used in offline applications.
The present video tracking technique outputs a Maximum A Posterior (MAP) solution for a target object based on two object templates obtained from a start and an end keyframe of a whole state sequence. The technique first minimizes the whole state space of the sequence by generating a sparse set of local two-dimensional modes in each frame of the sequence. The two-dimensional modes are converted into three-dimensional points within a three-dimensional volume. The three-dimensional points are clustered using a spectral clustering technique where each cluster corresponds to a possible trajectory segment of the target object. If there is occlusion in the sequence, occlusion segments are generated so that an optimal trajectory of the target object can be obtained.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Non-limiting and non-exhaustive embodiments are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. For convenience, the left-most digit of a reference number identifies the particular Figure in which the reference number first appears.
The following description is directed at a video tracking technique that outputs a maximum a posterior (MAP) solution for a target object based on two object templates obtained from a start and an end keyframe of a whole state sequence. In overview, the technique first minimizes the whole state space of the sequence by generating a sparse set of local two-dimensional modes in each frame of the sequence. The two-dimensional modes are converted into three-dimensional points within a three-dimensional volume. The three-dimensional points are clustered using a spectral clustering technique where each cluster corresponds to a possible trajectory segment of the target object. If there is occlusion in the sequence, occlusion segments are generated so that an optimal trajectory of the target object can be obtained. The present video tracking technique handles sudden motion, ambiguity, and short/long periods of occlusion. These and other aspects of the present video tracking technique are now described in detail.
Computing device 100 may have additional features or functionality. For example, computing device 100 may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in
Computing device 100 may also have input device(s) 124 such as keyboard, mouse, pen, voice input device, touch input device, etc. Computing device 100 may also contain communication connections 128 that allow the device to communicate with other computing devices 130, such as over a network. Communication connection(s) 128 is one example of communication media. Communication media may typically be embodied by computer readable instructions, data structures, program modules, or other data. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Computer readable media can be any available media that can be accessed by a computer. By way of example, and not limitation, computer readable media may comprise “computer storage media” and “communications media.”
Various modules and techniques may be described herein in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. for performing particular tasks or implement particular abstract data types. These program modules and the like may be executed as native code or may be downloaded and executed, such as in a virtual machine or other just-in-time compilation execution environment. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. An implementation of these modules and techniques may be stored on or transmitted across some form of computer readable media.
Before describing the details of the present tracking technique, a state model and observation model for the present tracking technique will be described. The state model and observation model apply a keyframe-based framework. For the state model, a target object is represented as a rectangle R={p,s*{circumflex over (ω)},s*ĥ}, where p is the center rectangle and s is the scaling factor, {circumflex over (ω)}and ĥ are a fixed width and height of the object template, respectively. The state of the object is denoted as x={p,s}∈χ, where χ0 is the state space. State x1 refers to the state in the first keyframe I1 and the state XT refers to the state in the last keyframe IT. For the present tracking technique, both of these states x1 and XT are known.
The observation model is the color statistics of the target object. The object's color model is represented as a histogram h={h1,hH} with H (typically, H=8×8×8) bins in RGB color space. The Bhattacharyya distance between the associated histogram h(x0) of the state x0 and the associated histogram h(xi) of the state xi is defined as follows:
This model captures global color statistics. In another embodiment, a more sophisticated multi-part color model may be used if there is a certain spatial configuration of the target object. For example, in one implementation a multi-part color model such as the color model described in Proc. European Conf on Computer Vision, volume 1, pages 611-675, 2002, by Patrick Perez, C. Hue, J. Vermaak, and M. Gangnet, is utilized.
For trajectory optimization, the posterior of the whole state sequence X={x2,Λ,xT−1} for a given video sequence or observation Y={y1,Λ, yT} and known two states {x1,xT} can be represented as follows under the first order Markov independence assumption:
where the local evidence Ψ(yi|xi, x1, xT) is defined using the Bhattacharyya distance as follows:
Ψ(yi|xi,xi,xT)˜exp(−min{B2[h(xi)], B2[h(xi), h(xT)]}/2σh2 (4)
where σk2 is a variance parameter. The variance parameter measures the similarity between the color histogram h(xi) of the state xi to the closest color histogram between h(x1) in the keyframe I1 or h(xT) in the keyframe IT. The potential function Ψ(xi,xi−1) between two adjacent states is defined as:
Ψ(xi,xi+1)˜exp(−D[xi,xi+1]/2δp2), (5)
where D[xi,xi+1]=∥pi+pi+1∥2+β∥si+si+1∥2 is the similarity between state xi and xi.
Parameter σp is a variance parameter that controls the strength of smoothness and β is a weight between location difference and scale difference. Thus, β is a smoothness constraint on the whole trajectory X′={X0, . . . , XT} of the target object.
The goal of the keyframe-based tracking is to obtain the MAP solution of Equation (3) as follows:
X*=arg max P(X|Y). (6)
However, the complexity O of determining the MAP solution is O(N2T), where N is the number of discrete states in one frame and T is the frame number of the input video. Thus, for a 320×240 video, the quantized state space is still very large even for the present state representation. Therefore, the present bidirectional tracking technique focuses on reducing the state space in an effective and efficient manner.
At block 204, an initial state for the target object is obtained from within the start frame and a final state for the target object is obtained from within the end frame. The initial state and final state may be selected by a user. For example, the user may place a rectangle around the target object. The initial state and final state are then determined based on the location of the rectangle within the frame. Processing continues at block 206.
At block 206, trajectory segment analysis is performed. Briefly, trajectory segment analysis, described in detail later in conjunction with
At block 208, occlusion analysis is performed. Briefly, occlusion analysis, described in detail later in conjunction with
At block 210, trajectory optimization is performed. After getting a set of trajectory segments (block 206) and occlusion segments (block 208), a single optimal trajectory between two keyframes is computed by trajectory optimization. Briefly, the trajectory optimization, described in detail later, is performed in a coarse-to-fine manner in order to obtain more accurate tracking results. In one embodiment of the coarse trajectory optimization process, states around the computed segments are uniformly sampled using three discrete scaling factors in each frame to determine an optimal trajectory. In one embodiment of the fine trajectory optimization process, states around the optimal trajectory are sampled using five discrete scaling factors in each frame. Each of these blocks will now be described in greater detail.
At block 304, after the state space has been reduced, a small number of three-dimensional trajectory segments are determined. In overview, the two-dimensional modes need to be converted into three-dimensional space in order to obtain the true trajectory for the target object. Briefly, the three-dimensional trajectory segment extraction process, described in detail later in conjunction with
At block 404, one embodiment of the present tracking technique obtains a set of starting positions within each frame. The starting positions are determined by uniformly sampling locations in the frame. Scaling may be performed using 3-5 discrete levels. In a further refinement (block 406), the spatial sampling interval is set based on the target object size. For example, the spatial sampling interval may be set slightly smaller than half the size of the target object. Processing continues at block 408.
At block 408, an evidence surface is computed. In one embodiment, the local evidence Ψ(yi|xi,x1,xT)is defined using a Bhattacharyya distance. The evidence surface is then computed using the following equation:
Ψ(yi|xi,x1,xT)˜exp(−min{B2[h(xi),h(x1)],B2[h(xi),h(xT)]}/2σh2). (7)
The parameter σh2 is a variance parameter. The Bhattacharyya distance measures the similarity between the color histogram h(xi) of the state xi to the closest color histogram between h(x1) in the keyframe I1 or h(XT) in the keyframe IT. Processing continues at block 410.
At block 410, modes are identified based on the evidence surface. In overview, the most likely positions (i.e., modes) are peaks (i.e., local maxima) on the evidence surface. Each “mostly likely” 2D mode represents a state x′ whose observation is similar to the object template in the keyframes. In other words, the local evident Ψ(y|X′,X1,XT) is high. To efficiently find these modes, one embodiment (block 412) utilizes a mean shift algorithm which is a nonparametric statistical method for seeking the nearest mode of a point sample distribution. In overview, given an initial location, the mean shift algorithm computes a gradient direction of a convoluted evidence surface by a kernel G using the following equation:
Thus, given an initial location p in the image, the mean shift computes new location p′=p+Δp according to the above equation, where G is a kernel function and summations are performed over pixels q in the object's rectangle. The mean shift vector Δp opposites to the gradient direction of a convolution surface computed with the shadow of kernel G. Due to this property, it was determined that the mean-shift algorithm provides an efficient iterative method for determining a local mode of the target object. Therefore, the mean shift algorithm runs independently from each starting position. After convergence, the outcome from the mean shift algorithm is a number of local modes. Processing continues at block 408.
At block 414, a sparse set of local 2D modes is generated. In one embodiment (block 416), a mode is rejected if the corresponding state mode x′ has a local evidence p(yi|x′)≦0.5. In a further refinement (block 418), very close modes may be merged. The outcome is a sparse set of local modes in each frame as shown in
Referring now to
The set of local 2D modes in each frame then represent the state space that is used for further tracking purposes in accordance with one embodiment of the present tracking technique. As illustrated by the set of local 2D modes 512-516, the state space has been significantly reduced. This reduction in state space allows further analysis for tracking purposes to be performed more efficiently. In contrast, other tracking techniques use the whole continuous state sequence space which has an enormous number of local minimums due to nonlinear dynamics and non-Gaussian observations. For example, gradient-based techniques often become stuck at a local minimum. Techniques that compute the MAP solution using the Viterbi algorithm using a discrete hidden markov model (HMM) representation must handle a large quantized state space, given a 320×240 video. Thus, the two-dimensional extraction process in the present tracking technique provides an efficient technique for significantly reducing the state space which leads to a more efficient tracking analysis.
At block 604, given a set of points M={mn}n=1N in 3, an affinity matrix is built where
3 is the 3D euclidean space. In one embodiment, the basic affinity matrix A∈
N×N is defined as follows:
Aij=exp(−∥pi−pj∥2/2σp2−∥ti−tj∥2/2 σt2) (9)
where the scaling parameters σp and σt control how rapidly the affinity matrix Aij falls off with the distance between two points mi and mj in space and time, respectively. In another embodiment of the present tracking technique, a more sophisticated but expensive affinity matrix Aij ′ may be used to encourage more compact trajectory segments. This affinity matrix Aij ′ may be defined as follows:
Aij′=αAij+(1−α)exp(−B2[h(mi),h(mj)]/ 2σh2). (10)
The last term considers the similarity between the appearances (e.g., color histogram) of two modes (mi, mj). Variable α is a weighting factor. A suitable weighting factor may be a value of 0.5. Processing continues at block 606.
At block 606, the 3D points are partitioned into clusters. Various spectral clustering techniques may be used to partition the 3D points into clusters. These various spectral clustering techniques each analyze eigenvectors in slightly different ways. In one embodiment of the present video tracking technique, the Ng's algorithm is used which uses K eigenvectors simultaneously for K-class clustering. For this technique, a matrix L=D−1/12AD−1/2 is constructed where D is a diagonal matrix.
Based on the largest eigenvectors of L, a matrix E=[e1,Λ, eK]∈N×K is computed, where ek is the set of normalized K largest eigenvectors of matrix L. Each row of matrix E is then treated as a point in
N. A conventional K-means algorithm then clusters the points into K clusters. If row i of the matrix E is assigned to a cluster k, the original point is assigned to cluster k. Processing continues at block 608.
At block 608, the trajectory segments are obtained based on the clusters. For example, all 3D points in cluster k are treated as a trajectory segment Trk. This is done for all the clusters to obtain K trajectory segments Tr={Tr1,Λ, TrK}. Using the spectral clustering as described for the present video tracking technique, a number of “meaningful” trajectory segments are produced. Thus, the 3D trajectory extraction process successfully extracts a number of 3D object trajectories Trk based on the 2D local modes M. Each object trajectory represents a possible segment of the tracked object in the 3D volume.
In comparison with other clustering techniques, this embodiment of spectral clustering embeds 3D data points on a surface of a unit sphere in a K dimensional space determined by the K largest eigenvectors of L. This allows curved trajectories or manifolds in the original 3D space to be separated which results in more meaningful trajectory segments. In contrast, every cluster determined using standard k-means clustering needs to be a convex region. Thus, the “true” trajectory can not be obtained if the trajectory is highly curved and/or some partition of the trajectory is not a convex region.
Once the trajectory segments have been extracted, an analysis is performed to determine the possibility that the target object may have been occluded during part(s) of the video sequence. This analysis, shown as block 208 in
Based on these observations, the present bi-directional, tree-growing process for occlusion analysis was formulated.
At block 704, a trajectory segment(s) is located that contain an object template in the keyframe. In the multiple tree embodiment, two such trajectory segments may be located and one of the trajectory segments is added to each tree TA and TB as an active node. Processing continues at block 706.
At block 706, the remaining trajectory segments are classified into an active candidate list. In overview, the segments in the active candidate list are then either excluded or used to determine an occlusion segment. Processing continues at block 708.
At block 708, trajectory segments in the active candidate list that are parallel to an active node are removed from the active candidate list. Two segments are parallel if the overlapping time and the shortest distance between them are not more than certain empirical thresholds. In one exemplary implementation, the time threshold is 30 frames and the distance threshold is 50 pixels. Processing continues at block 710.
At block 710, a bi-directional tree-growing process is performed. In overview, the bi-directional tree-growing process, described later in conjunction with
At block 802, a trajectory segment from the active candidate list is selected as a current active node (i.e., an active leaf node of the previous active node). The selected active leaf node does not have a child node at this point. Processing continues at block 804. Processing continues at block 804.
At block 804, Q-best occlusion segments for the current active node are determined. A forward search may be performed in the active candidate list to identify trajectory segments Tr* that overlap other trajectory segments along the time axis. The Q-best occlusion segments may then be generated between the trajectory segment of the active leaf node Tra and each of these identified trajectory segments Tr*. In one embodiment, the Q-best occlusion segments are determined based on Lo+γSo, where γ=10 is a weighting factor. Lo and So are the length (pixel) and maximum speed (pixel x frame−1) of the hypothesized B-spline between two connected trajectory segments. In overview, the B-spline is used to generate a missing occlusion segment O with respect to two disjoint trajectories Tr
uses weighted least squares:
where sj′=(tj=t1)/N′ is a temporal parameterization of the B-spline in frame tj. Even though the parameterization of the B-spine is an approximation, it was found to yield good results.
The weighting function ω(mj) is defined as:
where tAe and tBs are the last frame number in Tr1 and the first frame number in Tr2. The scaling parameter σωis set to 20. Using a weighting function, a larger weight can be given to a point nearby tAend and tBstart. Finally, the fitted B-spline curve between Tr1 and Tr2 is sampled as the occlusion segment O. Processing continues at decision block 806.
At decision block 806, a determination is made whether any one of the Q-best occlusion segments reach the other end. In other words, whether one of the occlusion segments meets up with the trajectory segment set as the current active node in the other tree. The current active node in the other tree may be the trajectory segment containing the object template or one of the other trajectory segments set as the active node during processing of the other tree. If one of the Q-best occlusion segments reaches the other end, processing continues at block 814. Otherwise, processing continues at decision block 808.
At decision block 808, a determination is made whether one of the Q-best occlusion segments is a dominant occlusion segment. This optional determination is based on whether one of the trajectory segments has a value for Lo+γSo that is significantly smaller than the values for the other trajectory segments. If there is a dominant occlusion segment, processing continues at block 810. Otherwise, processing continues at block 812.
At block 810, the dominant Q-Best occlusion segment is added as a child node and set as the current active node. The previous active current node is set as inactive because it now has a child node. Processing then continues at block 804 with this current active node.
At block 812, if there is not a dominant Q-best occlusion segment, each of the Q-best occlusion segments are added to the active candidate list. By doing so, multiple occlusion segments may be joined until one can be joined with a trajectory segment. Processing then loops back to block 802 to select a “segment” (e.g., either an occlusion segment or a trajectory segment) from the active candidate list and proceeds as described above.
At block 814, if one of the Q-best occlusion segments reaches the other end, the occlusion segment is added to the other end in the tree. In the embodiment with two trees, the occlusion segment is added to both trees and set as inactive. The occlusion segment is then a bridge between the two trees. Processing continues at block 816.
At block 816, the discontinuities along the time axis are connected using B-spline interpolation in the tree. The result is a possible trajectory path from the initial state to the final state that includes trajectory segments and occlusion segments.
Processing is then complete.
Process 800 is performed for each child node that does not have any children. Once a child node has a child (e.g., at least one Q-best occlusion segment), processing for that node is complete. When there are no longer any active nodes in either tree or trajectories that have been identified that connect the initial state to the final state, process 800 is complete. One will note that trajectory segments connect with occlusion segments because during the trajectory segment analysis process 300 the longest possible trajectory segments were obtained. Therefore, typically, trajectory segments do not connect with other trajectory segments.
After getting a set of object trajectories and occlusion trajectories, a single optimal trajectory between two keyframes is computed by sampling these trajectories in accordance with the present video tracking technique using trajectory optimization (block 210 in
In frame #016, cup 1404 is moved vertically down so that it is lower than cup 1414. In frame #027, cup 1404 is moved right so that it is below and to the right of cup 1414. During frames #028-035, cup 1404 is moved under the individuals arm holding cup 1414 and ends at a position to the right and slightly higher than cup 1414. Because cup 1404 is behind the left arm of the individual, there is considerable occlusion, as shown in frame #035. In frame #052, cup 1404 is moved up and approximately directly over cup 1414. In frame #065, cup 1404 is positioned at the same height as cup 1414 and to the left of cup 1414, with little occlusion. In frame #086, cup 1404 is lowered and moved slightly to the left in the frame. In frame #098, cup 1404 is positioned below cup 1414 and is partially occluded due to the individual's hand (not shown) in front of cup 1404. In frame #105, cup 1404 is moved behind the individuals left arm which is holding cup 1414. Therefore, cup 1404 is almost completely occluded. Interestingly, however, the predicted location 1410 accurately predicts the location of where cup 1404 would be located. In frame #132, cup 1404 is raised slightly so that a small portion can be seen over the individual's left arm (not shown). In frame #147, cup 1404 is moved to the top of the frame and is almost completely occluded by the individual's fingers holding cup 1404. Thus, as illustrated by the video sequence 1410, the present tracking technique accurately predicted the position of the target object even when the target object became completely occluded. The results of the present tracking technique at various stages during processing are illustrated in
Graphics 1000 and 1100 illustrate two views of the 2D mode in 3D, where the 2D mode points were generated during the two-dimensional extraction process of the present tracking technique. The vertical axis is the frame number in the sequence. The other axis are the x,y position of the target object in each frame. The fairly straight line down the center corresponds to the identical cup 1414 in video sequence 1400. Graphic 1200 illustrates meaningful trajectory segments obtained after performing the three dimensional trajectory segment analysis with spectral clustering. Because the target object was complete occluded during portions of the video sequence, there is not one trajectory from the initial state to the final state. Graphic 1300 illustrates an optimal trajectory for the target object after the occlusion analysis has been performed. The circles within the dashed rectangles represent the filled-in occlusion segments.
Thus, as described, the present tracking technique successfully handles partial or complete occlusions. The tracking technique employs an innovative trajectory segment representation to obtain a small number of trajectory segments out of the possible object trajectories in the 3D video volume. The trajectory segments are extracted from the input video using a spectral clustering method. With this representation, the MAP solution is obtained by employing a discrete Hidden Markov Model in a coarse-to-fine manner using the trajectory segments. In addition, the present tracking technique employs an occlusion analysis process to robustly infer possible occlusion trajectory segments of the target object.
Typical, parameter settings that may be used for the present tracking technique are now discussed. The selection of the cluster number K is an open problem in clustering. Because there is a grouping mechanism in the occlusion analysis of the present tracking technique, it was determined that a slightly larger K than the “best” one is acceptable. Therefore, the clustering number K may be set so that K=7 or K=10. In 2D mode extraction, G is a Gaussian kernel whose standard deviation may be about ⅙ the size of the target object in the keyframe. In the 3D trajectory extraction, scaling parameters σpand σtmay be set to 10 and 20, respectively. In trajectory optimization, the variance parameters σhand σpmay be set to 10 and 1, respectively.
As described above, the present bi-directional tracking using trajectory analysis takes the whole video sequence and the information in two keyframes as inputs and outputs the MAP solution of the whole state sequence of the target object in the Bayesian framework. Bi-directional tracking is based on trajectory analysis which extracts a number of 3D trajectories, a compact representation for object tracking, from the input video. Then, the occlusion trajectories are inferred at the trajectory-level. Finally, the MAP solution is obtained by trajectory optimization in a coarse-to-fine manner. Experimental results show the robustness of the present technique with respect to sudden motion, ambiguity, short-time and long-time occlusion.
The present bi-directional tracking technique may be applied in many offline applications. One new area in which it may be applied is online advertisements. For this application, a moving target may be “clicked” to automatically link to a related website. Because the present bi-directional tracking technique accurately tracks the target object, the position of the moving target object may be given to a potential application for its own use. This and many other applications are envisioned for the present tracking technique.
While example embodiments and applications have been illustrated and described, it is to be understood that the invention is not limited to the precise configuration and resources described above. Various modifications, changes, and variations apparent to those skilled in the art may be made in the arrangement, operation, and details of the methods and systems of the present invention disclosed herein without departing from the scope of the claimed invention.
This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 60/727,119 filed Oct. 14, 2005.
| Number | Date | Country | |
|---|---|---|---|
| 60727119 | Oct 2005 | US |
