Patent Grant
9898677
The present invention relates to identifying and tracking objects in a video captured from at least one camera.
Video tracking, also called VMTI filtering (video motion tracking indication filtering) is a process of detecting and uniquely identifying moving objects in a video sequence, e.g., as captured by a video camera. The information determined can be used in a variety of applications, e.g. traffic monitoring, security surveillance, etc.
One class of video tracking systems use a feature detector to detect features, e.g., regions in an image (a frame) of an input video, then match the detected features from frame to frame, including using a motion model to determine paths for each detected part. These paths are classified into those of moving objects and those of static objects. These paths are then grouped into individual objects. The classification and grouping may be prone to errors, for example, parts of different paths may be erroneously grouped into a single path because of overlapping paths, and also a path of a single object may be erroneously separated into two objects.
Overview
Particular embodiments of the invention include a video tracking method, a --transitory medium containing computer-readable instructions that when executed carries out the video tracking method, and a video tracking system. An important application is carrying out video tracking in videos obtained from one or more airborne cameras.
One set of embodiments of the present invention relates to a method of operating a hardware system to process video that includes a sequence of frames, the method to determine and track moving objects in the video. The method comprises accepting frames of a video, e.g., from a source of video data such as a video camera, a computer-readable file, a packetized video stream, and so forth. The method further comprises detecting and extracting regions from an accepted frame, e.g., extremal regions, or maximal stable extremal regions (MSER), each region possibly being a part of an object. The method further comprises matching and tracking parts, including using the extracted parts of a current frame and matching each part from the previous frame to match to a region in the current frame, and further to track parts to form part tracks. The part tracking comprises changing of reference frames from time to time, e.g., periodically such that the tracking being in relation to the most recent reference frame or frames. The part tracker also may comprise from time to time, e.g., periodically introducing new parts to track that were not previously matched. The method further comprises, for each tracked part path, to determine a set of path features, and to classify using the path features, each path as that of a moving part (a mover) or a stationary path (a static). The method further comprises clustering the paths of movers, including grouping parts of movers that likely belong to the same object in order to generate moving objects and the tracking thereof. In some versions, the clustering comprises, for a frame, determining a set of connectedness features that capture the relationship between pairs of movers in the frame. In one embodiment, such connectedness features include features related to a velocity difference, proximity of the parts, overlap between the parts, and background similarity of respective background of the parts. These connectedness features are normalized, such that the method robust to scale. The clustering includes estimating two sets of probability functions (in some versions, two joint probability density functions) for the connectedness features, a first set for features of pairs belonging to the same part and a second set for features of pairs not belonging. One aspect of some embodiments is that the estimated probability function sets are obtained by supervised learning using a training set of videos containing static and moving parts that have been manually grouped into objects, i.e., that are pre-labeled. The clustering uses these estimated probability functions to determine a measure of probability of any two pairs of parts belonging to the same object, with any probability measure less than a pre-defined or user selected threshold being indicative of no connection.
The clustering further may comprise matching of moving objects between frames by determining correspondences between each candidate object in the current frame with objects in the previous frame. In one version, the clustered determines a measure of correspondence from current frame's candidate groupings to past object groupings based on the numbers of parts in a candidate object that are in objects that contain the respective parts in all previous frame. The clustering further may carry out bipartite matching to find the best correspondences for propagation of object groupings from frame to frame.
Another aspect of the method is that the clustering comprises short interval grouping (SIG) filtering that can eliminate some short interval erroneous groupings, the SIG filtering using hysteresis filtering of an objects stability, i.e., whether or not a part of an object of a current frame is the same as the previous one or more frames to ensure stable labeling.
A second set of embodiments of the present invention relates to a system for processing video that includes a sequence of frames to determine and track moving objects in the video. The system may be constructed, by programming and/or specialized hardware modules, to carry out the steps of the method embodiments described above for identifying the different moving objects in each of the frames. The system may include or be connectable to a source of video frames such as an imaging device for receiving the video data. The system comprises a region-extractor that uses features such as MSERs or to extract regions, e.g., extremal regions that may be part of one or another object from each frame. The extracted parts of a current frame are used by a part matcher and tracker that includes a trail data structure maintaining for each tracked part the track, e.g., as a list of coordinates. The part matcher and tracker comprises a part tracker that is operable to match each part from the previous frame to a region in the current frame, and further comprises a part tracker that tracks parts to form part tracks. The part tracker may changes reference frames from time to time, e.g., periodically with tracking being in relation to the most recent reference frame. Additionally, from time to time, e.g., periodically, e.g., in every frame or every few frames, parts that were not matched by the matcher may be introduced by the tracker as new parts to be tracked. In one embodiment, parts are tracked for a pre-pre-defined or user-defined amount of time, e.g., for the time corresponding to the number of reference frames in maintained by the system.
The system further comprises a part path classifier operative to accept tracked part paths, to determine a set of path features, and to use the path features to classify each path as that of a moving part (a mover) or a stationary path (a static). The system further comprises a part-path clusterer, operative to accept the paths of movers and to group parts that likely belong to the same object in order to generate moving objects and the tracking thereof. The clusterer when operating determines, for each frame, a set of connectedness features that capture the relationship between pairs of movers in the frame. In some versions, the features relate to the velocity difference (speed and direction), proximity of the parts, overlap between the parts, and background similarity of respective background of the parts. These connectedness features are normalized, such that the system is robust to scale. The clusterer is operative to estimate two sets of probability functions for the connectedness features, a first set for features of pairs belonging to the same part and a second set for features of pairs not belonging. The estimated probability function sets are obtained by supervised learning using a training set of videos containing static and moving parts that have been manually grouped into objects, i.e., that are pre-labeled. The clusterer uses these estimated probability functions to determine a measure of probability of any two pairs of parts belonging to the same object, with any probability measure less than a pre-defined or user selected threshold being indicative of no connection. Thus, the clusterer carries analysis is carried out for pairs of moving objects in the current frame, leading to a candidate set of groupings for the current frame.
The part-path clusterer is further operative to carry out matching of moving objects between frames by determining correspondences between each candidate object in the current frame with objects in the previous frame by determining a measure of correspondence from the current frame's candidate groupings to past object groupings based on the numbers of parts in a candidate object that are in objects that contain the respective parts in all previous frame. The clusterer further is operative to carry out bipartite matching to find the best correspondences for propagation of object groupings from frame to frame.
Another aspect of embodiments of the system is the inclusion in the clusterer of a short interval grouping filter that can eliminate some short interval erroneous groupings, for example an erroneous grouping when two overtaking objects are fused in the short term to a single object, or when an object is temporarily and erroneously split into two objects before being labeled later as a single object. The short term interval grouping filter uses hysteresis filtering of an objects stability to ensure stable labeling.
Particular embodiments may provide all, some, or none of these aspects, features, or advantages. Particular embodiments may provide one or more other aspects, features, or advantages, one or more of which may be readily apparent to a person skilled in the art from the figures, descriptions, and claims herein.
In the remainder of this document, citations are provided within square brackets, and a list of cited references is provided towards the end of the description.
A first set of embodiments of the present invention relates to a machine-implemented method of carrying out identification and tracking of objects from one or more videos, each video comprising a respective sequence of frames, each an image. In the description, we assume a monochrome video of frames, each a set of pixels arranged as an array. In the example embodiments described herein, each frame of the video is first preprocessed to a common frame size of 640×480 pixel monochrome image. Further, in the examples described, the video is at 30 fps (frames per second). However, the invention is not limited to such input. The input video can be any resolution, e.g., 1280×960 (as in 1080i and 1080p video), or 320×240 pixels, etc., 30 fps, 60 fps, etc., even 240 line 15 fps videos. Furthermore, the video may be monochrome, color, or multispectral. For a color image, in one embodiment, the 640×480 pixel monochrome image is the luminance or intensity of the color image. For multispectral images, many possibilities exit, e.g., concatenating the individual components, processing the non-visual components separately as separate monochrome images, and so forth. We have tested the method and system on several other resolutions and frame rates, e.g., on 240 line 15 fps videos.
Suppose a plurality of frames of a video are accepted into the video tracking system 1100 shown in
In subprocess 103 of video tracking method 100, region features in each frame, e.g., in the current frame, are detected, and from these regions extracted from the frame. In one embodiment, an MSER (Maximally Stable Extremal Regions) detector is used to extract both positive and negative MSERs and from these to extract blob-like regions (we call these MSERs, also parts) for every input video frame. Note that alternative embodiments may use another type of feature detector in addition or instead of using MSER. One alternate version, for example, uses the well-known SIFT (Scale Invariant Feature Transformation) detector. However, for our purpose, we found that using an MSER detector yields good results, and such is used in one embodiment of the invention. For any frame, the result of subprocess 103 is a set of MSERs, stored, e.g., as a list of MSERs, called a “part-list” herein for the frame. The MSERs may represent parts of objects. The method assigns an identifier (call it a “part-ID”) for each MSER. In later frames, a detected MSER may be determined to be previously detected MSER, so the newly detected MSER maintains the same part-ID. In any frame, each MSER has a one or more properties, e.g., the location of its centroid, e.g., as an x,y pair, and such information is maintained in the part list. This part list is made available to the other subprocesses. Note also that detecting MSERs also may detect extremal regions that are not maximal.
In the remainder of this document, the word “part” might be used interchangeably with “MSER” to refer to the detected MSERs and later-processed MSERs.
Matching and Tracking
Matching and tracking subprocess 105 of method 100 tracks detected parts through the frames as frames are accepted into the method. Each frame has several parts, e.g., represented by a list. As a result of tracking, each part has a set of locations of the part's locations over time, called a track, so a track can be represented as a list of coordinates, so such that all the parts' tracks over time can be described by a list of part tracks, i.e., a list of lists. One embodiment of the invention maintains and updates the list of lists (called a “trail” herein) as information about all the parts is obtained as more and more frames are accepted. For a part in a given frame that is found in the next frame, the information on that part is updated (and identified by the part-ID). Additionally, one aspect of our tracking is the adding from time to time of any one or more new parts that do not correspond to any previously encountered parts, wherein for each new part, a new respective list (for a part track) with a new part-ID is created in the list of lists. Rather than the searching and adding of not previously encountered MSERs being carried out for each new frame accepted into the system, one embodiment of the method only introduces new parts to track from time to time, e.g., periodically every KF frames, i.e., at each KF'th frame. Period KF is called the new-part-updating period herein, and may be a settable parameter. For 30 fps (frames per second) video, one embodiment uses KF=1, meaning we add new parts every new frame. In another embodiment, we KF=15, representing ½ second. In yet another embodiment, KF=30.
In more detail, initially, for the first frame of the video, detection process 103 detects (extracts) a set of parts, assigns to each part a part-ID, and generates a part list, each respective part in the part list having the respective newly assigned part-ID.
Referring first to tracking, since each tracked part has a list of its properties as it is being tracked, the list of parts from extracting subprocess 103 becomes a list of lists, stored in a data structure 511. Consider a current frame (the ith frame) accepted into the system and the extremal (not just maximum extremal) regions detected therein in 103.
Subprocess 105 has access to the part-list (the list of lists, also called trail data structure 521) that includes all MSERs encountered in the previous (the (i−1)'th) frame. Consider an existing part in the previous frame. In 503, subprocess 105 searches for an extremal region in the current frame that matches a previously encountered part, the matching according to a matching criterion, e.g., by matching a set of features of a region. If a match is found, subprocess 105 updates the part list with information on the matched part in the current frame. Those parts that are matched have the same part-ID frame to frame.
Furthermore in 505, if l mod KF=0, subprocess 105 updates the part-list in the train data structure 521 with newly detected MSERs.
In more detail, for any connected region, e.g., an MSER or any extremal region, one may calculate a set of features. We call these features region-features. Suppose we calculate ND such region features. The set of ND region features may form an ND-vector of such features of a region, which we call a region-feature vector. Subprocess 105 maintains a region-feature vector of each MSER in the list of paths data structure (trail data structure 521). To match an MSER in the previous frame (call it an input MSER) with (extremal) regions detected in the current frame, the method in 503 uses a distance measure to compare the input MSER's region-feature vector with the feature vectors of all extremal regions (not just MSERs) within a provided search radius (search extent) in the current frame. The search extent of course depends on the speed that parts might be moving. In one embodiment, the search extent, expressed in pixels, is a settable parameter. In one embodiment, three values are available, and selected according to expected speeds of objects. One embodiment uses three search extents: 20, 40, and 80 pixels. The default value is 20 pixels. In one embodiment, the distance measure for searching for a match is a weighted L2 norm (weighted Euclidean distance), where the weights indicate the relative importance of any region feature in the region-feature vector. In one embodiment, dataflow box 503 selects the extremal region in the current frame that is closest in features to the input MSER, i.e., has a region-feature vector that has the lowest distance measure to the input MSER. This is carried out for each MSER being tracked.
One embodiment calculates the following region features to form the region-feature vector of a region: a measure of the mean gray value in the region, a measure of the region's area, the location of the region's centroid, and one or more measures of the region's size, e.g., the area of the region and the width and height of the bounding box of the region. One version further includes the stability of the extremal region. How to calculate such region features would be known to those in the art. The set of respective weights for each respective region feature can be pre-defined according to importance to form a pre-defined weight vector. Different weight vectors can be defined for each different type of video. In some versions, the respective weights for the respective region features can be learned using training videos—actual videos with known objects therein. Again, different types of reference videos can be used in the learning to generate different weight vectors appropriate for videos like the respective training video.
Additionally, every KF frames (the first frame being frame 0), i.e., responsive to l mod KF=0, in 505 and 506, any parts detected in the current frame that were not matched in 503 to any part in the previous frame, are added to the list of lists, and tracked from frame i forward. Note that in one embodiment, KF=1 new parts are added for each frame entering the method;
Also note that based on so-detected and tracked parts, a homography between any two previous frames may be calculated. In one embodiment, point correspondences are extracted from parts being tracked in both frames and a homography between the frames is estimated using the well-known RANSAC (RANdom SAmple and Consensus) method, used widely in the field of computer vision, in particular in the computation of scene homographies.
Using Reference Frames
For any part (MSER) being tracked, the part's path is needed for further analysis, for example for classifying tracked parts into those of objects that are moving (movers) and those that are stationary (statics). The method carries out such analysis within a selected time interval, i.e., a window wherein the background (of statics) is unlikely to change. In one embodiment, a window size of 60 frames (2 s in a 30 fps video) is used, and other embodiments may use different window sizes, e.g., 30 frames, depending on the expected speed of background changes, e.g., for an airborne camera, depending on the speed of the airborne vehicle on which the camera is mounted. The locations of a tracked MSER, e.g., the locations of the MSER's centroid within any time interval are in one embodiment defined in relation to a reference frame. Recall a path of a part is a list. In one embodiment, the list is of coordinates, e.g., x,y coordinates of centroids of the part, each coordinate being expressed as a mapping to some reference video frame. For example, for an interval of 60 frames, one embodiment defines the path of a tracked part as a list or array of 60 elements, each elements representing x,y coordinates of the part at consecutive frames for that path. The reference frame is the same for all tracked parts in an analysis interval of 60 frames. One simple choice of a reference frame is to use the first frame of the video—denoted frame 0—as the reference frame for all paths. However, it would then be a problem to track a part in some distant frame, e.g., frame 1000, in the case of a videos from a rapidly moving camera, e.g., an airborne camera, because there likely would be no common region-features between them; it would not be possible to determine correct alignment. One possible solution is to use a concatenation of in-between homographies. However, the homographies may have error, and after a while, there could be a significant accumulation of error such that extracted paths will differ from those in reality, possibly leading to misclassifications.
To avoid such problems, one aspect of subprocess 105 that we call reference-frame shifting comprises changing reference frames from time to time. In one embodiment, the interval between reference frames is constant, such that reference frames are periodic. In one embodiment, the period between reference frames, denoted KR is 15 frames. The method includes maintaining a number denoted NR of reference frames in a FIFO (First In First Out) buffer or a similar mechanism, shown as structure 523 in
Hereinafter, the term FIFO refers to the FIFO reference-frame buffer 523.
For NR=5, there are 15 frames between reference frames. FIFO 523 maintains information about each of its 5 reference frames, including for each reference frame, an associated mapping between itself and a previous reference frame, so that reference frame has an associated mapping between frame 15 and reference frame 0, reference frame has an associated mapping between frame 30 and reference frame 15, and so forth. The information maintained for each reference frame in FIFO 523 further includes a data structure of information of respective paths of parts (a part-path-data-structure). Initially, i.e., when the reference frame is accepted, for each tracked part, the respective part's part-path-data-structure has information on the path up to the reference frame. As more frames are accepted by the method following the reference frame, the method updates all part paths existing in that reference frame's part-path-data-structure, and further adds new part paths in response to previously untracked parts being encountered in accepted frames. In one embodiment, the updating of the part-path-data-structure continues for a time interval equal to the FIFO interval, starting with the reference frame, so that in this example, the updating, including new part paths, possibly deleting some part paths, continues for the next 60 frames. In other words, updating of a reference frame's part-path-data-structure continues for as long as the subject reference frame is in FIFO 523, that is, until the reference frame is the oldest reference frame in FIFO 523. Thus, in one embodiment, a single part may have five tracks of up to 60 coordinates. If for any reason, the longest track is not available, one of the other tracks can be used.
In our example of
Classification
Subprocesses 103 and 105 determine tracked paths for parts of objects. Some of these objects are moving, and hence any tracked parts thereof are movers, while others are statics. Refer again to
In 531, subprocess 107 calculates a set of path descriptors (path features), representing object motion from the path tracks from subprocess 105. Note that in some embodiments, subprocesses 513 and 531 are combined into a step which both computes paths in the reference frame, and computes the path features for classification.
In one embodiment, 531 includes computing the following path features:
Path Elongation (Denoted f1)
In one embodiment, a path of a length denoted pathLength (the number of frames in a path) is represented by a matrix of dimension 2×pathLength, where the first and the second rows represent the x-coordinates and y-coordinates, respectively of the path. The subprocess 107 determines a dominant direction for the path by applying PCA (Principle Component Analysis) on the path matrix, which determines two eigenvectors and corresponding eigenvalues λmin and λmax. One feature we calculate is a measure of the elongation of the path, defined as:
f1=λmin/λmax.
From
Direction Changes (Zero-Crossings, Denoted f2)
Compared to moving at a truly fixed velocity (speed and direction), a part typically does change its direction relative to the dominant direction. If we consider the line of dominant direction, and consider the velocity relative to the dominant direction, the velocity vector projected onto the dominant direction line, one can see that the direction, e.g., as measured by the angle of the path to the dominant direction line, the angle crosses zero many times. The number of such zero crossings divided normalized to per frame, is a feature of the path that may be included for mover/static classification. We define the normalized zero-crossing path feature:
f2=(number_of_zero_crosses)/(pathLength),
where number_of_zero_crosses is the number of zero crossings of the angle of the velocity vector relative to the line of dominant direction.
Mean Velocity (Denoted f3)
The mean velocity f3 of the path of a part may be expressed in pixels/frame or pixels per second, and determined as the mean value of the projection of the velocity vector on to the dominant direction line, i.e., the mean speed in the dominant direction. In one embodiment, we determine the velocity using a Kalman filter, while in another embodiment; we use the common Laplacian of Gaussian technique of smoothing the part path by a Gaussian filter, and calculating the derivative of the smoothed path to obtain a velocity vector at each respective frame of the path. Denoting (for this subprocess) by scalar vj, j=1, . . . , pathLength the dominant-direction speeds in the path at frame j, one path feature useful for classification is the mean value of the speed:
f3=(Σj=1pathLengthvj)/(pathLength).
Velocity-Variance-Based Feature (Denoted f4)
Given the speeds in the dominant direction, another useful measure is one related to the deviation in the speed in the dominant direction. In one embodiment, this is expressed in units of velocity as
f4=Σj=1pathLength(vj−f3)2/[(pathLength−1)×f3].
Alternate embodiments might use a different dominant-direction-speed-deviation measure. One such measure is the variance of the dominant-direction speeds in the path. One would expect a mover to have low deviation in speed. In one embodiment, the velocities are normalized by the mean velocity value f3, so that the first alternative velocity variance, denoted f4′, is
f4′=Σj=1pathLength{(vj−f3)2}/[(pathLength−1)2×f32].
In another embodiment, a yet a different alternative dominant-direction-speed deviation measure is used. For this, rather than the mean velocity, the median of the dominant-direction speed, denoted vmed is used, and the measure of deviation is the mean (normalized) L1 norm of the dominant-direction speed:
deviation(v)=Σj=1pathLength|vj−vmed|/(pathLength×vmed).
Path Monotonicity (Denoted f5)
This path feature measures monotonicity trend of the sequence of Euclidean distances from the starting point of the path, denoted di, i=1 . . . pathLength, i.e., the sequence
{di},i=1 . . . pathLength.
The rationale behind this region feature is that for short time period (1-2 seconds) of frames, movers will increase distance from starting point in (almost) every step, while static object will oscillate around the starting point. Thus, even paths with larger number of zero crosses can be classified as moving objects. Monotonic trend of the sequence di can be calculated by Kendall's tau rank correlation coefficient. The idea is to calculate Kendall's tau (denoted τ(•)) for the sequence {di} and some pre-defined monotonically increasing sequence {ai}, for example, ai=i. The monotonicity feature f5 is defined as
f5=τ({di},{ai})
Background (f6)
Another feature we use for a path is the largest (over the path) average difference of the intensity in the part's region and the background of within the region occupied by the part, such average difference normalized by the number of pixels in the part. In more detail, for each input frame k, denote by Bk(x,y) the frame's intensity values without any of the moving parts, wherein x,y denotes pixel coordinates. We call Bk(x,y) the background, and it may be estimated using one of many standard techniques. One embodiment estimates the background using temporal median filtering of a plurality of input frame. In one version, Bk(x,y) is determined by median filtering of the KR (e.g., 5) reference frames in the FIFO. For any part that has a region defined by a set of pixel coordinates, we are interested in the difference between the part's intensity and that of the background to the part. Consider a particular part indexed by j, denote the j'th part's region in frame k by Rkj, and denote the number of pixels in the part by N(Rkj). The part's shape and size can change from frame to frame along the part's path. One path feature used for tracking is, for the track of part j, the maximum per pixel difference between the part's intensity and that of the background along the path. That is, for part j:
f6=maxk{Σ(x,y)εR
With these path features (also called path descriptors) calculated, the next step 533 of subprocess 107 carries classification, e.g., machine-learning-based classification using a decision tree-based classifier or some other classification method. Many methods are known to classify samples (tracks in our case) as being included in one class (inliers, in this case statics) or another class (the outliers in this case the movers) based on a set of features, which in one embodiment are the six path features. While classifiers are known within the field of machine learning, many of these methods were also known under the name of statistical pattern recognition.
In one embodiment, the classifying subprocess 533 uses the six features and training data, i.e., 533 uses a method of supervised learning. In one embodiment, classifying subprocess 533 uses a Support Vector Machine (SVM) classifier to classify each path of a part into inliers (statics) and outliers (movers). SVM classifiers are well known in the art for solving two-class classification. The training (535) uses videos that have moving objects with tracks similar to the type of videos used by the video tracking method. Thus, different SVMs would be trained for different types on input videos. In one example, we used over 7000 samples of inliers and outliers, e.g., from a training set of aerial videos, to train the SVM.
Classifying subprocess 107 identifies regions (parts of objects) that are movers for further processing in order to group the moving parts into objects.
Clustering
More than one of the parts classified as movers may belong to the same object, and the purpose of clustering is to group together those parts into moving objects.
Connected Part Analysis
To carry out connected part analysis, subprocess 603 constructs a weighted undirected graph denoted (V,E)k for each frame denoted by index k, where V is a set of vertices and E is a set of edges. This is carried out frame by frame. Denote each moving part from subprocess 107 by pi, with different index i indicating a different part in the frame, so that all values of i cover all movers in frame k. In the connectedness graph, each part pi corresponds to one vertex in V. An edge between parts pi and pj exists in E if the probability that parts pi and pj belong to the same object is greater than some threshold denoted ⊖. Each edge E is given a weight wi,j which is indicative of the probability (>⊖) that parts pj and pj belong to the same object. By labeling is meant assigning an object (a label) to a part.
One aspect of the invention is the use of features, called connectedness features herein, between two parts to connect parts that (likely) belong to the same object based on such features as proximity and speed. The reason we use the term likely is that we also improve on this connection be considering previous frames' labelings. Another aspect is that the connectedness features are normalized, such that the method is robust to scale change. For a current frame k, connected part analysis comprises determining the connectedness values, and estimating wi,j from the connectedness features using supervised learning. The following connectedness features between moving parts pi and pj in the frame are determined:
Velocity-Magnitude Difference (c1)
We define vector vi as the velocity of the part pi in the current frame, and note that such vector vi can be expressed as a magnitude (also called intensity) denoted vInti and an angle denoted vAnglei (the angle between velocity vector vi of the part pi in the current frame and positive part of the X-axis). In one embodiment, the velocity vi is estimated from the path history of the part pi using a Kalman filter within a path. In one embodiment, a velocity-magnitude difference connectedness feature c1 is calculated as:
where |•| denotes absolute value.
Velocity Angle Difference (c2)
In one embodiment, another connectedness feature we determine is the cosine of the difference in velocity angles between two parts, calculated as:
c2=cos(vAnglei−vAnglej).
Overlap Area (c3)
Define bi as the axes-aligned bounding-rectangle of a part pi and define area (bi) to be the area of such rectangle. Define the axes-aligned rectangle bij=bi∩bj as measure of the overlap region between parts pi and pj. The overlap area feature c3 is calculated as:
c3=area(bij)/min(area(bi),area(bj)).
Rectangle Distance (c4)
We denote the distance between two arbitrary points α and β as d(α, β) calculated as the Euclidian distance. The rectangle distance between two bounding rectangles bi and bj, denoted as d(bi,bj), is defined as the minimum distance between any two points in the two respective bounding rectangles of parts pi and pj, respectively, i.e.,
d(bi,bj)=min(d(α,β)|αεbi,βεbj).
Additionally, we denote by d(bi) the length of the diagonal of the bounding rectangle bi. Feature c4 is calculated as:
c4=d(bi,bj)/max(d(bi),d(bj)).
Background Difference (c5)
We define R(bi,bj) a region (called the connecting region) between two bounding rectangles bi and bj of parts pi and pj. Each of
c5=Σ(x,y)εR(b
where I(x,y) is the current frame intensity level at pixel coordinates x and y in the region R(bi,bj), and G(x,y) is the estimated background at pixel coordinates x and y (in the region R(bi,bj)).
Overall Connectedness Probability
With the connectedness feature set ci,j=(c1, c2, c3, c4, c5) between pairs of parts calculated, wi,j is calculated as a probability measure that is a function of the five connectedness features. For this, we use a set of manually labeled objects and parts thereof (a labeled data set) to determine the distribution of values of the five features, respectively for the case of parts being in, and not in the same object, and use the distributions to estimate wi,j in a supervised manner, i.e., using supervised learning. In more detail, the method includes a training phase comprising first discretizing any features that are in the continuous domain, and building two 5-dimensional ensembles (5-dimensional histograms) from the labeled data set, each dimension corresponding to one of the five features c1, c2, c3, c4, c5. Based on labeled dataset, we build two sets of histograms of the five features, one set for YES denoting the cases of parts i and j being in the same object, and another set for NO, denoting parts i and j not in same object. Each dimension corresponds to one of features c1, c2, c3, c4, c5. Hence there is a 5-dimensional histogram for estimating joint probability distribution for YES, and a 5-dimensional histogram for NO. In the training phase, we analyze our labeled dataset and for each of the 5-dimensional vectors ci,j and the objects of the parts i and j, we estimate corresponding bin for one of the 5-dimensional histograms. If parts i and j are in the same object, we update the YES histogram, otherwise we update the NO histogram. In the second phase, we calculate c1, c2, c3, c4, c5. to form a vector ci,j, we discretize each dimension to determine to which bin of the five-dimensional histograms the elements of vector ci,j correspond. The measure of probability wi,j is calculated as number of elements in that bin of the YES histogram divided by sum of number of elements in that bin for the YES and the NO histograms.
Matching Moving Objects from Frame to Frame
As a result of imperfect identification of moving objects in the connected part analysis of each frame, parts which belong to the same (moving) object in one frame might get separated into different objects in the next frame. Likewise, parts from different moving objects in one frame could be joined into one or more single objects in a later frame. The step 603 of matching moving objects is carried out after initial identification of potential moving objects to create correspondences between each object in the current frame and objects in the previous frame. Only then does the subprocess propagate object identifiers from frame to frame.
Defining Object Weights Between Frames for Matching
Based on the history of part-to-object association, define kti,j to be the weight between two objects, object i in previous frame k−1, denoted Objectk−1,i and object j in the current frame k that immediately follows frame k−1, such object denoted as Objectk,j. The weight kti,j is calculated as follows:
kti,j=Σmε
where
These numbers of frames are added up for all parts m in kMi∩j to determine the weight kti,j for current frame k.
After obtaining all weights kti,j for all pairs of objects i and j and the parts in the frame k, as described, one embodiment uses bipartite matching to find best correspondences for object ID propagation.
The four objects shown in frame k denoted, (Objectk,1),(Objectk,2), (Objectk,3) and (Objectk,4) show the results of connected analysis on frame k, thus the labeling of the parts are candidate object labeling.
Frame k's candidate object 1, (Objectk,1) includes two parts 821 and 822 that are matched to parts 811 and 812 labeled in frame k−1 as in object 1. (Objectk,1) further includes a new part 823 that does not seem to be connected to any part in the previous frame. Candidate object 2 (Objectk,2) has parts 824 and 825, and these are matched to parts 814 and 815, respectively, from the previous object 2 (Objectk−1,2), hence Objectk,2 is connected to Objectk−1,2 with a weight denoted t2,2. Candidate object 3 (Objectk,3) has two parts 826 and 827, and is connected (part 826) to one part 816 from the previous object 2 (Objectk−1,2), and via part 827 to a part 817 from the previous object 3 (Objectk−1,3). Frame k has a completely new candidate object 4 (Objectk,4) with a new part 828.
As more and more frames are matched to a previous frame, and after bipartite matching, we end up with drawing like
Short-Interval-Groupings Filtering
Finding connected components in one frame can result in imperfect grouping of parts into objects, e.g., in the next frame. However, the inventors have found that the history of grouping in past frames can be used to improve results, e.g., to correctly regroup parts of some of objects that were misgrouped in a current frame.
We call the subprocess 605 of correcting erroneous groupings “short-interval-groupings filtering” (“SIG filtering”) to reflect its purpose of eliminating some short interval erroneous groupings.
In one embodiment, SIG filtering 605 is implemented using hysteresis filtering, which uses two hysteresis thresholds, one when something decreases, and a second for increasing values. We define a stability level to be a measure of how consistent labeling is from frame to frame. Suppose a part in an object has stable labeling from frame to frame. For each new frame, if the part has the same labeling, the stability level for the part increases (up to a saturation level). When a part is encountered whose labeling from one or more previous frames changes, the stability level is decreased down, eventually to a bottom level, below which the labeling of the part changes.
Such approach enables stability of the grouping subprocess 109, preventing parts to incorrectly change an object's parts change from frame to frame which would cause objects to flicker in size and shape.
System Embodiments
A second set of embodiments of the present invention relates to a system for processing video that includes a sequence of frames to determine and track moving objects in the video. The system furthermore may be constructed, by programming and/or specialized hardware modules, to carry out the steps of the method embodiments described above for identifying the different moving objects in each of the frames. An example of such a system 1100 is shown in
One aspect of system 1100 is the inclusion of a part-path clusterer 1109, operative to accept the paths of movers and to group parts that likely belong to the same object in order to generate moving objects and the tracking thereof. Clusterer 1109 when operating determines, for each frame, a set of connectedness features that capture the relationship between pairs of movers in the frame, including features related to the velocity difference (speed and direction), proximity of the parts, overlap between the parts, and background similarity of respective background of the parts. These connectedness features are normalized, such that the system 1100 is robust to scale. The clusterer 1109 is operative to estimate two multi-dimensional joint probability functions for the connectedness features, a first set for feature values of pairs belonging to the same object, and a second set for feature values of pairs not belonging. The estimated joint probability function sets are obtained by supervised learning using a training set of videos containing static and moving parts that have been manually grouped into objects, i.e., that are pre-labeled. The clusterer 1109 uses these estimated probability functions to determine a measure of probability of any two pairs of parts belonging to the same object, with any probability measure less than a pre-defined or user selected threshold being indicative of no connection.
Once the analysis is carried out for all pairs of moving objects in the current frame, leading to a candidate set of groupings for the current frame, Part-path clusterer 1109 is further operative to carry out matching of moving objects between frames by determining correspondences between each candidate object in the current frame with objects in the previous frame. To do this, the clusterer 1109 determines when operating a measure of correspondence from the current frame's candidate groupings to past object groupings based on the numbers of parts in a candidate object that are in objects that contain the respective parts in all previous frame. Once the weightings to previous objects are determined for all pairs of objects in the current frame, clusterer 1109 is operative to carry out bipartite matching to find the best correspondences for propagation of object groupings from frame to frame.
Another aspect of system 1100 is the inclusion in clusterer 1109 of a short interval grouping filter that can eliminate some short interval erroneous groupings, for example an erroneous grouping when two overtaking objects are fused in the short term to a single object, or when an object is temporarily and erroneously split into two objects before being labeled later as a single object. The short term interval grouping filter uses hysteresis filtering of an objects stability to ensure stable labeling.
One or more of different components of system 1100 may each comprise a processing engine for performing the respective component's function. The functionality of the different components of the system 1100 or different method steps of the method 100 may be implemented in separate processing units, or a processing system.
Storage subsystem 1211 includes, as is common, software to enable all the elements to work together, shown as operating system SW 1215. The memory of the storage subsystem 1211 may at some time hold part or all of a set of instructions shown as 1231 that when executed on the processing system 1200 implement the steps of the method embodiments described herein, e.g., method 100. In some versions, these instructions may be partitioned (of course with many common elements, and using common data structures) as part detection and tracking software (“SW”) 1233 that, in some versions that include parallel MSER calculator 1205, use element 1205, part matching tracking and SW 1235, classification of tracked parts SW 1237 that when operating, classifies tracks to movers and statics, and mover clustering SW 1239 that groups parts into object in one frame, then connects these objects with previously grouped objects, and then carried out filtering to eliminate erroneous short term groupings. Furthermore, the storage subsystem 1211, when the system 1200 is operational, may include one or more image(s) or portions thereof, track data 1243, parameters and training data 1245, and reference images data 1247.
Note that while a processing system 1200 such as shown in
Computer-Readable Medium Embodiments
Yet another set of embodiments include a non-transitory machine-readable medium that stores instructions that when executed by one or more processors of a processing system, such as system 1200, causes carrying out a method as described in any of the method embodiments described herein, e.g., method 100 and its variations.
Application System Embodiments
In another aspect, the present invention relates to an application system, such as for example a computerized system for an automotive applications, e.g., to include with a controller in an automobile, or such as an airborne system, or a surveillance system, or any other system or automated system which may use information regarding moving objects in its surrounding. The application system comprises the system as described in
Thus, one or more aspects of the present invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof.
General
In this description, it is assumed that a video frame when digitized contains a two-dimensional arrangement (an array) of pixels each having a gray-scale value. The invention however is not limited to monochrome images, or even visual images. For example, the methods described herein may be used for infrared (IR) images, for multispectral images as used in remote sensing, color images, and so forth. As one example, each pixel of a color image has typically has three coordinates that may be in, or readily transformed to first coordinate that is a measure of intensity or of luminance, and one or more other coordinates that provide the color information, e.g., as a measure of chrominance or of hue and saturation. The methods described herein can be used on one or more of these coordinates.
While one embodiment uses the weighted Euclidean norm as the distance measure for matching in the matching and tracking subprocess 105, other embodiments may use a different distance measure. One embodiment uses the weighted sum of absolute distances (the weighted L1 norm), another uses the weighted L∞ norm, yet another uses the squared weighted Euclidean distance, and yet another uses the (unweighted) Euclidean norm.
While embodiments described herein use specific data structures such as lists, vectors, and other data structures, such structures are described as one example, and the invention is in no way limited to the type of data structure used in the example.
While one embodiment of the clustering and clusterer uses five specific connectedness features, alternate embodiments may use a different number of connectedness features, and/or connectedness features that are defined differently.
While one embodiment of the tracking uses six specific path features, alternate embodiments may use a different number of path features, and/or path features that are defined differently.
While the term subprocess is used to describe a part of a method, this does not indicate that this is in any way related to sub-procedures, subroutines, or any other distinct elements of a computer program, but rather to a step in the method (or process).
Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “ascertaining,” “analyzing,” or the like, refer to the action and/or processes of a device, a processor, a computing system, or similar electronic computing device, that manipulate and/or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.
In a similar manner, the term “processor” may refer to any device or portion of a device that processes electronic data, e.g., from registers and/or memory to transform that electronic data into other electronic data that, e.g., may be stored in registers and/or memory.
The methodologies described herein are, in one embodiment, performable by one or more processors that accept machine-readable instructions, e.g., as firmware or as software, that when executed by one or more of the processors carry out at least one of the methods described herein. In such embodiments, any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken may be included. Thus, one example is a programmable DSP device. Another is the CPU of a microprocessor or other computer-device, or the processing part of a larger ASIC. A processing system may include a memory subsystem including main RAM and/or a static RAM, and/or ROM. A bus subsystem may be included for communicating between the components. The processing system further may be a distributed processing system with processors coupled wirelessly or otherwise, e.g., by a network. If the processing system requires a display, such a display may be included. The processing system in some configurations may include a sound input device, a sound output device, and a network interface device. The memory subsystem thus includes a machine-readable non-transitory medium that is coded with, i.e., has stored therein a set of instructions to cause performing, when executed by one or more processors, one of more of the methods described herein. Note that when the method includes several elements, e.g., several steps, no ordering of such elements is implied, unless specifically stated. The instructions may reside in the hard disk, or may also reside, completely or at least partially, within the RAM and/or other elements within the processor during execution thereof by the system. Thus, the memory and the processor also constitute the non-transitory machine-readable medium with the instructions.
Furthermore, a non-transitory machine-readable medium may form a software product. For example, it may be that the instructions to carry out some of the methods, and thus form all or some elements of the inventive system or apparatus, may be stored as firmware. A software product may be available that contains the firmware, and that may be used to “flash” the firmware.
Note that while some diagram(s) only show(s) a single processor and a single memory that stores the machine-readable instructions, those in the art will understand that many of the components described above are included, but not explicitly shown or described in order not to obscure the inventive aspect. For example, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.
Thus, one embodiment of each of the methods described herein is in the form of a non-transitory machine-readable medium coded with, i.e., having stored therein a set of instructions for execution on one or more processors, e.g., one or more processors that are part of the receiver forming a pen stroke capture system.
Note that, as is understood in the art, a machine with application-specific firmware for carrying out one or more aspects of the invention becomes a special purpose machine that is modified by the firmware to carry out one or more aspects of the invention. This is different than a general purpose processing system using software, as the machine is especially configured to carry out the one or more aspects. Furthermore, as would be known to one skilled in the art, if the number the units to be produced justifies the cost, any set of instructions in combination with elements such as the processor may be readily converted into a special purpose ASIC or custom integrated circuit. Methodologies and software have existed for years that accept the set of instructions and particulars of, for example, the processing engine 131, and automatically or mostly automatically great a design of special-purpose hardware, e.g., generate instructions to modify a gate array or similar programmable logic, or that generate an integrated circuit to carry out the functionality previously carried out by the set of instructions. Thus, as will be appreciated by those skilled in the art, embodiments of the present invention may be embodied as a method, an apparatus such as a special purpose apparatus, an apparatus such as a data DSP device plus firmware, or a non-transitory machine-readable medium. The machine-readable carrier medium carries host device readable code including a set of instructions that when executed on one or more processors cause the processor or processors to implement a method. Accordingly, aspects of the present invention may take the form of a method, an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form a computer program product on a non-transitory machine-readable storage medium encoded with machine-executable instructions.
Reference throughout this specification to “one embodiment,” “an embodiment,” some embodiments,” “or “embodiments” means that a particular feature, structure or characteristic described in connection with the embodiment or embodiments is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.
Similarly it should be appreciated that in the above description of example embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, drawing, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.
Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.
Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a host device system or by other means of carrying out the function. Thus, a processor with the necessary instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the invention.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
As used herein, unless otherwise specified the use of the ordinal adjectives “first”, “second”, “third”, etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
All publications, patents, and patent applications cited herein are hereby incorporated by reference, except in those jurisdictions where such incorporation by references is not permitted.
Any discussion of prior art in this specification should in no way be considered an admission that such prior art is widely known, is publicly known, or forms part of the general knowledge in the field.
In the claims below and the description herein, any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements/features that follow, but not excluding others. Thus, the term comprising, when used in the claims, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements/features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.
Similarly, it is to be noticed that the term coupled, when used in the claims, should not be interpreted as being limitative to direct connections only. The terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Thus, the scope of the expression a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means. “Coupled” may mean that two or more elements are either in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.
Any formulas given above are merely representative of procedures that may be used. Functionality may be added or deleted from the block diagrams, operations may be interchanged among functional blocks or flow chart elements, and steps may be added or deleted to described methods.
Thus, while there has been described what are believed to be the preferred embodiments of the invention, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the scope of the invention as defined by the claims attached hereto, and it is intended also to claim all such changes and modifications as fall within the scope of the invention.
The present invention claims priority of U.S. Provisional Patent Applications Nos. U.S. 62/241,132, filed 2015 Oct. 13 and U.S. 62/245,874, filed 2015 Oct. 23, each such application to Applicant Motion DSP, Inc. with inventors Andelković, et al. The contents of each of U.S. 62/241,132 and U.S. 62/245,874 are incorporated herein by reference, except in any jurisdiction where incorporation by reference is not permitted. In such jurisdictions, any portion of either of both of said U.S. 62/241,132 and U.S. 62/245,874 may be inserted into the present application by amendment.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6263088 | Crabtree | Jul 2001 | B1 |
| 6295367 | Crabtree | Sep 2001 | B1 |
| 6441846 | Carlbom | Aug 2002 | B1 |
| 6724915 | Toklu | Apr 2004 | B1 |
| 7519220 | Aharon et al. | Apr 2009 | B2 |
| 7697756 | Aharon et al. | Apr 2010 | B2 |
| 7706633 | Chefd'hotel et al. | Apr 2010 | B2 |
| 7725484 | Nister et al. | May 2010 | B2 |
| 8050454 | Yi | Nov 2011 | B2 |
| 8300935 | Distante | Oct 2012 | B2 |
| 8345769 | Diard et al. | Jan 2013 | B1 |
| 8358381 | Diard et al. | Jan 2013 | B1 |
| 8437558 | Medasani | May 2013 | B1 |
| 8737684 | Meloche | May 2014 | B2 |
| 9002055 | Funayama | Apr 2015 | B2 |
| 9396396 | Hasegawa et al. | Jul 2016 | B2 |
| 9589595 | Gao | Mar 2017 | B2 |
| 9679203 | Bulan | Jun 2017 | B2 |
| 20050225678 | Zisserman et al. | Oct 2005 | A1 |
| 20050226506 | Aharon et al. | Oct 2005 | A1 |
| 20050271302 | Khamene et al. | Dec 2005 | A1 |
| 20060104510 | Aharon et al. | May 2006 | A1 |
| 20070291040 | Bakalash et al. | Dec 2007 | A1 |
| 20090232358 | Cross et al. | Sep 2009 | A1 |
| 20100195914 | Isard et al. | Aug 2010 | A1 |
| 20110313984 | Zhou et al. | Dec 2011 | A1 |
| 20120288140 | Hauptmann | Nov 2012 | A1 |
| 20120314961 | Isard et al. | Dec 2012 | A1 |
| 20140003709 | Ranganathan et al. | Jan 2014 | A1 |
| 20140023270 | Baheti et al. | Jan 2014 | A1 |
| 20140023271 | Baheti et al. | Jan 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 101770568 | Jul 2010 | CN |
| Entry |
|---|
| Bhattacharya, Subhabrata, Haroonldrees, Imran Saleemi, Saad Ali, and Mubarak Shah. “Moving object detection and tracking in forward looking infra-red aerial imagery.” In Machine Vision Beyond Visible Spectrum, pp. 221-252. Springer Berlin Heidelberg, 2011. |
| Blei D. M., A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” J. Mach. Learning Res. 3, 993-1022 (2003). |
| Brown, Andrew P., Kevin J. Sullivan, and David J. Miller. “Feature-aided multiple target tracking in the image plane.” In Defense and Security Symposium, pp. 62290Q-62290Q. International Society for Optics and Photonics, 2006. |
| Brox, Thomas, and Jitendra Malik. “Object segmentation by long term analysis of point trajectories.” In Computer Vision—ECCV 2010, pp. 282-295. Springer Berlin Heidelberg, 2010. |
| Chang, Chih-Chung and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol., vol. 2, No. 3, Article 27 (May 2011). |
| Coifman, Benjamin, David Beymer, Philip McLauchlan, and Jitendra Malik. “A real-time computer vision system for vehicle tracking and traffic surveillance.”Transportation Research Part C: Emerging Technologies 6, No. 4 (1998): 271-288. |
| Danescu, R. , S. Nedevschi, and M. Meineche, “Lane geometry estimation in urban environments using a stereovision system,” Proc. IEEE Transportation Systems Conf., Seattle, pp. 271-276 (2007). |
| Donoser, Michael, and Horst Bischof. “Efficient maximally stable extremal region (MSER) tracking.” In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1, pp. 553-560. IEEE, 2006. |
| El-Sheimym, N. and K. Schwarz, “Navigating urban areas by VISAT—a mobile mapping system integrating GPS/INS/Digital cameras for GIS application,” Navigation 45, 275-286 (1999). |
| Forssén, Per-Erik, and David G. Lowe. “Shape descriptors for maximally stable extremal regions.” In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, pp. 1-8. IEEE, 2007. |
| Hofmann, T., “Unsupervised learning by probabilistic latent semantic analysis,” Mach. Learning 42, 177-196 (2001). |
| John Hoperoft and Richard Carp, “An n^{5/2} algorithm for maximum matching in bipartite graphs,” SIAM J. Computing, vol. 2, No. 5, pp. 225-231, Dec. 1973. |
| Iqbal, Kamran, Xu-Cheng Yin, Xuwang Yin, Hamza Ali, and Hong-Wei Hao. “Classifier comparison for MSER-based text classification in scene images.” InNeural Networks (IJCNN), The 2013 International Joint Conference on, pp. 1-6. IEEE, 2013. |
| Jones, R., B Ristic, N.J. Redding, D.M. Booth, “Moving Target Indication and Tracking from Moving Sensors,” Digital Image Computing: Techniques and Applications, 2005. DICTA '05. Proceedings 2005 , pp. 46,46, Dec. 6-8, 2005. |
| Jones, R, D. M. Booth, and N.J. Redding, “Video moving target indication in the Analysts' Detection Support System,” Tech. Report No. DSTO-RR-0306. Intelligence Surveillance and Reconnaissance Div, Defence Science and Technology Organisation (DSTO), Edinburgh, South Australia (Australia) 2006. |
| Kang, Jinman, Isaac Cohen, Gérard Medioni, and Chang Yuan. “Detection and tracking of moving objects from a moving platform in presence of strong parallax.” In Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on, vol. 1, pp. 10-17. IEEE, 2005. |
| Kelman, A., M. Sofka, and C. V. Stewart, “Keypoint descriptors for matching across multiple image modalities and non-linear intensity variations,” in Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, Minneapolis, pp. 1-7 (2007). |
| Kim, Z., “Robust lane detection and tracking in challenging scenarios,” IEEE Trans. Intell. Transp. Syst. 9, 16-26 (2008). |
| Kreucher K., and S. Lakshmanan, “LANA: a lane extraction algorithm that uses frequency domain features,” Robot. Autom. 15, 343-350 (1999). |
| Lin, Liang, Yongyi Lu, Yan Pan, and Xiaowu Chen. “Integrating graph partitioning and matching for trajectory analysis in video surveillance.” Image Processing, IEEE Transactions on 21, No. 12 (2012): 4844-4857. |
| Liu, D. and T. Chen, “Unsupervised image categorization and object localization using topic models and correspondences between images,” in Proc. of 11th Int. Conf. on Computer Vision, Rio de Janeiro, pp. 1-7 (2007). |
| Lowe, D. G., “Distinctive image features from scale-invariant key-points,” Int. J. Comput. Vis. 60, 91-110 (2004). |
| Lu, Wei-Lwun, Jo-Anne Ting, Kevin P. Murphy, and James J. Little. “Identifying players in broadcast sports videos using conditional random fields.” InComputer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 3249-3256. IEEE, 2011. |
| Martinez-Garcia, Edgar A., and Dulce Torres. Robot Visual Odometry by Tracking, Rossum 2011 Jun. 27-28,2011 Xalapa, Ver., Mexico. Retrieved Jul. 23, 2015 from http://www.uv.mx/rossum/CD/papers/029.pdf. |
| Matas, J., O. Chum, M. Urban, and T. Pajdla, “Robust wide baseline stereo from maximally stable extremal regions,” in Proc. of 13th British Machine Vision Conf., Cardiff, Wales, pp. 384-393 (2002). |
| McCall, J. C., and M. M. Trivedi, “Video-based lane estimation and tracking for driver assistance: survey, system, and evaluation,” IEEE Trans. Intell. Transp. Syst. 7, 20-37 (2006). |
| Matas, Jiri and Karel Zimmermann “A New Class of Learnable Detectors for Categorisation,” In Proceedings of 14th Scandianavian Conference on Image Analysis (SCIA'05), pp. 541-550, LNCS 3540,2005. |
| Mikolajczyk, K. and C. Schmid, “A performance evaluation of local descriptors,” IEEE Trans. Pattern Anal. Mach. Intell. 27,1615-1630 (2005). |
| Mikolajczyk, K., T. Tuytelaars, C. Schmid, A. Zisserman, and J. Matas, “A comparison of affine region detectors,” Int. J. Comput. Vis. 65, 43-72 (2005). |
| Niebles, J. C., H. Wang, and L. Fei-Fei, “Unsupervised learning of human action categories using spatial-temporal words,” Int. J. Comput. Vis. 79, 299-318 (2008). |
| Obdrázlek, S., “Object recognition using local affine frames,” PhD thesis, Center for Machine Perception, Faculty of Electrical Engineering, Czech Technical University, Prague, Czech Republic, Sep. 2006. |
| Obdrázlek,,Stepan, and Jiri Matas. “Object Recognition using Local Affine Frames on Distinguished Regions.” In BMVC, vol. 1, p. 3. 2002. |
| Rao, Shankar R., Roberto Tron, René Vidal, and Yi Ma. “Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories.” In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1-8. IEEE, 2008. |
| Samadzadegan, F. , A. Sarafraz, and M. Tabibi, “Automatic lane detection in image sequences for vision-based navigation purposes,” in Proc. of Int. Soc. for Photogrammetry and Remote Sensing Commission V Symp., Dresden, pp. 251-257 (2006). |
| Sivic, J., B. Russell, A. Efros, A. Zisserman, and W. Freeman. “Discovering objects and their location in images,” in Proc. of 10th IEEE Int. Conf. on Computer Vision, Beijing, pp. 370-377 (2005). |
| Wang, X., X.Ma, and W.E. Grimson,“Unsupervised activity perception in crowed and complicated scenes using hierarchical Bayesian models,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 539-555 (2009). |
| Wills, Josh, Sameer Agarwal, and Serge Belongie. “What went where [motion segmentation].” In Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, vol. 1, pp. I-37. IEEE, 2003. |
| Yilmaz, Alper, Omar Javed, and Mubarak Shah. “Object tracking: A survey.”Acm computing surveys (CSUR) 38.4 (2006): vol. 13. |
| Yasin, Hashim, BjörnKrüger, and Andreas Weber. “3D Reconstruction of Human Motion from Video.” (2013). Retrieved Aug. 26, 2016 from http://cg.cs.uni-bonn.de/aigaion2root/attachments/yasin2013b.pdf. |
| Fauqueur, Julien, Gabriel Brostow, and Roberto Cipolla. “Assisted video object labeling by joint tracking of regions and keypoints.” Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. IEEE. 2007. |
| “Kanade-Lucas-Tomasi feature tracker” available at https://en.wikipedia.org/wiki/Kanade-Lucas-omasi—feature—tracker. |
| Oh, Il-Seok, Jinseon Lee, and Aditi Majumder. “Multi-scale image segmentation using MSER.” International Conference on Computer Analysis of Images and Patterns. Springer Berlin Heidelberg, 2013. |
| Riemenschneider, Hayko, “Online Object Recognition using MSER Tracking,” Master's Thesis, Graz University of Technology, Erzherzog-Johann-Universit at Graz, Jan. 2008, available Feb. 14, 2017 at www.vision.ee.ethz.ch/˜rhayko/paper/thesis2008—master—riemenschneider.pdf. |
| Duc Phu Chau, Francois Bremond, Monique Thonnat, and Etienne Corvee,“Robust Mobile Object Tracking Based on Multiple Feature Similarityand Trajectory Filtering,” The International Conference on Computer Vision Theory and Applications (VISAPP) (2011). |
| Michael Teutsch, Thomas M¨uller, Marco Huber, and J¨urgen Beyerer , “Low Resolution Person Detection with a Moving Thermal Infrared Camera by Hot Spot Classification,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2014, pp. 209-216. |
| Vedaldi, Andrea. “An implementation of multi-dimensional maximally stable extremal regions.” Feb. 7, 2007: 1-7, retrieved Mar. 14, 2017 at https://pdfs.semanticscholar.org/ae34/c2f21184a7b70e18c0094037482ff33986c5.pdf. |
| Mikolajczyk, Krystian, Tinne Tuytelaars, Cordelia Schmid, Andrew Zisserman, Jiri Matas, Frederik Schaffalitzky, Timor Kadir, and Luc Van Gool. “A comparison of affine region detectors.” International journal of computer vision 65, No. 1-2 (2005): 43-72. |
| L. Zhang, G. J. Dai, C. J. Wang, “Human Tracking Method Based on Maximally Stable Extremal Regions with Multi-Cameras”, Applied Mechanics and Materials, vols. 44-47, pp. 3681-3686, Dec. 2010. |
| Number | Date | Country | |
|---|---|---|---|
| 62241132 | Oct 2015 | US | |
| 62245874 | Oct 2015 | US |
