The present invention relates to moving object segmentation in a video sequence, and in particular to moving object segmentation utilizing multiple stages to reduce computational load.
Detection of events happening in a monitored area is very important to providing security of the area. Some typical areas include large open spaces like parking lots, plazas, airport terminals, crossroads, large industrial plant floors, airport gates and other outdoor and indoor areas. Humans can monitor an area, and easily determine events that might be important to security. Such events include human and vehicular traffic intrusions and departures from a fixed site. The events can be used in an analysis of traffic and human movement patterns that can be informative and valuable for security applications.
A variety of moving object segmentation techniques have been used to detect events in an area. For fixed/static cameras some techniques utilize: temporal differencing, background separation and adaptive background separation. Temporal differencing is adaptive to dynamic changes but usually fails to extract all the relevant objects, and can be easily confused by environmental nuances such as cast shadows, and ambient light changes.
Background separation provides more reliable solution than many temporal solutions, but is extremely sensitive to dynamic scene changes. A standard method of constructing an adaptive background for a dynamic scene is averaging the frames over time, creating a background approximation that is similar to the current static scene except where motion occurs. While this is effective in situations where objects move continuously and the background is visible a significant portion of the time, it is not robust to scenes with many moving objects particularly if they move slowly. It also cannot handle bimodal backgrounds, recovers the background slowly when it is uncovered, and has a single, predetermined threshold for the entire scene.
Changes in scene lighting can cause problems for the motion detection methods. One background method models each pixel with a Kalman Filter which makes the system more robust to lighting changes and cast shadows in a typical scene. While this method is based on a pixel-wise automatic thresholding for adaptation, it still recovers the background slowly and does not handle bimodal backgrounds well.
A further method includes implementation of a pixel-wise Expectation Maximization (EM) framework for detection of vehicles. The technical approach of this method attempts to explicitly classify the pixel values into three separate predefined distributions representing the background, foreground, and noise. Another more advanced moving object detection method is based on a mixture of normal representations at the pixel level. This mixture modeling method distributions reflects the expectation that more than one background characteristics at each pixel may be observed over time. This approach is well suited for outdoor applications with dynamic scenes.
Some previous approaches simply model the values of a particular pixel as a mixture of Gaussians. Based on the persistence and the variance of each of the Gaussians of the mixture, the approach determines which Gaussians correspond to background colors. Pixel values that do not fit the background distributions are considered foreground until their distributions are adapted into the scene and persistently became part of the background representation with sufficient, consistent evidence.
Following a similar approach a further method models each pixel in an image as a mixture of multiple tri-variate normal distributions. The method attempts to explicitly classify the pixel values into 5 weighted distributions, a few of which represent the background and the rest are associated with the foreground. The distributions are continuously updated to account for the dynamic change within the scene. Attempts to mediate the effect of changes in lighting conditions, and other environmental changes (e.g. snow, swaying tree leaves, rain, etc.) are successful, but it is all on the account of using more of the CPU capacity even for low resolution images and at slow frame rates. This computational burden has resulted in a limitation on the system use.
There is a need for a system that is robust and also fast and practical to implement for real-time operations. Robustness refers to detecting consistent true motion of an object while not generating false alarms on noisy movement and variations in illumination, weather, and environmental conditions. Fast processing refers to the capability of a processor to detect all motions in all frames for multiple input sequences.
A method of detecting motion in a monitored area receives video or image frames of the area. A high speed motion detection algorithm is used to remove still frames in which a less than minimal amount of motion is portrayed. The remaining frames are subjected to a high performance motion detection algorithm to detect true motion from noise.
In one embodiment, each frame comprises pixel blocks that have one or more pixels, each block being represented as a single combinatory value (e.g. average or median pixel value) and a variance value. A model of the area is initialized, and comprises multiple weighted distributions for each pixel block. The model is updated differently depending on new frames matching or not matching the model.
A multi-stage process is used for motion segmentation. A first screening stage applies a fast video motion segmentation (VMS) to reject still images that do not portray any motion. A second stage, which is invoked when necessary, applies a robust VMS to detect the true motion of an object. Sequencing, initialization and adaptive updating of the stages is provided by a resource management controller.
In one embodiment, the fast VMS stage is based on intelligent sampling of video frames, operating in a single-pixel or multiple-pixel block mode, in an uncompressed- or compressed-image domain, and simplistic frame differencing approach.
Several approaches are selectively employed to reduce computational resource requirement for the second, robust VMS stage. Color separation is employed where appropriate. Grey pixels are sensitive in the RGB domain and are modeled separately. Since only luminance is required for representing a scene in grey, utilizing the grey model where it provides adequate detectability of motion reduces computational resource requirements.
A further approach employs non-uniform sampling. In other words, a scene may be divided into different areas whose pixels are grouped in different sizes depending on a desired resolution. Smaller pixel-block size areas or blocks are used for areas where motion likely occurs and higher resolution is desired. The number of pixels in a block may also be varied based on depth of field and range to target in order to maintain a consistent object size in pixel.
Adaptive mixture modeling is also provided based on operation environments. A fast technique adapts the number of normal distributions in the mixture modeling. The number of mixture is based on the amount of insignificant changes in the scene and how dynamic that change is. The technique coops well with multi-modal backgrounds (e.g. swaying tree branches, etc), whereas a single normal distribution may be used for stable scenes, especially for indoor applications.
Distributions of Sample Moments, similar to K-means, are used to initialize the models, rather than the cumbersome expectation maximization (EM). In one embodiment, a Look-Up-Table (LUT) registers the indices of frames' clusters for weight updates. Initial weights and distribution values are computed based on a predefined set of N frames. The set is clustered into subsets. Each subset represents the population of each distribution. The weights of each distribution is the ratio of the number of samples per subset over the predefined number of the initialization frames, i.e. N. In contrast to EM approximation, the approach provides more accurate initial statistical support that facilitates fast convergence and more stable performance of the segmentation operations.
A FIFO (first in, first out) procedure is used to update the mixing proportions weights of the mixture models with no thresholds or learning parameters. Weights are updated based on the new counts of samples per each distribution. The first frame entered in the LUT will be excluded out of the record to update the weights of each classes of distributions. When a match is found, the non-matching distribution (representing foreground) counter is set to zero. The LUT is updated by excluding the first index and including the new record. The weights are updated based on the new variation in the subsets. If there is no match, the foreground distribution counter is incremented. Weights of background distributions are kept the same. The smallest distribution is replaced, once an adequate number of consecutive hits of the foreseen foreground is reached. During the replacement of the smallest distribution with the new foreground distribution adapting to be part of the new background, no weights will be updated. Updates will follow afterwards using the above role.
These approaches reduce computational requirements to make the approach well suited to real-time applications.
In the following description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that structural, logical and electrical changes may be made without departing from the scope of the present invention. The following description is, therefore, not to be taken in a limited sense, and the scope of the present invention is defined by the appended claims.
The functions or algorithms described herein are implemented in software or a combination of software and human implemented procedures in one embodiment. The software comprises computer executable instructions stored on computer readable media such as memory or other type of storage devices. The term “computer readable media” is also used to represent carrier waves on which the software is transmitted. Further, such functions correspond to modules, which are software, hardware, firmware or any combination thereof. Multiple functions are performed in one or more modules as desired, and the embodiments described are merely examples. The software is executed on a digital signal processor, ASIC, microprocessor, or other type of processor operating on a computer system, such as a personal computer, server or other computer system.
A multi-stage process is used for motion segmentation. A first screening stage applies a fast video motion segmentation (VMS) to reject still images that do not portray any motion. A second stage, which is invoked when necessary, applies a robust VMS to detect the true motion of an object.
In one example of the invention, three modules work synergistically to provide the optimal functional and computational performances in detecting motion in a video sequence as indicated at 100 in
Operation Controller: Operation Controller (OC) module 110 serves as a resource management unit that maintains the frame rate and functional performance via setting the processing-speed-related parameters. To maintain the frame rate, some video frames will be skipped and not processed. The OC computes a maximum allowable number of frames to be skipped based on the speed of the expected objects, the camera frame rate and the physical area coverage within the field of view (FOV). When the actual processing frame rate falls behind the desirable rate, the OC skips processing up to the maximum allowable number of frames. This is achieved by setting the D1 and D2 decision blocks 150 and 160 respectively to “No Op.” as represented at 165 and 170 in the motion detection modules.
The operation controller also directs the operation flow which consists of initialization and motion detection. Executing the alternative operation flow achieves overall high speed, high performance motion detection and tracking.
Initialization:
The operation controller 110 directs an initialization process for the HSMD and the HPMD during the startup. Another condition to execute the HSMD and the HPMD initialization processes is when the video camera carries out a pan and tilt operation. The OC calls the HSMD initialization when the HPMD completes a motion detection sequence. In addition, the OC also calls the HSMD initialization, when a different HSMD function is selected.
Motion Detection Operation Flow:
The OC sets the conditions in the D1 150 and D2 160 decision blocks, which directs the input and output data to the appropriate processing modules. The D1 decision block directs the input video frame to a decompression module 175, HSMD sub-module 180, or the No Op module 165. The D2 decision block directs the output video frame from stage one to a HPMD sub-module 185 or the No Op module 170. The rules in setting the D1 and D2 blocks are:
Several functions can be applied in the HSMD module. Each function achieves different functional and computational performances. OC selects the proper function based on the desirable frame rate, the object whose motion is of interest, and the scene complexity.
Parameter Setting in the HSMD and HPMD Modules:
OC computes the values of critical functional and computational parameters in the HSMD and HPMD modules. Some of the parameters include the video frame boundary, the number of DCF coefficients to compute HSMD motion detection, and subsample pattern.
The frame size often is so large that number of computations on all pixels is astronomical. Subsamples of the video frame are necessary to maintain the desirable processing frame rate. A uniform sample in both the row and column dimensions is frequently used. Some approaches apply different sample rates in the row and column dimensions. The sample rate, however, is uniform across each direction. In one embodiment, the sampling approach, as illustrated in
High Speed Motion Detection:
The high speed motion detection module 120 achieves the high speed performance via a combination of two approaches. The first approach is to detect motion in the transformed domain when the input video frame is compressed. This approach avoids the intense, inverse transform computation, e.g., the inverse DCF in the JPEG video stream. In addition, in some embodiments, detections along the boundaries of the FOV are limited, assuming that the camera 220 is mounted on a fixed site and that the motion enters into the FOV across its boundaries first. This approach can be applied to compressed or non-compressed video frames.
High Performance Motion Detection:
Distributions of sample moments shown at 410 in
The approach provides more accurate initial statistical support that facilitates fast convergence and more stable performance of the segmentation operations. An improved divergence measure is used as the matching criterion between normal distributions of incoming pixels/blocks and existing pixel or blocks model distributions. A modified Jefferey's divergence measure is an accurate and simplified measure to the fixed values (constant incoming variance) as illustrated below.
Modified Jeffrey's Divergence Measure:
A modified measure is based upon the Jeffrey's divergence to measure similarity and divergence among distributions. The procedure is similar to the earlier approaches where the algorithm checks if the incoming pixel/ROI value can be ascribed to any of the existing normal distributions. The matching criterion used is referred to as modified Jeffrey's divergence measure.
While Jeffrey's divergence measure (J) H. Jeffreys, “Theory of Probability,” Universal Press, Oxford, 1948, is used, unlike earlier work, the measure is reformulated to fit the application in hand. Thus, a much simpler formulation of Jeffrey's measure used is manageable to be computed in real time while preserving the accuracy and integrity of Jeffrey's formula.
Jeffrey's measure of information associated with the probability distribution of g and fi (where fi˜N3({right arrow over (μ)}i,σi2I) and g˜N3({right arrow over (μ)}g,σg2I) is given by
J(fi,g) is the symmetric measure of directed divergence as shown in (MPT)V. Morallas, Y. Pavlidas, et al, “DETER: Detection of Events for Threat Evaluation and Recognition,” 1999. Since the J(fi,g) measure relates to distributions and not to data points, the incoming pixels data points are modeled with a fixed, predefined distribution regardless of the application and conditions. In fact, the incoming distribution g˜N3({right arrow over (μ)}g, σg2I) is assumed to have {right arrow over (μ)}g={right arrow over (x)}(t); σg2=const. The choice of the constant variance was based on some experimental observations. σg2 may not be predefined as it varies with the operating environments. Better estimates of the incoming distribution are dependent upon the current operations and not predefined constants. While this approximation is framed to simplify the divergence measure, estimates of the incoming distribution yield a much compact form of divergence measure. This simplified formulation is referred to as a modified Jeffrey's measure. The new measure is equivalent to the original Jeffrey's measure.
To model the incoming dist., g, two assumptions are introduced to simplify the formulations:
Thus eq (I) can be rewritten as follows:
To have an equal-likelihood estimate, assume αi=α (i.g. similar factor across all distributions) for simplicity without loss of generalization. Hence, the first term becomes
as an additive constant and can be dropped from measurement.
Thus the term becomes a scalar factor and can be also excluded from measure.
Hence the new measure is
Equation (III) presents the new Modified Jeffery's divergence measure, which is greatly simplified. The divergence measure is an unbiased estimate as shown in the counter example below.
Counter Example:
The estimate of incoming pixel dist yields an unbiased measure. For instance, assuming no change in the scene, the incoming dist will be identical to one of the predefined distributions, fo˜N3({right arrow over (μ)}o, σo2I). Thus:
and {tilde over (J)}(fi,g)≠0; ∀i≠o; i.e. this is consistent with the hypothesis.
The measure defined in [PM], however, yields to the wrong measurement when using a predefined σg (e.g. σg=25). This results into a non-zero divergence measure,
Which contradicts the hypothesis.
For illustration purposes, we assumed a block of pixels is represented by a three-distribution model as shown in
When a match is not found the update is performed only after a number of hits (i.e. consequent non-matches with the same incoming distribution) is reached. The minimum required number of hits in one embodiment is equal to N times the weight of the smallest wi(t). Once the minimum number of hits is reached, the update is performed in a way that guarantees the inclusion of the incoming distribution by using it to replace the lowest weighted current distribution.
The method described above allows identification of foreground pixels or ROI in each processed frame. The method is implemented to run in the pixel domain as well as in the compression domain.
Motion Detection Enhancements:
The speed motion detection algorithm represents portions of images in grey scale pixels when such portions are not high in color content, or are not expected to have motion. These areas may be selected on initialization based on knowledge of an operator, or may be selected based on a real time assessment of the scene. Portions of images are represented with color pixels, RGB for portions of the images higher in color content or those that are expected to have higher probability of motion. The portions for representing in grey scale and color may also be determined based on a real time assessment of dynamic change in the area.
In one embodiment, frames comprise pixels that are grouped in blocks of pixels, each block being represented as a single average pixel. The distributions and other statistics may be based on an average pixel for each block. In further embodiments, the blocks of pixels are of different sizes. Portions of the scene or area requiring higher resolution to detect motion are represented by smaller blocks of pixels, while those requiring lower resolution may be represented by larger blocks of pixels. In one embodiment, the size of the blocks is varied based on depth of field. In still further embodiment, the number of values per pixel is varied between 1 and 5, and may be varied based on dynamics of motions or expectations.
A block diagram of a computer system that executes programming for performing the above algorithm is shown in
Computer-readable instructions stored on a computer-readable medium are executable by the processing unit 602 of the computer 610. A hard drive, CD-ROM, and RAM are some examples of articles including a computer-readable medium. For example, a computer program 625 capable of providing a generic technique to perform access control check for data access and/or for doing an operation on one of the servers in a COM based system according to the teachings of the present invention may be included on a CD-ROM and loaded from the CD-ROM to a hard drive. The computer-readable instructions allow computer system 600 to provide generic access controls in a COM based computer network system having multiple users and servers.
A real-time segmentation method of moving objects is used to monitor their movements in large open spaces like parking lots, plazas, airport terminals, crossroads, large industrial plant floor, airport perimeter and gates, and other outdoor and indoor applications. Of particular interest is monitoring human and vehicular traffic intrusions and departures from a fixed scene. The method and devices are applied as the engine for detecting motions and tracking of traffics to analyze traffic and human movement patterns. These patterns can be informative and valuable for a security application.
The method is also useful in homeland security applications where human or vehicle traffic is monitored to investigate events, increase situational awareness of all activities, learn about an abnormal and suspicious events, and detect a threat before its occurrence.
Abandoned objects may be detected and traced back to whom they belong and how they have been introduced into the scene. Collection of traffic statistics around a commercial or government buildings are also valuable for security reasons and marketing reasons, or to support a functional redesign of the open space for better safety needs and traffic management.
The method is based on two advanced object motion detection stages. The fast segmentation stage applies intelligent sampling, and differencing techniques in compressed or uncompressed image domains The robust segmentation stage adopts a statistical mixture modeling approach and provides changes that improve the computational and functional performances of this stage. In addition, some embodiments of the method are suited to real-time, e.g. full frame rates at high spatial resolutions, applications. A resource management controller determines the sequencing, initialization and adaptive updates.