Patent Grant
8159537
The present application claims priority from Japanese application serial no. 2006-287087, filed on Oct. 23, 2006, the content of which is hereby incorporated by reference into this application.
The present invention relates to a video surveillance equipment having functions for capturing pictures from an imaging apparatus such as a camera and detecting abnormalities and the like in a monitoring area by image recognition.
A video surveillance equipment having a function for detecting moving object that appear in a monitoring area by image recognition, such as persons and vehicles and the like can record only pictures on which a moving object appears by using detection results, and can call observer's attention by displaying warning icons on a display section or making a buzzer or the like sound. Accordingly, this type of video surveillance equipment is useful for reducing the burden for monitoring jobs in which confirmation has been needed at all times.
The above video surveillance equipment uses a known method of detecting changes in persons' motion and the like on pictures by comparing input images with a background image prepared in advance. In generally, this method is called the image subtraction method. The image subtraction method is used in many video surveillance equipment because the computational cost in the method is relatively low.
To detect a state in which a dangerous object or the like is left, a method of not only detecting changes on pictures but also recognizing, as a left object, an area in which changes from the background image are consecutively detected is disclosed in, for example Japanese Patent Laid-open No. Hei. 10-285586.
In the method described in Japanese Patent Laid-open No. Hei. 10-285586, a background image and a comparison image are generated. This background image includes only a scene with no moving objects and no left objects, and this comparison image is obtained by removing only the moving objects in processing for comparing input images with the background image. The background image is older than the comparison image on a time axis. A detection area obtained based on a difference between the background image and the comparison image is defined as a stationary object. Whether there is a left object is determined according to the presence of the stationary object.
However, in the above method, a change area is just derived from the background image and comparison image, and a left object itself cannot be recognized. If a plurality of changes occur in the same area, the changes cannot be distinguished individually. When, for example, an left object is placed in front of another left object, these left objects cannot be distinguished. When another object may be left in the area where a missing object has been placed after an object is missed, the missing object and the left object can not be distinctly detected.
In a conceivable method to solve this problem, when a left object is detected, the image data of its area is stored, and the image data obtained from input images is compared with that area so as to determine the presence of a left object and detect changes. When traffic of persons is heavy, the amount of image data by which objects can be referenced is reduced due to moving objects passing between a left object and the camera. Accordingly, the method may affect the discrimination of left objects.
An object of the present invention is to provide a video surveillance equipment and a video surveillance system which can individually detect changes of objects in an area for monitoring a stolen object and a left object, etc. even when the changes occur in the same image area.
The present invention to accomplish the above object is a video surveillance equipment which is provided with an area detecting unit for detecting a second image area nonmoving in a first image area which is a different portion between a reference image being used as a reference in image processing and input images, and a memory medium for storing an image retrieved from the second image area, wherein the images in the second image area included in the input images compares with a stored image over a plurality of times.
According to the present invention, changes of objects in the monitoring area can be individually detected from pictures input from the camera or stored pictures, without being affected by moving objects.
By this method for detecting missing objects and left objects, it is possible to provide a video surveillance equipment which can record left objects and missing objects, issue an alarm when an object is left or missed, and call observer's attention on the display of a monitor.
(First Embodiment)
An embodiment of the present invention will be described with reference to the drawings.
The picture capturer 10 captures image data from the camera in real time and image signals received from a picture memory apparatus and the like in which image data is stored, as one-dimensional or two-dimensional array image data. The image data may undergo the preprocessing of smoothing filtering, edge enhancer filtering, image conversion and the like to reduce noise and flicker. A data format such as RGB colors or monochrome may be selected according to the purpose. The image data may also be compressed to a predetermined size to reduce the processing cost.
Next, the image recognizer 20 will be described in detail with reference to
Ideally, the reference image generated by the reference image generation section 201 is an image that adapts to environmental changes such as ever-changing weather conditions and illumination conditions and does not include the moving objects. This is because if the reference image including the moving object is compared with an input image, the moving object included in the reference image may be detected as a change. Furthermore, if the reference image are not followed the environmental changes, a difference in illumination between the reference image and the input image may be detected as a change in brightness. In the present embodiment, therefore, the reference image generation section 201 performs statistical processing on images obtained by removing the effect of a moving object from the input images captured in a set period so as to reconstruct an image excluding the moving object by using information, described later, output from the motion detector 203. The reference image can also be registered by the observer. Accordingly, a reference image, which is free from moving objects and adapt to the environmental changes, can be generated, and the moving objects can be detected with high precision. The reference images generated by the reference image generation section 201 are stored in the reference image management section 202 at set time intervals.
Next, the motion detector 203 will be described. The motion detector 203 carries out comparison processing between the reference image, which is obtained in advance by the reference image generation section 201 and stored in the reference image management section 202, and the input images obtained by the picture capturer 10. Information used in the comparison processing may include a brightness value and RGB values calculated for each pixel of the input images, an amount of characteristics calculated by using arbitrary operators such as edge intensities and directions calculated through a differential filter by using, for example the Sobel operator, or characteristic vectors obtained by the integration of the brightness value, the RGB values and the amount of characteristics. The robustness against environmental changes and detection precision vary with the amount of characteristics, so an amount of characteristics suitable to the situation needs to be set. In the present embodiment, however, brightness values, which are most prevailing, are used. Methods conceivable in the comparison processing include each method of calculation on a per-pixel basis, determination in the local area near a particular pixel, and expansion of a determination criteria in the time axis direction relative to the reference image by use of several frames of input image. In the present embodiment, however, the calculation method based on differential operation on each pixel is used.
A concrete calculation method will be described below. When image data is a two-dimensional array, a pixel position p at arbitrary x and y coordinates is represented as p=(x, y). The brightness value of the reference image and that of the input image at the pixel position p will be denoted Bp and Ip, respectively. An amount Δp of change between the reference image and the input image at the pixel position p is calculated by Δp=Bp−Ip. When Δp is equal to or greater than a predetermined threshold, the pixel is determined as being included in a moving object. When this determination is carried out over the entire image, the area of the moving object can be extracted.
In processing on a per-pixel basis, however, responses to noise and the like may occur, and thus a false-positive phenomenon may be detected as the area of the moving object or the detected area may include lack of defect areas. Therefore, an area determined as including a moving object undergoes shaping processing such as image processing of the extracted pixels.
In the above motion detection processing, whether the object has stopped cannot be determined. Even when the object has stopped, the reference image management section 202 stores a reference image close to the current time at which processing is performed and subsequent older reference images. Therefore, by making a comparison between the input image and the nearest reference image, it is possible to determine whether the object is stopping or moving, as approximately illustrated in
Next, the left object detector 204, which is the most basic part in the video surveillance system, will be described with reference to
Processing that are performed by the left object determination apparatus 402 will be described in detail with reference to
Comparisons that can be carried out in the comparison operation of the images include comparison of spatial brightness distributions on the images by a correlation method or analysis of spectrum, comparison of geometrical structures including edges, outlines, textures, and corner points, and comparison between images based on the degree of similarity got by comparison of characteristics obtained by other operators. Although various types of pattern matching methods are applicable in these comparison operations, the comparison operation which is performed in the left object determination apparatus 402 uses the SAD (sum of absolute differences), which is the most simple.
In the SAD method, the brightness value of a template image (here, which is the left object data) at the pixel position p is denoted Tp, and the brightness value of the input image at the pixel position p is denoted Ip. When the total number of pixels in the image data be M, a degree of similarity S is then calculated by, for example, S=Σ|Tp−IP|/M. This degree S of similarity can be used to determine the degree of similarity of the image data.
When a moving object appears between the left object and the camera, exception values are also included in the area of the stationary object obtained from the input image, together with the image data R53. Therefor, the degree of similarity in the SAD method is reduced. To solve this problem, when the effect of the moving object is reduced in the determination as to the presence of a moving object, information about the motion area obtained by the motion detector 203 is used to perform mask processing for the motion area.
Mask processing is performed on both the stationary object data R52 and the image data R53 of the input image. When SAD is executed for the remaining area, for which the mask processing is not performed, the motion area does not serve as a factor to reduce the degree of similarity.
However, when the motion area is removed, the amount of data to reference is reduced. This may reduce the reliability of the degree of the similarity. Even when there is a left object, determination of the degree of the similarity based on a fixed threshold may cause a mismatch. Accordingly, on the reference image, comparison operation is also performed between the stationary object and the image data (R51) in the same area.
As in the method described above, the stationary object data (R52), and the image data (R51) of the reference image are acquired (S51), and mask processing is performed for the motion area to derive degree of a similarity on the basis of SAD. The degree of similarity between the stationary object data (R52) and the image data (R51) of the reference image is derived by a reference image similarity degree calculation apparatus 501, and the degree of similarity between the stationary object data (R52) and the input image is derived by an input image similarity degree calculation apparatus 502. A similarity degree comparison apparatus 503 compares the two similarity degrees; when the degree of the similarity with the input image is higher, it is determined that a left object exists; when the degree of the similarity with the reference image is higher, it is determined that there is no left object.
The reliability of the degree of similarity with the input image is determined according to the dimensions of the stationary object data on which mask processing has been performed. If the dimensions are smaller than a predetermined threshold, the amount of referenceable data is small, lowering the reliability of the degree of similarity with the input image. In this case, pattern matching with the reference image is also performed and then the similarity degree comparison apparatus 503 determines the presence of an object. This pattern matching processing increases the accuracy of the determination as to whether an object is left. If the dimensions of the stationary object data on which mask processing has been performed are greater the predetermined threshold, the reliability of the degree of similarity with the input image is high, so the degree of similarity with the input image is used to determine whether there is an object. This processing enables the computational cost to be reduced while the precision of determination as to whether there is an object is maintained, as compared with the method of determining the presence of an object by the similarity degree comparison apparatus 503.
So far, the description has been focused on detection of left objects. When an image including an object to be monitored is set as a reference image, missing of the object in the monitoring area can also be detected. If the object is missed in the monitoring area, since the image in the area in which the object has been present before being missed changes, the motion detector 203 determines that the remaining object is a stationary object. Since there is no motion in the changed area, missing of the object can also be detected as in detection of a left object.
In the image processing method in the present embodiment, the image data of a stationary object image is compared with an input image and reference image, so changes caused in the stationary object can also be detected. Specifically, deformation, damage, and other changes caused in the area of the stationary object can be detected.
(Second Embodiment)
A video surveillance system of second embodiment, in which an arrangement for acquiring information about a face, clothes, and the like is added to the video surveillance system of the first embodiment, will be described with reference to
(Third Embodiment)
In the left object detection method of the present invention described in the first embodiment, pictures input to the image recognizer 20 may be picture data already stored in the memory medium 40. That is, a function to search for the stored pictures can be implemented.
Some applications of the method of the third embodiment can be considered. A cashbox, jewel, or other important object is set through a human interface, and object data is acquired. Degrees of similarity between the important object data and the input images and between the important object data and the reference image are derived to determine whether the object is present, implementing an important object watching function. When the similarity degree derived by the input image similarity degree calculation apparatus 502 in
Furthermore, the method in which a normal time is specified can be applied to implement a monitoring system for detecting left and missing objects, as illustrated in
In the above image processing method, in which an area of an object is specified from the operation panel and processing similar to processing by the left object determination apparatus 402, in which a specified image data is substituted for the left object data R51 in
The transfer section 60, shown in
The present invention includes a program that implements the left object recognition method on a computer.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2006-287087 | Oct 2006 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7847675 | Thyen et al. | Dec 2010 | B1 |
| 20020163577 | Myers | Nov 2002 | A1 |
| Number | Date | Country |
|---|---|---|
| 10-285586 | Oct 1998 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20080158361 A1 | Jul 2008 | US |
