This invention relates to the analysis of images and more particularly to the analysis of images in a sequence of images and in a region of interest.
There are many applications where it is desirable to analyze an image to determine if there has been a change to that image. Examples of such applications are surveillance, industrial or manufacturing processes. One example of such a process is the making of paper.
There are systems for monitoring a paper web that use multiple cameras. For example in the Smart Advisor system sold by ABB Automation Inc. a papermaking machine may have cameras stationed at various locations along the paper web. If the paper web should break, the papermaking machine operator can look at the sequence of images captured by each of the cameras for the time period just prior to the tear in order to try to find the specific image that shows the reason for the tear. The sequence of images from the cameras are synchronized with each other so that each sequence of images shows the same section of the paper web.
Using the present system the operator cannot determine if the paper web has localized defects such as holes, discoloration, edge flips and changes in takeoff angle unless these defects cause the web to tear. The detection of localized defects when they occur will increase the quality of the paper produced by the papermaking machine. Thus it is desirable to make the detection of the cause of tears in the paper automatic and also to automatically detect upon occurrence localized defects that do not give rise to tears in the web.
There are cameras such as the Panasonic WV-CL830/WV-CL834 color CCTV camera, which are typically used in surveillance systems, that have the ability to set up a grid that is divided into cells. The camera has the capability to monitor the cells to see if there has been a change to the viewed image, for example, the brightness has changed, and if so send an alarm signal to the surveillance system. Upon receipt of the alarm signal an individual must view the scene to determine why the image changed and if that change is significant.
U.S. Pat. No. 5,245,424 describes a defect defection method. In the method described therein, a video camera picks up an image of a light irradiated object. Two inspection zones adjacent each other are established on the picture screen. The zones each have a preset size and the brightness of each of the two zones is sensed and accumulated as the inspection zones are indexed across the screen. When the difference between the accumulated values of the respective inspection zones exceeds a preset threshold a defect is detected.
While the image analysis techniques of the prior art do allow a change in an image to be detected they do not include a user designated zone which will not have interesting changes therein but can be used to assess the image quality. Further none of the prior art techniques provide for the automatic assignment of cells into a zone which may include interesting changes and a zone of the image that does not have to be analyzed. Still further none of the prior art image analysis techniques allow for the analysis of the image cells to be carried out on compressed data. Yet still further none of the prior art image analysis techniques remove normal periodic variations from the image prior to computation of the feature statistics. The image analysis technique of the present invention has these properties.
A method for analyzing a time sequence of images. The method:
divides a reference image based on its visual content into:
(a) first spatially fixed areas of each image in the time sequence of images to analyze for detection by selecting in the reference image one or more detection cells collectively comprising a detection zone in which a statistically significant change in one or more predetermined features in the time sequence of images to be analyzed is expected to occur; and
(b) second spatially fixed areas of each image in the time sequence of images to veto a detection of a statistically significant change in one or more predetermined features in the time sequence of images by selecting in the reference image one or more veto cells collectively comprising a veto zone in which any detection in the veto zone of a statistically significant change in one or more predetermined features in the time sequence of images is used to disable any detection of a statistically significant change in the one or more predetermined features in the time sequence of images in the detection zone.
A method for analyzing a time sequence of images. The method:
(a) uses a reference image to position analysis cells in the each image of the time sequence of images; and
(b) divides the reference image based on its visual content into one or more types of analysis cells by:
(ii) defining second spatially fixed areas of each image in the time sequence of images by selecting in the reference image one or more veto cells collectively comprising a veto zone in which any detection in the veto zone of a statistically significant change in one or more predetermined features in the time sequence of images is used to disable any detection of a statistically significant change in the one or more predetermined features in the time sequence of images in the detection zone.
A method for analyzing a time sequence of images. The method:
(a) uses a reference image to position analysis cells in each image of the time sequence of images; and
(b) divides the reference image based on its visual content into one or more zones, each of the zones comprising a collection of cells of the same type, the one or more zones comprising:
A system for analyzing a time sequence of images. The system has:
a computing device operative to:
divide a reference image based on its visual content into:
(a) first spatially fixed areas of each image in the time sequence of images to analyze-for detection by selecting in the reference image one or more detection cells collectively comprising a detection zone in which a statistically significant change in one or more predetermined features in the time sequence of images to be analyzed is expected to occur; and
(b) second spatially fixed areas of each image in the time sequence of images to veto a detection of a statistically significant change in one or more predetermined features in the time sequence of images by selecting in the reference image one or more veto cells collectively comprising a veto zone in which any detection in the veto zone of a statistically significant change in one or more predetermined features in the time sequence of images is used to disable any detection of a statistically significant change in the one or more predetermined features in the time sequence of images in the detection zone.
A computer readable medium having instructions for causing a computer to execute a method for analyzing a time sequence of images, the mixture having a liquid region. The instructions are for:
dividing a reference image based on its visual content into:
(a) first spatially fixed areas of each image in the time sequence of images to analyze for detection by selecting in the reference image one or more detection cells collectively comprising a detection zone in which a statistically significant change in one or more predetermined features in the time sequence of images to be analyzed is expected to occur; and
(b) second spatially fixed areas of each image in the time sequence of images to veto a detection of a statistically significant change in one or more predetermined features in the time sequence of images by selecting in the reference image one or more veto cells collectively comprising a veto zone in which any detection in the veto zone of a statistically significant change in one or more predetermined features in the time sequence of images is used to disable any detection of a statistically significant change in the one or more predetermined features in the time sequence of images in the detection zone.
Referring now to
The image is analyzed by processor 14 using the method described below and, when an image is identified, the human interface 16 or another machine interface is notified so that the image can be saved or marked and further action taken.
The identification of significant regions in the image is an important feature of the method of the present invention. Consider a schematic image such as the image 20 shown in
Referring now to
As used herein a zone is a collection of possibly connected or disconnected cells of the same type. Thus, for example, Detection cells 30, 32, 34 and 36 form the detection zone. Therefore, at the end of the setup procedure the reference image has one detection zone, one veto zone and one ignore zone and the cells in each zone may or may not be connected to each other.
Only statistically significant changes in the Detection cells can identify an image as interesting. The Ignore cells are areas, of the image that are not analyzed. Any pixel, for example pixel 37 shown in
A change to the image that is not of interest may also trigger a change in a Detection cell. The Veto cells are used to identify changes to the image that are not of interest and ensure that the method of the present invention ignores those changes. Examples of changes to the image that are not of interest include a loss or degradation of the video signal. A Veto cell would generally be located in an area of the image in which no change is expected.
The Detection, Veto and Ignore cells can be of arbitrary size, shape, location, and number. As is shown in
As was described above, one application of the method of the present invention, by way of example and not by way of limitation, may be in a papermaking machine. The reference image in that application would usually be the paper web and surrounding parts of the machine. In the setup procedure the operator of the papermaking machine would typically select all of the cells in the paper web as the detection zone and the cells off sheet as the veto zone.
The identification of the pixels in the cell performed by block 44 is based on all the pixels in the identified cell region. Let the notation “ij in Ck” denote the set of pixels ij in cell Ck. If a Detect or Veto cell contains an Ignore cell, then the pixels in the Ignore cell are not included in the pixels in the Detect or Veto cell.
For a cell Ck, consisting of N pixels, a set of features or discriminants are computed 46 based on the pixel amplitudes in the cell. Possible features can be based on the intensity of each pixel in the cell:
Pixel intensity,
Average cell intensity,
Intensity variance,
Other features can be based on the colors in the cell.
Average color,
The method requires a training period so that the feature statistics can converge to reliable values and a decision block 48 determines if the training is completed. During the training period no cell alarm logic is applied and only the feature statistics are updated in block 50. The training period consists of a sequence of M images In with the cell feature computation applied to each cell of each image. The user of the method of the present invention defines either the time duration of the training period or the number of images, us that is, M.
Let fnmkl be the value of feature l for cell k of zone m in image n with m=Detect or Veto. For convenience of notation, the zone, cell, and feature subscripts will be dropped in the discussion below and only the image index is retained. During the training period the mean and variance of each feature is computed.
Feature mean,
Feature variance,
Once the training period is complete, the feature statistics continue to be maintained so that the method can adapt to slow changes in the statistics. For each image, zone, cell, and feature a running mean and variance is maintained.
History feature mean,
History feature variance,
In addition to the linear statistical average described above, other averaging techniques such as an exponential average could be employed for computing the historical feature means and variances.
As each image is analyzed once the training period is over, the feature values are compared in decision block 52 to the statistical values and the sensitivity value αkl for cell k and feature l. If for any feature l in cell k of zone m in image n,
(fnmkl−{tilde over (f)}n−1,m,k,l)>αkl√{square root over({tilde over (V)})}n−1,m,k,l
then cell status Snmkl=1, otherwise it is 0. If the cell status is 0, then the feature statistics are updated in block 50 as described above. The cell status is set to 1 or 0 in blocks 54 and 56, respectively. A cell status of 1 is a preliminary indication of the detection of a statistically significant change in that cell.
Once the cell logic has been applied to every cell Ck in zone m (m=Detect or Veto), the zone detection logic 60 of
is greater than threshold pm then block 66 sets the zone status Tnm=1, otherwise block 68 sets the zone status to 0. If the zone status is 1, then the image capture time, tnm, is recorded. The threshold βm is set by the user of the method of the present invention.
The alarm logic 70 for the image is shown in FIG. 6. If decision block 72 determines that the Detect zone status Tn, Detect=0, then the image status Un is set to 0 in block 74 and no significant change is indicated. If block 72 determines that Tn, Detect=1, then the veto time period, τ, of block 76 must pass allowing the computation of the veto zone status of future images to be computed.
The image status Un is 1 (block 79) if in loop 78, for all images i with veto zone status ti,Veto=1 (block 78a), there are no veto zone times ti,veto (decision block 78c) such that abs(tm,Detect−ti,Veto)<τ (decision block 78b), otherwise the image status is 0. An image status of 1 indicates that a statistically significant change has occurred in the image that is not associated with any changes in the Veto Zone. This image can then be viewed from the user interface or marked for special use by another program or process.
An enhancement to the cell assignment procedure discussed above is shown in the logic 80 of FIG. 7. This technique allows the automatic assignment of cells into either Detect or Ignore zones. The method uses a training set of images of number M with each image divided into a grid pattern of cells consisting of K columns and L rows to identify in block 82 the pixels in a cell. For example if the image is 640×480 pixels, and a cell grid of 20 columns and 20 rows is defined, then each cell is 32 ×24 pixels. Based on the feature statistics, each cell is classified as either a Detect cell (block 92) or an Ignore Cell (block 94). Veto cells are not assigned by this technique. Logic 80 then proceeds to block 84 which has the same function as block 46 described in connection with FIG. 4. Thus block 84 computes the features in a cell based on the pixel amplitude in the cell.
Logic 80 then proceeds to block 88 where it computes the feature statistics for each cell. The computation of the feature statistics for each cell is performed in the same manner as described above in connection with the training period of FIG. 4.
History feature mean,
where fikl is the feature l value cell k in image i
History feature variance,
Thresholding logic in the form of decision block 90 is then applied to the mean and variance to determine the cell type. Let αl and αh the low and high threshold limits for the mean and βl and βh the low and high threshold limits for the variance. The threshold limits are set by the user of the method of the present invention.
Block 90 determines that a cell is an Ignore cell if
{tilde over (f)}kl<αl or {tilde over (f)}kl>αl or {tilde over (V)}kl<βl or {tilde over (V)}kl>βl,
otherwise the cell is a Detect cell. If the feature is the intensity in the cell, then the cell is an Ignore cell if it is too dark or too bright, or if the variance is too small or too large. The logic 80 is appropriate in cases where the method is tuned to detect activity in areas that are generally inactive.
An additional variation on the technique can be utilized when a compression algorithm is applied to the image prior to analysis. In many cases the video signal is both digitized and compressed by special hardware. The discussion above of the analysis method of
As an example, consider the MJPEG compression algorithm. Although it is beyond the scope of the description of the present invention to provide a detailed description of the algorithm, a brief overview is given herein. The MJPEG compression algorithm first converts the image from the RGB model to the YUV model. As described above, the Y component represents the intensity of the pixel and the U and V represent the chromaticity. The image pixels for each component are then divided into 8×8 blocks and the Discrete Cosine Transform is applied to each block. The resulting block DCT coefficients are quantized and then run length encoding and Huffman encoding are applied to the coefficients.
In the case where the feature is the intensity of the cell and the cell is composed of a number of DCT blocks, the average intensity of the cell can be computed by decompressing only the DC term of the included DCT blocks and averaging these values. This represents a significant savings in the feature computation time for the cell.
The described MJPEG compression analysis algorithm is designed to identify images with feature values in a selected region of the image with statistically significant deviation from the norm. Many scenes have a normal periodic variation that may generate a cyclical variation in the feature statistics. The sensitivity of the technique can be improved if the periodic variation is removed prior to computation of the feature statistics. This removal of the periodic variation is accomplished by the logic 100 of FIG. 8.
As described above, the feature statistics are computed in block 102 from a sequence of M feature values. Let fnmkl be the feature l value of cell k in zone m of image n where n spans the M images. To simplify the notation below, only the n index will be retained. A frequency analysis can be applied to this sequence to identify and remove any dominant periodic variation. The discrete Fourier transform of the sequence of features, computed in block 104, is given by:
The fractional amplitude of any particular frequency component is:
In block 106 all components that have a fractional amplitude exceeding a predetermined threshold γare identified and let the set S be the set of indices satisfying this criterion.
S=set of w such that r(w)>γ
The components in S are then removed by block 108 from the feature sequence by applying the inverse transform resulting in a filtered feature sequence, {circumflex over (f)}nmkl.
The features statistics and thresholding are replaced in block 110 with those based on the filtered feature values.
History feature mean,
History feature variance,
If for any feature l in cell k of zone m in image n,
then cell status Snmk=1, otherwise it is 0.
It is to be understood that the description of the preferred embodiment(s) is (are) intended to be only illustrative, rather than exhaustive, of the present invention. Those of ordinary skill will be able to make certain additions, deletions, and/or modifications to the embodiment(s) of the disclosed subject matter without departing from the spirit of the invention or its scope, as defined by the appended claims.
Number | Name | Date | Kind |
---|---|---|---|
5016173 | Kenet et al. | May 1991 | A |
5627907 | Gur et al. | May 1997 | A |
5768412 | Mitsuyama et al. | Jun 1998 | A |
6577757 | DeYong et al. | Jun 2003 | B1 |
Number | Date | Country | |
---|---|---|---|
20030016874 A1 | Jan 2003 | US |