1. Technical Field
The present invention relates to a target or object detection process.
2. Background Information
It is often desired to find a target or object against various surrounding backgrounds in very large sampling areas. For example, U.S. Pat. No. 5,848,189, “Method, Apparatus, and System for Verification of Patterns”, discloses the use of morphological operators to perform template matching and pattern verification. U.S. Pat. No. 5,068,910, “Image Processing for Reading Dark and Light Characters from Light and Dark Backgrounds Respectively”, discloses reading a dark-colored character on a light-colored background. U.S. Pat. No. 3,646,264, “Method of Acquiring a Moving Target” discloses a method for acquiring a moving target using the photo-cathode of a scanning tube and looking for sharply defined contrast edges. The disclosures of these patents are hereby incorporated by reference in their entireties.
Known systems involve searching for a target having predetermined patterns and characteristics, and involve knowing the specific shapes being searched for ahead of time. Searching an area for unknown objects of unspecified shape has been considered computationally intensive, and associated with high false alarm rates that adversely impact a specific search mission.
The present invention is directed to a method for detecting an object image within image data, the method comprising receiving image data, segmenting the image data into multiple windows, determining a likelihood that each window contains the object and probability rank ordering the multiple windows based on the step of determining, and selecting one of the multiple windows as a window wherein the object image is considered to reside.
The receiving step can comprise collecting and recording the image data as the image data emanates back to a receiver. The step of segmenting can comprise determining a set of image metrics, applying predetermined selection criteria to filter false detections and clutter from the image data, comparing image data, after applying the selection criteria, with the image metrics, and applying morphological operators on the image data.
Exemplary embodiments display at least one of the multiple windows, identify pixels having a lighter contrast compared to other pixels in the imagery, identify pixels having a darker contrast compared to other pixels in the imagery, and identify pixels having both lighter and darker contrast compared to other pixels in the imagery. A morphological operator can be used to isolate targets from their background. Two concatenated morphological filters can be applied to filter the image data.
The method can detect spatial discontinuities at the pixel level. Detected window images and other image data can be displayed to the user in a mosaic format. The detected window images can also be communicated to another system. The image data processed comprises, for example, visual data, thermal data, and synthetic aperture radar (SAR) data.
An alternate target detection process according to the invention comprises acquiring image data, down-sampling the image data n-times, processing the down-sampled image data for detecting at least one of a light target and a dark target, labeling subsets of the image data that may contain target data and rejecting clutter associated with these subsets of the image data, combining results of the image data that has been down-sampled, and forwarding combined results to a decision making authority. The decision making authority can extract windows and rank order them. An image can be down-sampled n-times using a series of low pass filters that can filter in horizontal and\or vertical directions. The process comprises an image that has been down-sampled n-times, where n comprises a large number that can accomplish target detection after accomplishing a larger amount of down-sampling. The filtering process can be performed by a six by six (6×6) convolution filter. The filtering process can also be performed by an N by N convolution filter, where N is a number greater than or equal to one.
Objects and advantages of the invention will be understood by reading the following detailed description in conjunction with the following drawings where:
The
The image metric data established in step 101 of
The Isotropic Detection Technique (IDT) used in an exemplary embodiment is an autonomous, two-dimensional image target detection process. The IDT works by acquiring and detecting targets that possess contrast discontinuity changes along their borders or in their internal structures. This technique can examine the effect that the target has on the background surrounding or proximately located next to the target. The IDT algorithm can process large image sets covering upwards of thousands of square miles. These image sets correspond to hundreds of mega-pixels. The complete pixel dynamic range or depth is used in the detection process. In a number of cases, pixel depths up to and including 16 bits have been accommodated. The detection portion of the IDT process can automatically determine both the ship's directional heading and the ship's size.
The segmenting step 103 outputs a series of target windows that in an exemplary embodiment are organized in a target likelihood order in step 104. For example, those windows most likely to contain target information can be displayed first, and those with a lesser likelihood to contain target information are displayed later. Those skilled in the art will recognize that the target information can be displayed in other ways also, such as starting with those target windows least likely to contain information. The target detection region size, plus some standard border outline helps to determine the size of the extracted windows. The border also provides some of the background for context and reference frame. The target windows that are output from the IDT algorithm and appear first in the rank ordered list are those target windows most likely to be a target, while those target windows appearing further down the rank ordered list are most likely to be background clutter.
One of the multiple windows is selected as a window wherein the object image is considered to reside. The window can be selected automatically based on the likelihood determined for the window, or the image likelihood determined for the window, or the image data in the multiple windows can be displayed for the user. For example, the rank ordered target windows of step 104 can be presented on a display to the user as a mosaic of the detected images. User review of the information can be enhanced by mosaicing a number of the windows on a single image 106. A second image 107 of the total scene down-sampled to fit on the display can also be used to provide a synoptic view of the total image, with possible target information being marked as small boxes 103.1. For example, in a maritime application, the possible target information as displayed in the second image 107, can be used to provide potential ship-to-ship relationships.
The mosaic of image 106 can be arranged in any manner the user desires. The mosaic of images selected corresponds to one of the multiple windows where a object image may reside. This mosaic of images can be arranged to make the information easy to understand. One possible arrangement format is to position the target windows from left to right and moving from the top row to the bottom row in probability rank order. The highest probability window step 104.1 would be in the left-most location on the first row at the top of the mosaic. The lowest probability window step 104.2 would be in the right-most location on the last row located at the bottom of the mosaic.
The mosaic format permits an efficient and human factor friendly presentation to the user. However, in addition to presenting the target window information to a human observer, the target window data can also be sent across a communication link to an automated system, computer, or other processing device that is designed to interpret the resulting target window data. Further, a threshold value or floor can be set for the image metric data and only that information that meets this image metric criteria, such as size and target orientation, will be forwarded to a user or automated system.
In step 120, the coordinates of a desired detected target window are obtained and then requests that the corresponding full resolution image relating to the target window be pulled.
The image output step 100 is sent to the IDT algorithm in step 103. Here the image data is down-sampled n-times in step 108 and the output is then processed for Light Target detection step 110, Dark Target detection step 112, or both. One or both of these operational paths can be run in the algorithm depending upon the type of target, the imagery data format accumulated and the background. The Light Target detection data output from step 110 undergoes a region labeling and clutter processing step in step 114. Similarly, the dark target detection data from step 112 then undergoes a region labeling and clutter processing step in step 116. The region that is being labeled is the target region and clutter is the unwanted and extraneous information that is not part of the target region and needs to be removed so as to not obscure the target region.
In 118, the outputs of the Light Target detection step 114 and the Dark Target detection step 116 are combined into a combined detection view and the target windows are then pulled based upon this output criteria. A metric is computed in both the Light and Dark Target Detection routines.
The step 104 of probability rank ordering the multiple windows uses the metric computed in step 103 to order the target windows to be pulled as shown in step 120. This information along with the location and extent is sent forward to step 120 where the full resolution imagery for the window is pulled from the original image data. All the windows are ordered in step 104 by their confidence or IDT metric. For example, high confidence windows are presented first, followed by the lower confidence windows.
Since most ships are typically brighter than an ocean background, especially in Synthetic Aperture Radar (SAR), most of the imagery can be processed with the Light Target detection. However, when a given situation lends itself more appropriately to Dark Target detection, the imagery can be processed by using Dark Target detection. An exemplary scenario where Dark Target detection would be effective is the presence of dark ships in glare or glint imagery in visual or thermal imagery.
The particular selection of the total number of n down-samples is governed, at least in part, by the minimum target size to be detected. The down sample inter-pixel distance can, for example, be sized so that the minimum object size, such as a ship and all wakes, is approximately 1½ to 2 pixels in the maximum direction after down-sampling, although this sizing can be adjusted to achieve different performance levels.
The down sampling process can serve as a clutter rejecter. However, the level of down-sampling to reject clutter can be balanced against too much down-sampling that results in the loss of small objects. Another benefit of down sampling the original imagery is that down-sampling step can reduce the amount of pixel processing that is used in the algorithm suite. The down-sampling of pixels is not computationally expensive. Therefore, when down-sampling is combined with the rest of the algorithm suite, the resulting total pixel computation load factor is extremely low. As mentioned earlier, IDT simulation runs quickly on a computer, such as a typical desk top computer.
A further detailed view of the 2:1 down-sampling of 108a is shown in
Each time that a 2:1 down sample is performed, the resulting number of pixels remaining to be processed is reduced by ¼. Therefore, if a total of three down-samples are performed, the resulting image would be 1/256 as large as the original. Consequently, a 256 mega-pixel image would be reduced to a 1 mega-pixel image for processing in the rest of the algorithm.
Although target reduction to one or two pixels can be performed, higher or lower levels of target reduction can be implemented, depending upon the specific application. For example, if a ship having a length of 100 meters is to be detected and the image resolution is about 15 meters per pixel, then two to three 2:1 down-samples, or any other number and/or ratio of down-samples, are performed.
As shown in
Convolution operators are well known to those skilled in the art. One filter that can be used in removing both aliasing artifacts and clutter is a 6×6 low pass convolution filter. This filter can remove high frequencies that cause aliasing when the image is resampled by the down-sampler.
Those skilled in the art will appreciate that various convolution operators, filter types and sizes can be used in implementing the IDT operator.
The 6×6 convolution array 304 can, in an exemplary embodiment, be calculated in the following manner. Starting with the value “1” in the left most position of the x direction filter 301 can be multiplied together by each value in the y direction filter 302 moving from top to bottom. These values become the first column of the 6×6 array 304. Moving to the next value “4” in the second position of the x direction filter, each value in the y direction can be multiplied together by each value in the y direction filter 302 moving from top to bottom. These values become the second column of the 6×6 array. This process is repeated for each value in the x direction filter to produce the full 6×6 array depicted in 304.
Image processing routines are used to define the extent of the targets and remove objects that are not considered to be targets of interest, based on size or extent. For example, single point targets are usually rejected as clutter based on the size and the chance that noise spikes will be captured and classified as targets. Next, each remaining region is given a unique label and considered as a candidate target. The extent of each labeled region is determined and a size criteria is applied. If an object's dimensions are too large to be a target, it is eliminated as clutter. In other cases, a threshold minimum sizing criterion can be applied, and candidate targets failing to meet this requirement can be eliminated.
At this stage of the process, objects possessing target like properties have been retained and are tested to determine if they have other ship or target like characteristics. For example, the length to width relationship of the candidate target can be assessed, along with a simple texture measure across the candidate target.
Because of the flexibility to accommodate Light and Dark Target detection and the low false alarm rate, it is possible to run either or both Light and Dark Target detection algorithms, and optionally combine the detection results, to achieve desired system performance and accuracy.
A highest intensity region can be considered to be the most likely target. Light Target detection detects candidate targets or elements that are lighter than their background. In
In the
The contents of the Pattern 1 filter erosion/dilation neighborhood are shown in step 410 of
In Filter Pattern 2, the input image pixels step 505 are eroded with the neighborhood step 506 and results in the Output Image Pixels (prime) step 507, where e′=maximum of {a, b, c, d, e, f, g, h, i}. Therefore pixel e′ of the output image is the maximum of the nine pixel values of the input image pixels.
In
In
The contents of the Pattern 1 filter dilation and erosion neighborhood are shown in step 612 of
In the Dark Target detection process, the dilation operation is performed first, followed by the erosion operation. This sequence of operations is followed for both Patterns 1 and 2. In the Dark Target detection case, the subtraction is performed in the reverse order and the input image is subtracted from the operated image.
The output from the Dark Target detection module are all the regions where candidate dark ships or dark targets (e.g. an oil spill) have been detected. As with the Light Target detection, the IDT processes will clean up the resulting image, label and rank order the detection. The rank ordering can, for example, be determined by the minimum intensity produced by the Dark Target detection module of the algorithm, for each of the label regions that pass all the other processes down the algorithm chain.
In Filter Pattern 2, the input image pixels of step 511 are operated on in conjunction with the dilation neighborhood step 512 to produce the Output Image Pixels in step 513. This filter operation focuses on the center element “e′”, where e′=maximum of {a, b, c, d, e, f, g, h, i}.
These target regions are optionally sent through a series of single point cleaning and clutter rejection routines of step 700. Step 702 is a region labeler used to label detected candidate regions. Single point target cleaning is accomplished using morphological filters to reject regions as clutter. Single point targets can be rejected as clutter based on the size and chance that noise spikes will be captured and classified as targets. In step 704, minimum and maximum sizing criteria can be used to reject additional regions as clutter through application of sizing criteria with, for example, minimum and/or maximum rejection thresholds. For example, if a potential target is determined to be too large to be an actual target, it is eliminated as clutter. Conversely, if a potential target is determined to be too small to be an actual target, it can also be eliminated as clutter. Each remaining region is given a unique label and considered a candidate target.
The various features of the image data are computed in step 706. Target features such as heading, centroid and extent are computed. Further clutter rejection is accomplished by the classifier of step 708. As an example, the classifier can be a simple quadratic statistical classifier that has been trained to separate targets from clutter. This classification can be based upon any additional constraint requirements the user desires for a given application. For example, the classifier can reject ships having a length greater than, less than, or equal to a specified length. Targets that are moving in an unwanted heading or direction can be screened out. Additionally, target length to width ratios, texture, and the IDT metric determined in the Light or Dark Target detection process can be used, as can any other desired criteria.
The outputs of the process flow shown in
Exemplary IDT systems of the present invention can be used in any application including, but not limited to, maritime applications and terrestrial applications. Exemplary embodiments can process a wide range of data types, such as visual, thermal, and Synthetic Aperture Radar (SAR) imagery or any desired data. Exemplary embodiments can detect both man-made and naturally occurring structures and objects, such as ships, icebergs, ice flows, and oil slicks in a naturally occurring background, such as open ocean, with high detection rates that will, for example, 20,000 to 1 data compression ratios (e.g., 90% detection rate, with false alarm rate of less than one per 1,000 square nautical miles). Bandwidth compression obtainable is a function of the number of ships in the viewing area or scene and the false alarm rate. Because excellent data compression ratios can be achieved, it is practical for a person to review and digest the data, or the image data can be disseminated over low bandwidth communication networks.
The present invention has been described with reference to preferred embodiments. However, it will be readily apparent to those skilled in the art that it is possible to embody the invention in specific forms other than that described above, and that this may be done without departing from the spirit of the invention. The preferred embodiments are merely illustrative and should not be considered restrictive in any way. The scope of the invention is given by the appended claims, rather than the preceding description, and all variations and equivalents which fall within the range of the claims are intended to be embraced therein.
This application is a divisional application of application Ser. No. 09/976,040, filed on Oct. 15, 2001, now U.S. Pat. No. 7,298,866, the entire content of which is hereby incorporated by reference.
Number | Name | Date | Kind |
---|---|---|---|
33536 | Wang | Oct 1861 | A |
3646264 | Waugh | Feb 1972 | A |
4521909 | Wang | Jun 1985 | A |
5068910 | Ogura | Nov 1991 | A |
5361311 | Wilson | Nov 1994 | A |
5434932 | Scott | Jul 1995 | A |
5694493 | Tuli | Dec 1997 | A |
5757954 | Kaun et al. | May 1998 | A |
5815590 | Buttner et al. | Sep 1998 | A |
5835634 | Abrams | Nov 1998 | A |
5848189 | Pearson et al. | Dec 1998 | A |
6259396 | Pham et al. | Jul 2001 | B1 |
6337654 | Richardson et al. | Jan 2002 | B1 |
6347264 | Nicosia et al. | Feb 2002 | B2 |
6437728 | Richardson et al. | Aug 2002 | B1 |
6489920 | Anders et al. | Dec 2002 | B1 |
6650273 | Obenshain | Nov 2003 | B1 |
6711279 | Hamza et al. | Mar 2004 | B1 |
6795590 | Chen | Sep 2004 | B1 |
6832000 | Herman et al. | Dec 2004 | B2 |
6841780 | Cofer et al. | Jan 2005 | B2 |
7298866 | Lee | Nov 2007 | B2 |
20020004692 | Nicosia et al. | Jan 2002 | A1 |
20020057216 | Richardson et al. | May 2002 | A1 |
20020124171 | Rhoads | Sep 2002 | A1 |
20020125435 | Cofer et al. | Sep 2002 | A1 |
20020140815 | Herman et al. | Oct 2002 | A1 |
20020147544 | Nicosia et al. | Oct 2002 | A1 |
20020154833 | Koch et al. | Oct 2002 | A1 |
20020176624 | Kostrzewski et al. | Nov 2002 | A1 |
20040146184 | Hamza et al. | Jul 2004 | A1 |
Number | Date | Country | |
---|---|---|---|
20070269078 A1 | Nov 2007 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 09976040 | Oct 2001 | US |
Child | 11878921 | US |