Patent Grant
7496218
1. Field of the Invention
This invention relates to image analysis and, more specifically, to a system and method for identifying objects of interest in image data.
2. Background of the Related Art
Computer-aided image recognition systems rely solely on the pixel content contained in a two-dimensional image. The image analysis relies entirely on pixel luminance or color, and/or spatial relationship of pixels to one another. In addition, image recognition systems utilize statistical analysis methodologies that must assume that the forms of the underlying density (distribution) functions distinguishing the image objects are known (i.e., parametric densities). Classical parametric densities are usually unimodal with a single local maximum distribution of optic characteristics, such as density or color.
However, most real-world image analysis problems involve multi-modal densities, often with distributed low-dimensional densities making identification with existing pattern recognition approaches difficult, if not impossible. The following are some of the specific issues limiting existing image analysis methodologies:
(1) input data (image objects) need to be parametric;
(2) did not adjust for scale, rotation, perspective, size, etc.;
(3) classes of objects need to be statistically distinct in the image;
(4) black and white and grayscale processing is insufficient to identify complex images; and
(5) color processing can be very computationally intensive.
An object of the invention is to solve at least the above problems and/or disadvantages and to provide at least the advantages described hereinafter.
Therefore, an object of the present invention is to provide an expert system capable of detecting objects of interest in image data with a high degree of statistical confidence and accuracy.
Another object of the present invention is to provide a system and method that does not rely on a prior knowledge of an objects shape, volume, texture or density to be able to locate and identify a specific object or object type in an image.
Another object of the present invention is to provide a system and method of identifying objects of interest in image data that is effective at analyzing images in both two- and three-dimensional representational space using either pixels or voxels.
Another object of the present invention is to provide a system and method of distinguishing a class of known objects from objects of similar color and texture in image data, whether or not they have been previously observed by the system.
Another object of the present invention is to provide a system and method of identifying objects of interest in image data that works with very difficult to distinguish/classify image object types, such as: (i) random data; (ii) non-parametric data; and (iii) different object types in original images.
Another object of the present invention is to provide a system and method of identifying objects of interest in image data that can cause either convergence or divergence of image object characteristics.
Another object of the present invention is to provide a system and method of identifying objects of interest in image data that can preserve object self-similarity during transformations.
Another object of the present invention is to provide a system and method of identifying objects of interest in image data that is deterministic and stable in its behavior.
To achieve the at least above objects, in whole or in part, there is provided a method of identifying a threat object of interest in X-ray image data, comprising receiving the X-ray image data, and applying at least one bifurcation transform to the X-ray image data to effect divergence of the threat object of interest from other objects.
To achieve the at least above objects, in whole or in part, there is also provided an apparatus configured to identify a threat object of interest in X-ray image data, comprising an input device configured to receive the X-ray image data, and an image transformation recognition system configured to apply at least one bifurcation transform to the X-ray image data to effect divergence of the threat object of interest from other objects.
To achieve the at least above objects, in whole or in part, there is also provided a method of creating a bifurcation transform for a class of threat objects, comprising selecting a point operation, performing said point operation on a subset of images, wherein said subset of images includes at least one image containing an object in said class of threat objects, and repeating said selecting and said performing steps until said point operation bifurcates said object.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objects and advantages of the invention may be realized and attained as particularly pointed out in the appended claims.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Patent Office upon request and payment of the necessary fee.
The invention will be described in detail with reference to the following drawings, in which like reference numerals refer to like elements, wherein:
The following definitions hold throughout the contents of this application. If additional or alternative definitions of the same or similar words are provided herein, those definitions should be included herein as well.
Statistically identical: Two sets of data are referred to as “statistically identical” if under one or more types of statistics or observation there is essentially no discernable difference between them.
Point operation: Point operation is a mapping of a plurality of data from one space to another space which, for example, can be a point-to-point mapping from one coordinate system to a different coordinate system. Such data can be represented, for example, by coordinates such as (x, y) and mapped to different coordinates (α,β) values of pixels in an image.
Z effective (Zeff): Is the effective atomic number for a mixture/compound of elements. It is an atomic number of a hypothetical uniform material of a single element with an attenuation coefficient equal to the coefficient of the mixture/compound. Z effective can be a fractional number and depends not only on the content of the mixture/compound, but also on the energy spectrum of the x-rays.
Transform: A transform is an operation that changes or transforms a first set of data points into a second set of data points.
Bifurcate: To bifurcate is to cause a data set to change in a manner such that information that was otherwise not readily or easily obtainable becomes available or accessible.
Bifurcation transform: A bifurcation transform is a transform which when operating on data such as a segment or subset of an image, causes information relating to the content of the data that otherwise would not have been readily or easily apparent to become available or more easily apparent or accessible.
For example, when applying a bifurcation transform to an image or a segment of the image, information regarding the contents of the image which would not have been easily recognized prior to application of the bifurcation transform becomes more apparent or known. For example, two objects in the same image that are statistically indistinguishable become statistically distinguishable after the bifurcation transform is applied.
Hyperspectral data: Hyperspectral data is data that is obtained from a plurality of sensors at a plurality of wavelengths or energies. A single pixel or hyperspectral datum can have hundreds or more values, one for each energy or wavelength. Hyperspectral data can include one pixel, a plurality of pixels, or a segment of an image of pixels, etc., with said content. As contained herein, it should be noted that hyperspectral data can be treated in a manner analogous to the manner in which data resulting from a bifurcation transform is treated throughout this application for systems and methods for threat or object recognition, identification, image normalization and all other processes and systems discussed herein.
For example, a bifurcation transform can be applied to hyperspectral data in order to extract information from the hyperspectral data that would not otherwise have been apparent. Bifurcation transforms can be applied to a plurality of pixels at a single wavelength of hyperspectral data or multiple wavelengths of one or more pixels of hyperspectral data in order to observe information that would otherwise not have been apparent.
Nodal point: A nodal point is a point on a transform at which pixels close to the nodal point can observe a significantly distinguishable change in value upon application of the transform.
Object: An object can be a person, place or thing.
Object of interest: An object of interest is a class or type of object such as explosives, guns, tumors, metals, knives, etc. An object of interest can also be a region with a particular type of rocks, vegetation, etc.
Threat: A threat is a type of object of interest which typically but not necessarily could be dangerous.
Image receiver: An image receiver can include a process, a processor, software, firmware and/or hardware that receives image data.
Image mapping unit: An image mapping unit can be a processor, a process, software, firmware and/or hardware that maps image data to predetermined coordinate systems or spaces.
Comparing unit: A comparing unit can be hardware, firmware, software, a process and/or processor that can compare data to determine whether there is a difference in the data.
Color space: A color space is a space in which data can be arranged or mapped. One example is a space associated with red, green, blue (RGB). However, it can be associated with any number and types of colors or color representations in any number of dimensions.
Predetermined color space: A predetermined color space is a space that is designed to represent data in a manner that is useful and that could, for example, causes information that may not have otherwise been apparent to present itself or become obtainable or more apparent.
RGB DNA: RGB DNA refers to a representation in a predetermined color space of most or all possible values of colors which can be produced from a given image source. Here, the values of colors again are not limited to visual colors but are representations of values, energies, etc., that can be produced by the image system.
Signature: A signature can be a representation of an object of interest or a feature of interest in a predetermined space and a predetermined color space. This applies to both hyperspectral data and/or image data.
Template: A template is part or all of an RGB DNA and corresponds to an image source or that corresponds to a feature or object of interest for part or all of a mapping to a predetermined color space.
Algorithms: From time to time, transforms and/or bifurcation transforms are referred to herein as algorithms.
Algorithms and systems discussed throughout this application can be implemented using software, hardware, and firmware.
The analysis capabilities of the present invention can apply to a multiplicity of input devices created from different electromagnetic and sound emanating sources such as ultraviolet, visual light, infra-red, gamma particles, alpha particles, etc.
The present invention identifies objects of interest in image data utilizing a process herein termed “Image Transformation Recognition” (ITR). The ITR process can cause different yet statistically identical objects in a single image to diverge in their measurable properties. This phenomenon can be compared to the dynamic discontinuities observed in other fields of science, such as fluid mechanics. An aspect of the present invention is the discovery that objects in images, when subjected to special transformations, will exhibit radically different responses based on the physical properties of the imaged objects. Using the system and methods of the present invention, certain objects that appear indistinguishable from other objects to the eye or computer recognition systems, or are otherwise statistically identical, generate radically different and statistically significant differences that can be measured.
An aspect of the present invention is the discovery that objects in images can be driven to a point of non-linearity by certain transform parameters. As these transform parameters are increased, the behavior of the system progresses from one of simple stability, through a series of structural changes, to a state of a unique and radical change based on the interaction of the real object to the imaging modality characteristics. This point of rapid departure from stability is called the “point of bifurcation.” Bifurcation theory, simply stated, means “applying a little bit to achieve a big difference.”
Catastrophe theory, of which bifurcation theory is a subset, indicates that a dynamical system is or can become chaotic if it: (1) has a dense collection of points; (2) is sensitive to the initial condition of the system (so that initially nearby points can evolve quickly into very different states); and (3) is topologically transitive (neighborhoods of points eventually get flung out to “big” sets).
Images meet all three of the above criteria. An aspect of the present invention is that one can apply this “principle of bifurcation” to the apparent (non-dynamic) stability of fixed points or pixels in an image and, by altering one or more parameter values, give rise to a set of new, distinct and clearly divergent image objects. Because each original object captured in an image responds uniquely at its point of bifurcation, the methods of the present invention can be used in an image recognition system to distinguish and measure objects. It is particularly useful in separating and identifying objects that have almost identical color, density and volume.
The ITR system and method provides at least the following advantages over prior image extraction methodologies:
(1) It is an expert system capable of detecting objects with a high degree of statistical confidence and accuracy/precision;
(2) It does not rely on a prior knowledge of an objects shape, volume, texture or density to be able to locate and identify a specific object or object type in the image;
(3) It is effective at analyzing images in multi-dimensional representational space using either pixels or voxels;
(4) It is most powerful where a class of known objects is to be distinguished from objects of similar color and texture, whether or not they have not been previously observed or trained by the ITR system;
(5) It works with very difficult to distinguish/classify image object types, such as: (i) random data; (ii) non-parametric data; and (iii) different object types in original images (threats and non-threats for example or different types of threats) have indistinguishable differences between their features when analyzed statistically (they continually are determined to be in the same class when applying pattern recognition/statistical analytic methods);
(6) It performs equally well with both parametric and nonparametric statistical data sampling techniques;
(7) It can more effectively apply statistical analysis tools to distinguish data;
(8) It can cause either convergence or divergence of image object features;
(9) It can preserve object geometrical integrity during transformations; and
(10) It is deterministic and stable in its behavior.
In one exemplary embodiment of the present invention, special transformations are applied to images in an iterative “filter chain” sequence. This process is herein referred to as a Simple Signature Iterative Clustering (SSIC) process.
The nature of the sequence of transforms causes objects in the image to exhibit radically different responses based on the physical properties inherent in the original objects. Using the SSIC process, certain objects that appear indistinguishable to the eye or computer recognition systems from other objects, generate radically different and statistically significant differences that can be easily measured.
As transform parameters are increased, the behavior of the objects progresses from one of simple stability, through a sequence of structural changes, to a state of a unique and radical change (point of non-linearity) based on the interaction of the real object to the imaging modality characteristics.
The ITR process works with an apparently stable set of fixed points or pixels in an image and, by altering one or more parameter values, giving rise to a set of new, distinct, and clearly divergent image objects. The ITR process is most effective when applied to images that exhibit the following three characteristics:
Because of the nature of the transformations known and utilized in the science of digital image processing to date, the ITR system and method of the present invention has neither been known nor utilized. Commonly used and understood transforms work within the domain where images maintain equilibrium. These transforms do not work where image/object discontinuities occur.
As will be discussed in more detail below, the ITR method starts by first segmenting the image into objects of interest, then applying different filter sequences to the same original pixels in the identified objects of interest using the process. In this way, the process is not limited to a linear sequence of filter processing.
Because of the unique nature of the segmentation process using this iterative approach, objects within objects can be examined. As an example, an explosive inside of a metal container can be located by first locating all containers, remapping the original pixel data with known coordinates in the image and then examining the remapped original pixels in the identified object(s) in the image for threats with additional filter sequences.
With the ITR process, transforms can be tuned to optimize clustering of images. In addition, the process works for both image segmentation and feature generation through an iterative process of applying image transforms. It is defined mathematically as a reaching a “Point of Attraction”.
The operation of the ITR system 130 of
At step 210, at least one bifurcation transform is applied to the image data 120 by the ITR system 130, and transformed image data 140 is generated. The at least one bifurcation transform is adapted to cause the object of interest to diverge from other objects in the image. The at least one bifurcation transform will be discussed in more detail below.
Next, at step 220, the object of interest is identified in the transformed image data 140 based on the object's response to the at least one bifurcation transform.
The at least one bifurcation transform is preferably a point operation. A point operation converts a single input image into a single output image. Each output pixel's value depends only on the gray level of its corresponding pixel in the input image. Input pixel coordinates correlate to output pixel coordinates such that Xi, Yi→Xo, Yo. A point operation does not change the spatial relationships within an image. This is quite different from local operations where the value of neighboring pixels determines the value of the output pixel.
Point operations can correlate both gray levels and individual color channels in images. One example of a point operation is shown in the histogram of
Point operations are predictable in how they modify the gray-level histograms of an image. Point operations are typically used to optimize images by adjusting the contrast or brightness of an image. This process is known as contrast enhancing or gray-scale transformations. They are typically used as a copying technique, except that the gray levels are modified according to the specified gray-scale transformation function. Point operations are also typically used for photometric calibration, contrast enhancement, monitor display calibration, thresholding and clipping to limit the number of levels of gray in an image. The point operation is specified by the transformation function and can be defined as:
B(x,y)=ƒ[A(x,y)],
where A is an input image and B is an output image.
The at least one bifurcation transform used in the ITR system 130 can be either linear or non-linear point operations, or both. Linear point operations contain straight lines in their histogram representation, while non-linear (logarithmic, exponential, and hyperbolic/parabolic) point operations have curved lines. Non-linear point operations are used for changing the brightness/contrast of a particular part of an image relative to the rest of the image. This can allow the midpoints of an image to be brightened or darkened while maintaining blacks and white in the picture.
The present invention utilizes radical grayscale, color channel or a combination of luminance and color channel bifurcation transforms (point operations) to achieve image object bifurcation for purposes of image analysis and pattern recognition of objects. The placement of the nodal points in the bifurcation transform is one key parameter. An example of nodal point placements are shown in the bifurcation transform example illustrated by the histogram of
The nodal points in the bifurcation transforms (point operations) used in the present invention are placed so as to frequently create radical differences in color or luminance between image objects that otherwise statistically identical.
This is illustrated in the sample bifurcation transform of
Note that the orange background has gone a very different color from the shoes 300 and the bar 310 on the left side of the suitcase. The bifurcation transform of
As can be seen by the input and transformed images shown in
Data points connecting the nodes can be calculated using several established methods. A common method of mathematically calculating the data points between nodes is through the use of cubic splines.
Additional imaging processes are preferably applied in the process of object recognition to accomplish specific tasks. Median and dilate algorithms cause neighboring pixels to behave in similar ways during the bifurcation transformation, and may be applied to assure the objects' integrity during the transformation process.
The method starts at step 400, where the original image, such as the suitcase image shown in
At step 410, the image is segmented by applying a color determining transform that affect specifically those objects that match a certain color/density/effective atomic number characteristics. Objects of interest are isolated and identified by their responses to the sequence of filters. The image segmentation step is preferably a series of sub-steps.
At step 420, features are extracted by subjecting the original pixels of the areas of interest identified in step 410 to at least one feature extraction process. It is at this step that at least one bifurcation transform is applied to the original pixels of the areas of interest identified in step 410.
In the image examples shown in
(1) Replace colors
(2) Maximum filter 3×3
(3) Median filter 3×3
(4) Levels and Gamma Luminance=66 black level and 255 white level and Green levels=189 black, 255 white and gamma=9.9
(5) Apply bifurcation transform
(6) Maximum filter 3×3
(7) Replace black with white
(8) Median filter 3×3
The image shown in
The second feature extraction process applied as part of step 420 distinguishes the two remaining objects 520 and 530. Replace color algorithms (same as process step (2) above) are applied, then two levels (two of process step (4) above) and then the bifurcation transform shown in
At step 430, the objects are classified based on their response at the feature extraction step (step 420). The object of interest 530 is measured in this process for its orange content. The peanut butter jar 520 shows green as its primary vector, and is therefore rejected. At step 440, the remaining object 530 is identified as an object of interest.
Determination of distinguishing vectors generated at step 420 is determined by statistical analysis of both objects of interest and other possible objects. This can be accomplished by applying existing statistical analysis. One example of a decision tree based on one process, along with a sample logic rule set for the decision tree (which is integrated into software) is shown below in Appendix A. In one approach, only leafs that show 100%/0% differences between objects with no overlap are used.
This is then entered into code and accessed from an object oriented scripting language called TAL. TAL (Threat Access Language) is linked to functions and logic in the PinPoint software code. Its design allows for rapid extension of the principals to new objects of interest. A sample of TAL is shown below.
A second pass is now made with all images. The rules defined above can now eliminate objects identified in process 1. A second process that follows the logic rules will now create objects of new colors for the remaining objects of interest. The vectors (metrics) of the transformed objects of interest are examined. Multiple qualitative approaches may be used in the evaluation of the objects, such as prototype performance and figure of merit. Metrics in the spatial domain, such as image amplitude (luminance, tristimulus value, spectral value) utilizing different degrees of freedom, the quantitative shape descriptions of a first-order histogram, such as Standard Deviation, Mean, Median, Skewness, Kurtosis, Energy and Entropy, % Color for red, green, and blue ratios between colors (total number of yellow pixels in the object/the total number of red pixels in the object), object symmetry, are some, but not all, of the possible measurements that can be used. Additional metrics can be created by applying spectrally-based processes such as Fourier and Wavelet transforms to the previously modified objects of interest or by analyzing eigenvalue plots produced from a Principal Components Analysis.
A color replacement technique is used to further emphasize tendencies of color changes. For example, objects that contain a value on the red channel>100, can be remapped to a level of 255 red so all bright red colors are made pure red. This is used to help identify metal objects that have varying densities. The ratio of these two colors is now a fixed ratio between the two. Color 1/Color 2=Invariant Ratio (IR).
This IR can now help indicate the presence of a certain metal objects regardless of its orientation in the image. It can also be correlated to geometric measurements using tools that determine boundaries and shapes. An example would be the correlation of IR with boundaries and centroid location. Other process may additionally be used as well.
The system and methods of the present invention are based on a methodology that is not restricted to a specific image type or imaging modality. It is capable of identifying and distinguishing a broad range of object types across a broad range of imaging applications. It works equally as well in applications such as CT scans, MRI, PET scans, mammography, cancer cell detection, geographic information systems, and remote sensing. It can identify and distinguish metal objects as well.
In medicine, the present invention is capable of, for example, distinguishing cancer cell growth in blood samples and is being tested with both mammograms and x-rays of lungs. For example,
The statistical processing provided by the present invention can be extended to integrate data from a patient's familial history, blood tests, x-rays, CT, PET (Positron Emission Tomography), and MRI scans into a single integrated analysis for radiologists, oncologists and the patient's personal physician. It can also assist drug companies in reducing costs by minimizing testing time for new drug certification.
The ITR system 130 can be implemented with a general purpose computer. However, it can also be implemented with a special purpose computer, programmed microprocessor or microcontroller and peripheral integrated circuit elements, ASICs or other integrated circuits, hardwired electronic or logic circuits such as discrete element circuits, programmable logic devices such as FPGA, PLD, PLA or PAL or the like. In general, any device on which a finite state machine capable of executing code for implementing the process steps of
Input channel 110 may be, include or interface to any one or more of, for instance, the Internet, an intranet, a PAN (Personal Area Network), a LAN (Local Area Network), a WAN (Wide Area Network) or a MAN (Metropolitan Area Network), a storage area network (SAN), a frame relay connection, an Advanced Intelligent Network (AIN) connection, a synchronous optical network (SONET) connection, a digital T1, T3, E1 or E3 line, Digital Data Service (DDS) connection, DSL (Digital Subscriber Line) connection, an Ethernet connection, an ISDN (Integrated Services Digital Network) line, a dial-up port such as a V.90, V.34bis analog modem connection, a cable modem, and ATM (Asynchronous Transfer Mode) connection, or an FDDI (Fiber Distributed Data Interface) or CDDI (Copper Distributed Data Interface) connection. Input channel 110 may furthermore be, include or interface to any one or more of a WAP (Wireless Application Protocol) link, a GPRS (General Packet Radio Service) link, a GSM (Global System for Mobile Communication) link, CDMA (Code Division Multiple Access) or TDMA (Time Division Multiple Access) link such as a cellular phone channel, a GPS (Global Positioning System) link, CDPD (Cellular Digital Packet Data), a RIM (Research in Motion, Limited) duplex paging type device, a Bluetooth radio link, or an IEEE 802.11-based radio frequency link. Input channel 110 may yet further be, include or interface to any one or more of an RS-232 serial connection, an IEEE-1394 (Firewire) connection, a Fiber Channel connection, an IrDA (infrared) port, a SCSI (Small Computer Systems Interface) connection, a USB (Universal Serial Bus) connection or other wired or wireless, digital or analog interface or connection.
The foregoing embodiments and advantages are merely exemplary, and are not to be construed as limiting the present invention. The present teaching can be readily applied to other types of apparatuses. The description of the present invention is intended to be illustrative, and not to limit the scope of the claims. Many alternatives, modifications, and variations will be apparent to those skilled in the art. Various changes may be made without departing from the spirit and scope of the present invention, as defined in the following claims.
This application claims priority to U.S. Provisional Patent Application No. 60/574,220, filed May 26, 2004, U.S. Provisional Patent Application No. 60/574,221, filed May 26, 2004, U.S. Provisional Patent Application No. 60/578,872 filed Jun. 14, 2004 and U.S. Provisional Application No. 60/661,477, filed Mar. 15, 2005, which are incorporated herein by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4991092 | Greensite | Feb 1991 | A |
| 5280428 | Wu et al. | Jan 1994 | A |
| 5754676 | Komiya et al. | May 1998 | A |
| 5854851 | Bamberger et al. | Dec 1998 | A |
| 5970164 | Bamberger et al. | Oct 1999 | A |
| 5987345 | Engelmann et al. | Nov 1999 | A |
| 6088473 | Xu et al. | Jul 2000 | A |
| 20010019623 | Takeo | Sep 2001 | A1 |
| 20020094114 | Ogino | Jul 2002 | A1 |
| 20030053673 | Dewaele | Mar 2003 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20060013463 A1 | Jan 2006 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60661477 | Mar 2005 | US | |
| 60578872 | Jun 2004 | US | |
| 60574220 | May 2004 | US | |
| 60574221 | May 2004 | US |
