1. Field of the Invention
The invention relates generally to ultrasonic and radiographic medical imaging, and particularly to three-dimensional, ultrasonic mammography for the purpose of detecting and visualizing breast lesions.
2. Description of the Related Art
The standard of care for managing breast disease includes (in order) detection, diagnostic work-up, definitive diagnosis, treatment, adjuvant therapy and follow-up. In a large number of cases, the health care provider will not consider a biopsy justified and rather will opt to follow the situation with examinations every three to six months. In other cases the health care provider will typically also want to follow the progress of the lesion after adjuvant therapies are administered. This implies repeated imaging of the region of tissue containing the lesion and comparison of the earlier and later images, to detect and characterize any changes in the lesion.
Various imaging modalities are available to visualize lesions, including X-ray (most common), ultrasonography, magnetic resonance imaging and other known methods. Whatever imaging method is used, to make meaningful before/after comparisons it is necessary to insure that the same region of tissue is under examination. Preferably the tissue should be projected from the same viewing angle in both before and after images. Comparison and even interpretation of images is complicated in part because the region of interest is a three-dimensional volume, while the image is a projection onto a two-dimensional plane or planes. Three dimensional positions and relationships must be inferred or calculated, based upon the available projections. Additional difficulties result because the breast may move slightly or be slightly deformed between the earlier and later images. Even slight misalignment makes it difficult to determine whether the same lesion is comparable in earlier and later images. Even if positions, compressions and projection angle correspond well in before and after images, side-by-side comparison of complex tissue patterns is difficult, and slight changes in tissue may escape detection.
An image processing system and method visually documents historical changes in a medical image by processing and comparing historical with later images. A composite image is created which visually emphasizes the temporal differences between historical and later images. Preferably three-dimensional, digitized images, displayable in various projections, are stored for archival purposes on computer readable media.
To properly relate the historical and later images, an image processor preferably exploits an optical correlator to register the images accurately, by finding a transformation which produces a pre-determined degree of correlation between the images, then adjusting one image accordingly. The images are then compared, preferably volume element by volume element (“voxel by voxel”), to detect temporal differences between historical and later images. A composite image is displayed with synthetic colors or other visual clues to emphasize the scalars in the imaged tissue, based on temporal differences between the historical and later images.
a is a summary level flow diagram of a method for comparing historical with more recent medical imagery in accordance with:the invention;
b is a flow diagram showing the continuation of the method of
a is a flow chart showing the initial steps of a method of registering earlier with later images, suitable for use in the registration step of
b is a flow chart continuing from
a, 6b, and 6c, show simplified examples of an input image, a filter template, and a resulting correlation output image, respectively, in an example of a suitable cross-correlation operation which discovers positional offset of correlated images in the method of
Overview of Apparatus
The image processor 24 is preferably a 64 bit workstation such as the Silicon Graphics 02, although less powerful processors could be used at the expense of speed or resolution. The ultrasonographic imaging system 20 should preferably be capable of sweep scanning, to provide the multiple slices of imagery for assembly into a three-dimensional data set. Either a dedicated processor integral with the ultrasonographic imaging system 20 could provide such assembly, or it could alternatively be performed by-the image processor 24.
Overview of Method
a and 2b summarize a method used in accordance with the invention to compare historical with later mammographic images. First, the image processor 24 inputs (step 50) historical image data. Preferably the image data is available from a data network 39, but alternatively it could be obtained by scanning film imagery by digitizing scanner 40. If available, three dimensional historical image data is preferred. As another option, the historical data could be retrieved from a mass storage device 38 rather than loading by inputting the data.
Next, in step 52, pathological feature data is preferably loaded by the image processor 24 (preferably from a data network but alternatively by retrieval from storage or through another input channel). The pathological feature data is useful for narrowing the region of interest for image processing, and could be provided in any of several useful forms. Typically, a physical region of interest (ROI) in the historical image has been previously identified by a radiologist or other professional, and the coordinates of that region recorded in association with the historical image. In that case, the coordinates of the ROI are the pathological feature data. Alternatively, certain image shapes or characteristics, stored in a pathological image library, could be compared with the current image by image correlation to define the pathological region as described below in connection with
Next. (step 54) a coarse resolution, three-dimensional image sweep of the patient's breast is taken, preferably by sweeping an ultrasonic imaging head linearly across the breast thereby obtaining a plurality of image slices which collectively define a three-dimensional image. The resulting coarse image is stored (also in step 54). The coarse breast image of step 54 is then manipulated by the image processor to properly register it (step 56) with the historical imagery, to at least a preliminary degree of registration. This initial, coarse level of registration can be performed by processing only a very simplified image, to improve processing speed. For example, the image can be first compared pixel by pixel (or voxel by voxel) to a threshold value, and a binary product image produced based on the result at each pixel (voxel). In the threshold processed image, only the most obvious features, such as the breast outline and chest wall, will be recognizable. The images can then be rotated and shifted either manually (by operator command) or by the registration method described below in connection with
Next, to obtain improved registration between the historical and new image, the image processor correlates the pathological feature set with the new image (step 58), preferably by the method described below in connection with
The image processor 24 next proceeds to register the new, finely resolved ROI image with the historical image of the corresponding ROI (step 62). This new registration step 62 is performed with finer resolution than the previous coarse registration (step 56), but preferably the operations are restricted to a smaller region of imagery, thus preserving a practical, limited computation time. This results in accurate and precise registration between the go historical and the new tissue ROIs. Preferably, both ROIs are three-dimensional volume regions which are aligned by registration in three dimensions.
After the historical and new images are aligned, they are compared in step 64 on a point-by-point basis, to obtain a temporal difference image which emphasizes the temporal changes which have occurred. For example, if digitized two dimensional images are considered, the images are preferably first normalized to a common average value, then corresponding pixels are subtracted and the temporal difference values stored in a corresponding position in a temporal difference image. Analogously, if three-dimensional imagery is available (which is greatly preferred), voxels at corresponding locations in the early and later images are subtracted one from the other, and the result stored in a voxel of a temporal difference (volume) image. (A “voxel” is a unit of graphic information that defines a small volume element in three-dimensional space. It is the three-dimensional analog of a “pixel” which defines an area element in two-dimensional space.) Thus, normalized before and after images can be subtractively compared, voxel-by-voxel, to obtain a three-dimensional temporal difference image which represents the temporal differences between the before and after images.
As a result of step 64, at least three images are available to the image processor 24: an early (historical) image; a current image; and a temporal difference image. These images are then combined and/or selected for display (step 66) in various ways, as directed by user input from input device 32. Finally, the resulting image or images are displayed (step 68) and preferably stored (step 70) and/or exported (step 72, preferably via the data network). It is highly preferable that the imagery be coded to emphasize temporal differences between the earlier and later images. For example, the temporal difference image can be coded in a distinct color for display based on correlation output values, then superimposed on the early image, the later image, or a composite. Various other conventional methods could be equivalently employed to visually emphasize temporal image differences. For example, in one variation the temporal difference image can be displayed in intermittent, flickering display superimposed on either of the historical or current images. Significantly, the invention easily detects temporal differences between the earlier and later images, visually displays the images, and visually emphasizes these temporal differences.
The product images from step 66 should preferably be recorded (step 72) for archival purposes, preferably on a high-density, long persistence medium such as magnetic tape memory or writable CD-ROM. Preferably a three-dimensional volume image should be stored, but to save storage space, operator selected projections can be derived (by programming the image processor 24 under control from the operator) and stored as two-dimensional imagery. Three dimensional imagery is preferred, despite its greater storage demands, because it can be more easily compared with other images (either previous history or subsequent development). The three-dimensional imagery can be digitally manipulated by rotation, scaling, translation then projected to provide comparison with any view; two dimensional imagery does not provide sufficient information to carry out the aforementioned manipulations without loss of information or introduction of ambiguities in the resulting image. The location of significant changes in the image should preferably be tagged with an identifier, to cross-link to a potential future tissue sample specimen number.
The method may be repeated to compare multiple historical images. Either one or multiple combined images can be stored for archival and to document historical developments.
Registration Method:
The success of the above described procedure depends in part upon the accuracy with which the earlier and later images are registered (in steps 56 and 62 of
With the breast 22 in position, ultrasonic scanning is preferably performed in planes (or surfaces), which will in general be non-parallel to that of the Y-Z plane in FIG. 3.
Multiple slices can be scanned either by providing multiple scanheads, a two-dimensional scanhead array, or by moving the scanhead across the breast, for example in the x′ direction in FIG. 4. The planes of only a few of the multiple slices, specifically slices 98, 100, and 102, are shown. In practice a large number of slices is desirable, for better resolution. A complete set of such slices is preferably scanned to form a three dimensional information set for at least some region of interest (ROI) chosen from the breast, which is preferably stored in a data structure (such as a three-dimensional array) to represent a three-dimensional image.
Ultrasonographic equipment is available commercially which can be used as the ultrasonographic imaging system 20 described above. A two-dimensional array of ultrasonographic data is usable by the invention, but with a diminished amount of useful information in the resulting display.
Keeping in mind the exemplary coordinate system of
The purpose of collapsing the three-dimensional images into two dimensions is to facilitate correlation, which is preferably accomplished by use of an optical correlator. Such correlators conventionally require two-dimensional imagery as correlation inputs. In some applications, the historical imagery will already be in two-dimensional projection, having been previously selected and projected to adequately visualize the lesion(s) of interest. In such a case only the later imagery must be collapsed.
As an alternative to collapsing the images by simple projection, as described above, a three-dimensional image can be processed to create a “discriminate filter.” Such filters are known and facilitate registration by providing a correlation filter which a composite of many images which differ by rotation about an axis. This allows testing of multiple, rotated slices simultaneously, by correlation with a composite discriminate filter. Once a discriminate filter is determined with acceptable correlation (at least equal to a pre-defined level of correlation) an individual filter is then selected from the set of slices which was used to compose the composite. In some cases, this method can greatly accelerate the search for best correlation.
The projected images are optionally further pre-processed (step 116). Pre-processing 116 can include any of a variety of known image processing techniques including (without limitation) contrast modification, smoothing, geometric transformation, thresholding or the selection of a region of interest. Depending on the type of optical correlator 30 used, as discussed in detail below, this step may also include two-dimensional Fourier transformation of the digitized x-ray image to prepare it for subsequent optical correlation in a Vanderlugt optical correlator.
Next the image processor 24 adjusts (step 118) the relative scale of the two images so that they better correspond in scale. This can be done by various methods. For example, one method is to match the total area of the cross section of the breast area between the outer outline and the chest wall in both images. In this method, the images should preferably first be processed to remove low contrast features, leaving only the easily visible outline of the breast and the chest wall. The area between these features in the two dimensional images is then measured, for example by numerical integration by the image processor 24. The area should correspond in both images. If the areas do not correspond, it is possible that the degree of compression has changed, which can compress or expand the breast. A scaling correction factor is then preferably applied in step 118 to correct as closely as possible. On the other hand, it is possible in many cases to maintain a relatively constant compression. In such cases, little or no re-scaling is necessary.
After scale correction; the image processor 24 determines the rotation and translation necessary to align the images, preferably by interactively performing the steps grouped within the instruction loop 122 in
Next, if the counter variable j has not reached jmax, the image processor 24 continues and rotates (step 136) one of the dual images relative to the other by some angular increment, for example by rotating the historical image one degree about an axis centered in the frame and parallel to the z axis. The counter is incremented (step 138) and the procedure loops back to step 130 to perform another cross-correlation, this time with the images rotated incrementally. The procedure is repeated until some number (jmax) of differently rotated correlations has been performed. The parameter jmax should be chosen to be large enough so that the range of the associated angle αj encompasses the anticipated maximum possible range of rotation. For breast examination in the geometry shown in
The rotations applied in step 136 are not limited to rotations about a single axis, but can include rotations about multiple independent axes (or, equivalently rotations about an axis obliquely oriented with respect to the orthogonal axes shown). This allows the correction for an oblique viewing angle of one image with respect to the viewing angle of the other.
After the counter j reaches jmax the image processor 24 exits the instruction loop 122. The procedure continues as diagramed in
It should be understood that in addition to discovering the correct scale and rotation angles, the cross-correlation (step 130) in the above described procedure produces an output image which reveals the correct translation (position shift) which best aligns the dual images. The translation information is contained in the result of the cross-correlation operation (preferably in two-dimensions) defined as:
H(x,y)=g(x,y)*f(x,y)=∫∫f(α,β)g(x-α,y-β)dαdβ (1)
Where f and g are functions of two variables (images), x and y are the spatial variables of the two-dimensional images, α, and β are dummy variables of integration, H is the cross-correlation of functions f and g, and the range of integration is across the entire image. If f and g differ only by a positional offset in x and y, H(x,y) will have a sharp peak at a position xp, yp which is displaced from a central, correlation alignment position (typically defined as x=0, y=0) by a displacement corresponding to the offset between f and g. This well known result has been used to recognize and locate features of initially unknown locations in a field by reference to a template. See, for example, Russ, John C., The Image Processing Handbook (CRC Press, 1992), pp. 218-24. A simplified example is shown in
Returning to
In one embodiment the invention utilizes additional three-dimensional information about the subject body by further correlating the images in order to align the z direction (depth). To accomplish this, the ultrasonographic imagery is first partitioned by image processor 24 into conveniently defined slices, for example slice 164 as shown in FIG. 7. Each slice includes one or more layers of the three dimensional ultrasonographic image data. The slices are defined and calculated by image processor 24, for example by summing data points along vertical vectors such as 165, to collapse multiple thin layers into a thicker slice (a “partial cumulative projection”). In
In a typical application, slices with thickness of 5 millimeters or less are suitable. Although planar slices are convenient for purposes of illustration, in some applications the slices might usefully be taken along non-planar contours. Such slices are also within the scope of the invention. Thinner slices are desirable for better depth definition with thin features.
To best align the earlier and later images, corresponding slices in each are individually correlated to find their best registration, for example by the method described above. In one variation of this technique, the slices are defined parallel to the X′Y′ plane. In another variation, the slices are defined in another plane (the slice under the scanhead, parallel to the Z′Y′ plane, for example) and registered by the above described method, then reassembled. By registering each slice, the three-dimensional imagery can be aligned to account for variations in shear or torque between the earlier and later image acquisitions, as might be caused by variable deformation of the breast under varying pressure.
The fine registration step of step 62 (in
After the earlier and later ROI images are adjusted to best register them, they can be either combined by adding, voxel-by-voxel, or compared by subtracting, voxel-by-voxel. Temporal differences between the images are easily highlighted by changing the color at voxels which show high variation between the dual images.
In the procedures depicted in
Optical correlators use wave optics to correlate images in two dimensions by first performing essentially a two-dimensional spatial Fourier transform on a two-dimensional source image. This method takes advantage of a well known mathematical property of the Fourier transform: many operations including correlation are more easily calculated in the Fourier transform domain than in the original spatial domain. Specifically, a two-dimensional correlation operation is defined by equation 1 (above), where f(x,y) and g(x,y) are the two-dimensional functions or images to be cross-correlated, and α and β are dummy variables of integration. This operation can be performed digitally for each point x,y by numerical techniques, but a very large number of calculations are required even for one image correlation. Performing such an operation digitally is very time consuming and requires inconvenient lengths of time on any but the fastest digital computers.
Unlike a conventional digital computer, an optical correlator can very rapidly perform a correlation operation, correlating a source image with a filter image by (1) optically Fourier transforming a source image, (2) comparing the source and filter image in the Fourier transform domain, and then (3) performing an inverse Fourier transformation to produce the correlation pattern in a spatial representation. An optical correlator can accomplish these operations much faster that a digital computer because the optical Fourier transformation is executed as a simultaneous operation on all points of the source image, using inherent properties of wave optics to generate the Fourier transform in two dimensions. The speed of the device is limited for practical purposes only by the available read and write speed of the data transfer to the correlator; the actual optical processes occur in fractions of a nanosecond in typical optical correlators.
The principles of the optical correlator are known, and have been described for example in the U.S. Pat. No. 5,311,359, to Lucas et al. Compact optical correlators suitable for use in the present invention are commercially available from Litton Data Systems, Inc., in Agoura Hills, Calif., as well as from other sources. Alternate types of optical correlators such as the Joint Transform Correlators described in U.S. Pat. No. 5,650,855 to Kirsch et al., U.S. Pat. No. 5,216,541 to Taksue et al. or U.S. Pat. No. 5,438,632 to Horner, may also be employed with the invention.
For purposes of describing the present invention, the optical correlator 30 may be considered functionally as an electro-optical device having three (electronic) ports, as shown in FIG. 9. The three ports include: (1) an image input port 210 for receiving an electronic signal encoding an input image for correlation; (2) a filter input port 212 for receiving a second electronic signal encoding a second image or “filter” for correlation; and (3) an output port 214, typically from a charge coupled device (CCD) imager, which converts the correlation image into electrical form for output. In addition the device requires a source (not shown) of preferably coherent electromagnetic radiation, typically a laser, which provides the medium used for computation.
Both the image input port 210 and the filter input port 212 are realized as two-dimensional spatial light modulators (SLMs) organized as two-dimensional image matrices, with addressable image pixels (typically arranged in the familiar row and column pattern). Accordingly, the input image must be formatted (suitably by image processor 24) to fit the matrix; and each pixel of data should preferably be addressed, under control of the image processor 24, to the spatially corresponding pixel on the SLM. For example, in one embodiment of the invention, the image input port and the filter input port are realized as 256×256 pixilated matrices. Accordingly, in this embodiment the image processor 24, as part of pre-processing step 116 (in
Note that in conventional, practical high speed optical correlators, the SLMs and the photodetector matrix consist of discrete pixels rather than a continuously modulatable surface. Thus, the Fourier transformation and correlation operations can only approximate the discrete cross-correlation (given by equation 1, above). However, the approximation obtained is sufficient for image processing for most applications.
When the input the filter images have been written to the input and filter ports 210 and 212, the optical correlator produces an output image which is a two dimensional output correlation pattern having an optical peak or peaks (bright spot) at the position of greatest correlation between the collapsed sonographic image and the radiographic image. The degree of correlation is indicated by the intensity of the output signal. The position of the output peak on the two-dimensional matrix of the correlator output CCD indicates the translations or shifts of the images relative to one another. The output image is read from the output photodetector (CCD) 214 by the image processor 24 in the conventional manner, typically by shifting the CCD voltage values out sequentially in rows (or columns) and then digitizing, the output levels. The peak amplitudes of the 2D array provide the extent of correlation (degree of correlation) between the images. This scalar information is a measure of the temporal difference, and can be used as a measure of knowledge between images and supports actions that lead to a successful treatment or evaluation.
In many applications in which two images are to be correlated, it is irrelevant which image provides the filter and which the corresponding input image, provided that the filter image is presented (in a Vanderlugt type correlator) in a Fourier transform representation. However, in some particular embodiments it may be preferable to route the historical imagery to the input port 210 rather than the filter port 212. For example, if discriminate filters are derived from the later ultrasound imagery and are to be correlated, these filters would usually be derived in a Frequency domain representation and would thus more appropriately routed to the Filter port 212. On the other hand, if a search for a correlation is to be done with a brute-force, match search, then the multiple projections derived from the later imagery should be applied to the input port 210 (thus avoiding the need to perform numerous two-dimensional Fourier transformations).
Although the invention is described in terms of linear transformations of the coordinates, such as translation, rotation, and scale multiplication, the invention is not limited to linear transformations. Non-linear transformations of coordinate systems may be useful in some applications. . For example, the ultrasonographic information may be obtained with the breast differently deformed, as by a change of position of the subject, or by instrument pressure. By applying a mathematical transformation, which may in general be non-linear, a better mapping of the deformed subject breast onto the original subject breast can be obtained. Similarly, some scanning techniques may involve curvilinear, non-cartesian coordinate Systems which would be treated with non-linear transformations.
While illustrative embodiments of the invention are described above, it will be evident to one skilled in the art that numerous variations, modifications and additional embodiments may be made without departing from the invention. For example, the construction of the ultrasound imaging system or the geometries and coordinate systems employed may be varied. Various means of data storage, transmission, or processing may be employed. The resolution or type of image that is sent from the image processor to the optical correlator could also be altered. Three-dimensional cross-correlations are also possible (but computationally complex). To the extent that such operations can be decomposed into multiple planar operations, the use of the optical correlator as described above could greatly accelerate computations. Accordingly, it is intended that the invention be limited only in terms of the appended claims.
Number | Name | Date | Kind |
---|---|---|---|
5042077 | Burke | Aug 1991 | A |
5433202 | Mitchell et al. | Jul 1995 | A |
5491627 | Zhang et al. | Feb 1996 | A |
5627907 | Gur et al. | May 1997 | A |
5644649 | Schoeters et al. | Jul 1997 | A |
5784431 | Kalend et al. | Jul 1998 | A |
5982953 | Yanagita et al. | Nov 1999 | A |
6285787 | Kawachi et al. | Sep 2001 | B1 |
6310477 | Schneider | Oct 2001 | B1 |
6351660 | Burke et al. | Feb 2002 | B1 |
6363163 | Xu et al. | Mar 2002 | B1 |
6453058 | Murthy et al. | Sep 2002 | B1 |
6477262 | Wang | Nov 2002 | B2 |
6484047 | Vilsmeier | Nov 2002 | B1 |
6538791 | Trezza | Mar 2003 | B2 |