METHOD FOR COMBINING MULTIPLE IMAGE DATA SETS INTO ONE MULTI-FUSED IMAGE

Abstract

In a method and an apparatus to visualize nuclear medicine data from different modalities in a single image, image data sets acquired with different modalities are aligned with a data set representing corresponding anatomical data. The image data in each data set are segmented into separate regions, representing respective structures of interest, with reference to a segmentation derived from anatomical data. For each region, a corresponding segment of image data is selected from a selected image data set. The selected segments of the image data set are combined to generate a multi-fused image of the regions, by applying spatially dependent look-up tables to the multiple image data sets, thereby to determine whether each data set is hidden or displayed in each region.

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention concerns a method and an apparatus to visualize nuclear medicine data from different modalities, in a single image.

2. Description of the Prior Art

The following definitions, acronyms and abbreviations are used herein:

PET Positron emission tomography

SPECT Single-photon emission tomography

NaF Sodium fluoride

NM Nuclear medicine

MIP Maximum intensity projection

CT Computed tomography

FDG Flodeoxyglucose

MPR Multi-planar reformatting/reconstruction/rendering

MRI Magnetic resonance imaging

RGBA Colour representation as red/green/blue and alpha (opacity)

LUT Look-up table: in this context, a LUT contains the transformation from observed values (here, uptake values) to visualized pixel/voxel values, often encoded as RGBA.

VOI Volume of Interest

HSV Hue Saturation Value

With the clinical availability of more and more nuclear imaging tracers, a detailed examination of a patient may combine information collected from image data acquired with more than one radioisotope. For instance a combination of ¹⁸F-FDG and ¹⁸F—NaF PET/CT increases the sensitivity for detection of osseous lesions compared to ¹⁸F-FDG PET/CT alone, independent of whether both scans were performed as separate studies or as a single study with simultaneous tracer injection, as described in Lin et al., “Prospective comparison of combined ¹⁸F-FDG and ¹⁸F—NaF PET/CT vs. ¹⁸F-FDG PET/CT imaging for detection of malignancy,” Eur J Nucl Med Mol Imaging 2012, 39, pp. 262-270. While in this example ¹⁸F—NaF PET/CT provides particular advantages in detecting bone metastasis (See Hetzel et al., “F-18 NaF PET for Detection of Bone Metastases in Lung Cancer: Accuracy, Cost-Effectiveness, and Impact on Patient Management,” Journal of Bone and Mineral Research 2003, 18(12), pp. 2206-2214), it can be expected that other tracers will prove themselves useful for detecting pathologies in other structures/organs. Hence, a combination of more than one tracer will, for certain medical questions, reveal more information to the reading physician than a single tracer alone can.

The increasing amount of information for the reading physician makes it at the same time more and more difficult to get a fast overview of the patient's condition and to focus on diagnostically relevant organs and structures. This is particularly true if more than one scan with multiple radioisotopes has been acquired.

Currently multiple studies with different tracers are typically reviewed individually or side-by-side. For this purpose, a LUT, which may be a colour LUT, is often applied to the functional data and the result is visualized as overlay on the corresponding anatomical data, if available. In the following, such visualization is referred to as fused imaging, or fusion.

Furthermore, 3D visualizations may be rendered using, for instance, a maximum-intensity-projection (MIP) to display an overview of the NM data.

In the case of multiple-tracers being combined in one study, no conventional dedicated rendering techniques are known to the inventors.

That is, similar to above, one LUT is applied for fused imaging or to generate a MIP. In that case the correlation between each tracer and the observed uptake is lost. Note that cases where uptake can be associated to the individual tracer, e.g., SPECT imaging of tracers emitting photons of different energies, can be regarded as the first use-case that includes multiple studies.

SUMMARY OF THE INVENTION

The present invention accordingly provides apparatus and methods to visualize NM data, e.g., acquired with PET/CT, SPECT/CT, SPECT/MR or PET/MR scanners, using multiple tracers in a single image.

The present invention accordingly provides methods wherein multiple image data sets are combined into one multi-fused image, as defined in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1C represent images of a patient according to respective modalities.

FIG. 1D represents a combined image assembled from respective regions of interest in different respective modalities, according to an embodiment of the invention.

FIGS. 2A-2C represent images of a patient according to respective modalities.

FIG. 2D represents a combined image assembled from respective regions of interest in different respective modalities, according to an embodiment of the invention.

FIG. 3 schematically shows an apparatus according to an embodiment of the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention provides systems and methods to visualize NM data from different modalities in a single image. The different modalities may include images from PET/CT, SPECT/CT, SPECT/MR or PET/MR scanners, using multiple tracers.

Tracer uptake values from the scans and acquired with multiple tracers are displayed in a meaningful way, allowing a user to get an overview of a patient in a single image dataset.

The present invention may be applied regardless of whether the different tracers were imaged separately or simultaneously.

The present provides organ-specific LUTs for multi-tracer studies. That is, for each segmented organ or structure (see T. Kohlberger, J. Zhang, M. Sofka, et al. “Automatic Multi-Organ Segmentation Using Learning-based Segmentation and Level Set Optimization”, MICCAI 2012, Springer LNCS.) a selected tracer can be visualized using an appropriate LUT and combined into one single image. In other words, different colour maps are spatially applied depending on a segmentation result.

FIGS. 1A-1D: One patient has been consecutively scanned with 3 different NM tracers. FIG. 1A indicates a fused visualization of anatomical information (body outline) superimposed by a colour-coded overlay (checked pattern) of tracer A. FIGS. 1B and 10 show similar fused visualizations using tracer B (vertical stripes) and tracer C (horizontal stripes), respectively. All overlays are combined into one multi-fused image, shown in FIG. 1D, by spatially applying different LUTs for respective tracers A-C utilizing anatomical segmentation results of lung and liver.

The example illustrated in FIGS. 1A-1D and will be used for explanation. FIGS. 1A-1C represent image data acquired from three respective NM studies where different tracers have been used.

The acquired data and all associated volumes are registered to each other, for example using a registration algorithm or the scanner alignment.

In this example, tracer A is used in the acquisition of the image of FIG. 1A. It has high sensitivity for detecting liver lesions. Tracer B, used in the acquisition of the image of FIG. 1B, is advantageous for imaging the lung. Tracer C, used in the acquisition of the image of FIG. 1C, can be considered as an “all-rounder”, and is reasonably effective at detecting lesions in most parts of the body.

A segmentation of the lungs and liver is made available from associated anatomical data. Accordingly, regions of the images which correspond to each segment of the data can be identified and isolated. In a method according to the present invention, a composite image, as shown in FIG. 1D, may be produced using identified segments from each of the images 1A-1C, each segment being chosen from the image which is expected to produce the best definition for that segment based on the tracer(s) used in the acquisition of that image.

In the present example, a liver LUT would be used that selects the liver image data acquired using tracer A (FIG. 1A) but hides liver image data acquired using tracers B & C (FIGS. 1B and 1C). A lung LUT would be used that selects the lung image data acquired using tracer B (FIG. 1B) but hides lung image data acquired using tracers A & C (FIGS. 1A and 1C). For the remaining part of the body, a LUT would be used that selects the image data acquired using tracer C (FIG. 1C) but hides image data acquired using tracers A & B (FIGS. 1A and 1B).

The result, illustrated in FIG. 1D, is an image generated by using an organ-specific LUT that spatially highlights different tracers for different organs. In the following description, such an image will be referred to as multi-fused image.

Note that the utilized LUTs can be chosen in different ways. In one example, the utilized LUTs can be designed such that uptake from different tracers can be easily visually differentiated, e.g., using different colour schemes for each tracer. In another example, the utilized LUTs can be adjusted to compensate for the naturally varying dynamic range of each tracer such that the resulting multi-fused image looks similar to a conventional single-tracer image.

As a consequence, for each organ or structure, the image data generated from the tracer with the optimal clinical sensitivity is shown. This allows a user to get a detailed overview of the patient's condition, and may assist the user in deciding whether to examine each individual image separately. Preferably, each individual image can be visualized in parallel to the multi-fused image. This may be achieved by providing a multi-image display on a monitor, similar to the combined representation of FIGS. 2A-2D.

FIGS. 2A-2D show a possible realization of the invention similar to the combination of the representations shown in FIGS. 1A-1D. FIG. 2D shows an axial slice as multi-fused image. FIGS. 1A-1C show separate images as derived from respective individual tracers A-C, made available for detailed examination. Preferably, that all four views are synchronized with each other such that the multi-fused image of FIG. 2D always represents a combination of the images of FIGS. 2A-2C.

The same applies to MIP rendering. The required different LUTs can be selected in such a way that the uptake of a particular tracer is either hidden or visualized in the combined MIP.

The present invention provides a method in which display properties of different image data sets are modified, based on anatomical location such that a combination of partial image data sets representing multiple tracers can be displayed in a single image. It does not directly provide highlighting of certain organs within an image.

In the case of multiple-tracers being combined in one study, so generating a single set of image data representing multiple tracers, the same concept can be utilized to compensate for the different uptake functions of the involved tracers in different organs/structures.

As an example, a tracer D may feature predominant uptake located inside the bones with a higher normal uptake than the other tracers. To improve the usefulness of, for example the MIP, the LUT may be adjusted for the bone regions in such a way that the average normal intensity is similar to the normal intensity in other structures/organs.

Manual adjustment of visualization parameters may be provided within the present invention.

Manual adjustment of window level: for conventional functional data displays, the user typically holds down a mouse button and moves the mouse left or right to adjust the bottom and up or down to adjust the top. This could be extended to this invention to consider the anatomical region in which the cursor was positioned prior to window level adjustment. For instance, following on from the above example, if the user depressed the mouse button whilst in the liver, only the window level for the liver (i.e., displaying tracer A) would be affected by the following movements of the mouse. Alternatively, if short cut keys are used to select a predefined window level, this change could be applied only to the anatomical region in which the mouse cursor, or cross-hair, or centre of field of view as appropriate, is located.

Manual adjustment of LUT: for conventional functional data displays, a user typically changes the LUT by either selecting a different LUT from a menu such as a right-click menu or using short cut keys. As in the case of adjusting the window level, changes could be applied only to the anatomical region in which the mouse cursor or cross-hair, or centre of field of view as appropriate is located. Beyond that, the user might want to toggle between different tracers for an organ of interest. For example, a short cut key may be utilized to toggle between different LUTs that visualize different tracers for a currently-selected anatomical region.

The visualization parameters on an organ and tracer basis can be stored to enhance the user's experience while toggling. Organ-specific workflows may be defined that store (default) settings for visualization parameters, such as windowing and zooming parameters, for single-tracer studies.

The above-described examples of the present invention do not conflict with the possibility of having global LUT/window combinations available for selection, either pre-defined or user-defined, or with adjusting visualization parameters such as windowing, globally as is conventional for single-tracer studies.

If the user wanted to create a clinical finding, for example defining a volume of interest VOI on the multi-tracer image such as FIG. 1D, the intensity values associated with that finding could be determined based on the functional image data being displayed in the region in which the finding was created. For example, if tracer A was dominant tracer displayed in the liver LUT and the user created a finding in the liver, it would be the intensity values from tracer A that would be displayed and reported for that finding. If desired by the user, corresponding values from the other tracers for this finding could also be displayed and reported.

The invention can be applied to any number of combinations of tracers for imaging systems such as PET/CT, SPECT/CT, SPECT/MR or MR/PET. The present invention may be applied for any combination of modalities, for example by utilizing colour-coded LUTs and/or 3D visualization techniques such as MIPs. An example may be the use of different MRI diffusion/perfusion protocols as functional data on top of an anatomical T1/T2/PD-weighted scan or quantitative CT results on top of conventional anatomical data.

The present invention requires the alignment of the involved volumes (via hardware or software solutions) and a segmentation of the organs or structures of interest. This alignment may be performed automatically or semi-automatically by appropriate known methods, or may be performed manually. Such organs or structures include, but are not limited to, major organs such as lungs, liver, kidneys, brain, bladder, prostate, spleen, etc. or structures such as bone, muscles, fat, etc.

The described LUTs are specific to each combination of organ/structure and tracer and can be either pre-defined or adjustable by the user. LUTs may be adapted to automatically adjust for differences in dynamic range of the different tracers in different organs.

That is, each LUT, specific to a certain combination of organ/structure and tracer can be automatically adjusted based on data such as average tracer uptake within a VOI within the organ to align the dynamic range so that the visual brightness of the displayed volume is consistent throughout the volume.

The present invention may include use of LUTs that combine uptake measurements from multiple tracers, as an alternative to defining respective LUTs to show uptake from a single functional image alone.

One embodiment of an aspect of the invention can provide apparatus for combining a plurality of image data sets into one multi-fused image for display to a user, comprising: a processor adapted to align all image data sets with a data set representing corresponding anatomical data; segment the image data in each data set into separate regions representing respective structures of interest by reference to a segmentation derived from anatomical data; for each region, select a corresponding segment of image data from a selected image data set; and combine the selected segments of the image data set to generate a multi-fused image of the regions, including the steps of applying spatially dependent look-up tables to the plurality of image data sets, thereby to determine whether each data set is hidden or displayed for each region; and a display device adapted to display the multi-fused image of the regions.

One embodiment of an aspect of the invention can provide a media device storing computer program code adapted, when loaded into or run on a computer, to cause the computer to become apparatus, or to carry out a method, according to any of the above embodiments.

Referring to FIG. 3, the above embodiments of the invention may be conveniently realized as a computer system suitably programmed with instructions for carrying out the steps of the methods according to the invention.

For example, a central processing unit 4 is able to receive data representative of medical scan data via a port 5 which could be a reader for portable data storage media (e.g. CD-ROM); a direct link with apparatus such as a medical scanner (not shown) or a connection to a network.

For example, in an embodiment, the processor performs such steps as

(a) aligning all image data sets with a data set representing corresponding anatomical data;

(b) segmenting the image data in each data set into separate regions representing respective structures of interest by reference to a segmentation derived from anatomical data;

(c) for each region, selecting a corresponding segment of image data from a selected image data set; and

(d) combining the selected segments of the image data set to generate the multi-fused image of the regions, including the steps of applying spatially dependent look-up tables to the plurality of image data sets, thereby to determine whether each data set is hidden or displayed for each region.

Software applications loaded on memory 6 are executed to process the image data in random access memory 7.

A Man-Machine interface 8 typically includes a keyboard/mouse/screen combination (which allows user input such as initiation of applications) and a screen on which the results of executing the applications are displayed.

Although modifications and changes may be suggested by those skilled in the art, it is the intention of the inventors to embody within the patent warranted hereon all changes and modifications as reasonably and properly come within the scope of their contribution to the art.

Claims

1. A method combining a plurality of image data sets into one multi-fused image for display to a user, comprising in a processor: (a) aligning all image data sets with a data set representing corresponding anatomical data;(b) segmenting the image data in each data set into separate regions representing respective structures of interest by reference to a segmentation derived from anatomical data;(c) for each region, selecting a corresponding segment of image data from a selected image data set; and(d) combining the selected segments of the image data set to generate the multi-fused image of the regions, by applying spatially dependent look-up tables to the plurality of image data sets, thereby to determine whether each data set is hidden or displayed for each region, and making the multi-fused image of the regions available in electronic form from said processor.

2. A method according to claim 1 wherein one or more of the image data sets is displayed alongside the multi-fused image.

3. A method according to claim 2 wherein the images are synchronised.

4. A method according to claim 1 wherein, for each selected segment, one or more display parameters are adjusted to optimise the displayed image segment for each region.

5. A method according to claim 1 wherein at least some of the structures correspond to organs of a patient's body.

6. A method according to claim 1 wherein segments of image data are differently coloured in the multi-fused image.

7. A method according to claim 1 wherein the image data sets each represent functional data derived from a different tracer or combination of tracers.

8. A method according to claim 1 wherein the look-up tables are automatically adjusted to align a dynamic range of each selected segment of image data to compensate for differences in uptake of respective tracers and/or combinations of tracers.

9. A method according to claim 1 wherein the application of the look-up tables promotes visual differentiation between the uptake of respective tracers and/or combinations of tracers.

10. A method according to claim 1 wherein the plurality of image data sets is acquired simultaneously.

11. Apparatus for identifying a region of interest in medical imaging data of a subject, comprising: a processor adapted to align all image data sets with a data set representing corresponding anatomical data; segment the image data in each data set into separate regions representing respective structures of interest by reference to a segmentation derived from anatomical data; for each region, select a corresponding segment of image data from a selected image data set; and combine the selected segments of the image data set to generate a multi-fused image of the regions, including the steps of applying spatially dependent look-up tables to the plurality of image data sets, thereby to determine whether each data set is hidden or displayed for each region; and a display device adapted to display the multi-fused image of the regions.

12. A non-transitory, computer-readable data storage medium encoded with programming instructions, said storage medium being loaded into a processor and said programming instructions causing said processor to: (a) align all image data sets with a data set representing corresponding anatomical data;(b) segment the image data in each data set into separate regions representing respective structures of interest by reference to a segmentation derived from anatomical data;(c) for each region, select a corresponding segment of image data from a selected image data set; and(d) combine the selected segments of the image data set to generate the multi-fused image of the regions, by apply spatially dependent look-up tables to the plurality of image data sets, thereby to determine whether each data set is hidden or displayed for each region, and make the multi-fused image of the regions available in electronic form from said processor.