Apparatus and method for visualizing digital breast tomosynthesis and other volumetric images

Description

FIELD OF INVENTION

The invention pertains to novel ways of viewing volumetric images used for medical diagnosis.

BACKGROUND

Volumetric images play an increasingly important role in medical diagnosis including cancer treatments such as site directed chemotherapy and radiology. Volumetric images are being generated by a multitude of different devices, including Magnetic Resonance Imaging (MRI) scanners, see for example Nuclear magnetic resonance imaging apparatus, U.S. Pat. No. 4,534,358, or Computed Tomography (CT) scanners, see for example Patients' support installation for a tomographic X-ray apparatus, U.S. Pat. No. 3,974,388, or certain C-Arm devices, see for example C-Arm computerized tomography system, U.S. Patent Application Publication No. 2010/0284601.

A certain class of these modalities, such as the CT scanner computes the volumetric images from a series of 2D projections from different angles, see for example (i) Methods and Apparatus for Reconstruction of 3D Image Volumes From Projection Images, U.S. Pat. No. 7,876,944; (ii) Method of Reconstructing Computer Tomography (CT) Volumes Suitable for Execution on Commodity Central Processing Units (CPUS) and Graphics Processors, and Apparatus Operating in Accordance with those Methods, U.S. Pat. No. 7,778,392 and (iii) Method of Reconstructing Computer Tomography (CT) Volumes Suitable for Execution on Commodity Central Processing Units (CPUS) and Graphics Processors, and Apparatus Operating in Accordance with those Methods, U.S. Pat. No. 8,107,592, which references (i)-(iii) are herein expressly incorporated by reference in their entireties.

A recent advance in the field is the development of a Digital Breast Tomo synthesis (DBT) scanner which generates volumetric mammography images, see for example Integrated multi-mode mammography/tomosynthesis x-ray system and method, U.S. Pat. No. 7,869,563, which is herein expressly incorporated by reference in its entirety. Similar to CT or C-Arm devices, the DBT devices acquire a number of 2D X-Ray images, or 2D projections, from different angles. From these projections a volumetric image is computed.

SUMMARY OF THE INVENTION

In an embodiment of the present invention, a method for displaying volumetric images comprises computing a projection image using a viewing direction, displaying the projection image and then varying the projection image by varying the viewing direction. In an embodiment of the present invention, the viewing direction can be varied based on a periodic continuous mathematical function. In an embodiment of the present invention, a graphics processing unit (GPU) can be used to compute the projection image and bricking can be used to accelerate the computation of the projection images. In another embodiment of the present invention, a sequence of projections covering one period can be rendered, cached and then played back one or more times, where the rendering is carried out on a server and the caching and play back is carried out on a client computer. A render server program is described in U.S. application Ser. No. 13/831,967, entitled “Multi-User Mult-GPU Render Server Apparatus and Methods”, which was filed Mar. 15, 2013 is herein expressly incorporated by reference in its entirety. A rule based render server program is described in “Method and System for Rule-Based Display of Sets of Images” which issued as U.S. Pat. No. 8,976,190 on Mar. 10, 2015, and is herein expressly incorporated by reference in its entirety. In an alternative embodiment of the present invention, the viewing direction can be varied based on user input. In a different embodiment of the present invention, a system that displays two or more volumetric images by computing a projection image of each of the volumetric images, using the same viewing direction v for each volumetric image, displaying each projection images, and varying the projection image by varying the viewing direction, where the varied viewing direction is changed in the same way for each of the projections. In an embodiment of the present invention, the volumetric images are computed from a number of 2D X-Ray images, or 2D projections, from different angles generated by a DBT device. In an embodiment of the present invention, volumetric mammography images are displayed. In an alternative embodiment of the present invention, volumetric images are computed from a number of 2D X-Ray images generated by angiography. In an embodiment of the present invention, the volumetric cerebral angiography images of the human brain are displayed. In another alternative embodiment of the present invention, volumetric images are computed from a confocal microscope using antibody staining. In an embodiment of the present invention, volumetric cell tissue generated by the confocal microscope is displayed.

BRIEF DESCRIPTION OF THE DRAWINGS

This invention is described with respect to specific embodiments thereof. Additional features can be appreciated from the Figures in which:

FIG. 1A shows the specimen imaged using an X-Ray source from two positions spanning an angular range;

FIG. 1B shows the specimen imaged using an X-Ray source and an X-Ray detector from a multitude of positions. The positions span a certain angular range that is defined by the physical constraints of the machine and the patient's position;

FIG. 2 illustrates the calculation of a projection P from the volumetric image I, where the projection is defined by the viewing direction v, which defines the Projection plane, according to an embodiment of the invention;

FIG. 3A shows a specimen with two areas of increased density, according to an embodiment of the invention;

FIG. 3B shows the two areas in FIG. 3A projected to the same spot in the projection Image, according to an embodiment of the invention;

FIG. 3C shows the two areas in FIG. 3A projected to different spots in the projection Image, according to an embodiment of the invention;

FIG. 4 shows how only a subset of the acquisition volume is covered by the specimen, while other areas (hatched) only contain background pixels, according to an embodiment of the invention;

FIG. 5 shows the volume subdivided into sub-volumes, according to an embodiment of the invention;

FIG. 6 illustrates the dynamic variation of the viewing direction v according to Equation 2, according to an embodiment of the invention;

FIG. 7A shows an artists impression of an image of a human breast computed from a number of images recreated from a specific angle where a micro calcification is occluded by denser breast tissue, according to an embodiment of the invention;

FIG. 7B shows an atrists impression of an image of a human breast taken from a different angle to that shown in FIG. 7A, where the micro calcification is visible and not occluded by the denser breast tissue, according to an embodiment of the invention;

FIG. 8A shows an artists impression of a screen dump of a video image at approximately the two (2) second time point, where the video shows a dynamic comparison of a human breast computed from a number of images recreated as the viewing direction is changed, where micro calcification occluded by denser breast tissue can be revealed, according to an embodiment of the invention;

FIG. 8B shows an artists impression of a screen dump of a video image at approximately the five (5) second time point, where the video shows a dynamic comparison of a human breast computed from a number of images recreated as the viewing direction is changed, where micro calcification occluded by denser breast tissue can be revealed, according to an embodiment of the invention;

FIG. 8C shows an artists impression of a screen dump of a video image at approximately the nine (9) second time point, where the video shows a dynamic comparison of a human breast computed from a number of images recreated as the viewing direction is changed, where micro calcification occluded by denser breast tissue can be revealed, according to an embodiment of the invention;

FIG. 8D shows an artists impression of a screen dump of a video image at approximately the twelve (12) second time point, where the video shows a dynamic comparison of a human breast computed from a number of images recreated as the viewing direction is changed, where micro calcification occluded by denser breast tissue can be revealed, according to an embodiment of the invention;

FIG. 9A shows the artists impression of a screen dump of a video image at at approximately the five (5) second time point shown in FIG. 8B, according to an embodiment of the invention;

FIG. 9B shows the artists impression of a screen dump of a video image at approximately nine (9) second time point shown in FIG. 8C, according to an embodiment of the invention;

FIG. 10A shows the image of a human breast represented in FIG. 7A, according to an embodiment of the invention;

FIG. 10B shows the image of the human breast taken from a different angle to that shown in FIG. 10A, represented in FIG. 7B, according to an embodiment of the invention;

FIG. 11A shows a screen dump from the mp3 video at approximately the two (2) second time point, represented in FIG. 8A, according to an embodiment of the invention;

FIG. 11B shows a screen dump from the mp3 video at approximately the five (5) second time point, represented in FIG. 8B, according to an embodiment of the invention;

FIG. 11C shows a screen dump from the mp3 video at approximately the nine (9) second time point, represented in FIG. 8C, according to an embodiment of the invention;

FIG. 11D shows a screen dump from the mp3 video at approximately the twelve (12) second time point, represented in FIG. 8D, according to an embodiment of the invention;

FIG. 12A shows the screen dump from the mp3 video at approximately the five (5) second time point, as represented in FIG. 9A, according to an embodiment of the invention; and

FIG. 12B shows the screen dump from the mp3 video at approximately the nine (9) second time point, as represented in FIG. 9B, according to an embodiment of the invention.

DESCRIPTION OF THE INVENTION

Definitions

The transitional term ‘comprising’ is synonymous with “including,” “containing,” or “characterized by,” is inclusive or open-ended and does not exclude additional, unrecited elements or method steps.

The transitional phrase ‘consisting of’ excludes any element, step, or ingredient not specified in the claim, but does not exclude additional components or steps that are unrelated to the invention such as impurities ordinarily associated with a composition.

The transitional phrase ‘consisting essentially of’ limits the scope of a claim to the specified materials or steps and those that do not materially affect the basic and novel characteristic(s) of the claimed invention.

The term ‘bandwidth’ and ‘send bandwidth’ refer to various bit-rate measures, representing the available or consumed data communication resources expressed in bits per second or multiples of it.

The phrase ‘adaptive bandwidth management’ means methods that continuously adjust the amount of data that is sent into a network per time in order to avoid or reduce network congestion and transfer delay.

The term ‘client-server’ refers to a computer system that selectively shares its resources with ‘clients’. A ‘client’ is a computer or computer program that initiates contact with a ‘client-server’ or ‘server’ in order to make use of the server resources. A client-server can be especially useful to undertake volume rendering tasks. Such a server can have one or more graphics processing units. Further, by sharing the server's computer resources, multiple clients can access and use the server resources at the same time. Because a computer does a limited amount of work at any moment, a time-sharing system must quickly prioritize its tasks to accommodate the clients. Clients and servers exchange messages in a request-response messaging pattern: The client sends a request, and the server returns one or multiple responses, synchronously or asynchronously.

The term ‘video’ means the display of three (3) or more 2-D projection images where there is a time delay between the first 2-D projection image and a second 2-D projection image and a time delay between the second 2-D projection image and a third 2-D projection image. A video may be displayed using a number of formats including avi, fly, H.262, H.263, H.264, m4v, mov, MPEG-1, MPEG-1 Part 2, MPEG-2, MPEG-4 Part 2, nsv, ogv, roq, vp6, vp8, vp9, webm, and wmv.

The phrase ‘host computer’ means a server or other processor with associated memory. In an embodiment of the invention, a host computer is enabled to provide measured 2-D projection images to a client.

The term ‘caching’ means storing in memory. A generated projection image from a volumetric image can be cached in one or both a client associated memory and a server associated memory, where the memory can be accessed rapidly by either the client processor or the server processor respectively.

The phrase ‘measured 2-D projection image’ means a two-dimensional (2-D) scan of biological tissue produced by forward-projection or back-projection of medical imaging equipment as described in U.S. Pat. No. 8,107,592 to A. Berman, and U.S. Pat. No. 7,876,944 to D Stalling et al.

The phrase ‘volumetric image’ refers to a three-dimensional (3-D) representation reconstructed from the data produced from a series of measured 2-D projection images or other 2-D representations of a tissue, an organ or an entity.

The term ‘reconstruction’ means generating a 3-D volumetric image based on a plurality of measured 2-D projection images. The phrase ‘reconstruction of a volumetric image’ means calculating a 3-D volumetric image based on a plurality of measured 2-D projection images.

The term ‘generated’ means constructing one or more generated 2-D projection images from a 3-D volumetric image. The phrase ‘generating an image’ or ‘generating a plurality of images’ means constructing one or more generated 2-D projection images from a 3-D volumetric image. In an embodiment of the invention, the one or more generated 2-D projection images can be generated at different viewing directions.

The phrase ‘viewing direction’ means the line constructed passing through a viewing position to an object. As the designated position changes, the viewing direction changes. As shown in FIG. 1A a first viewing direction 111 is generated by the line between position 110 and the object 105. A second viewing direction 116 is generated by the line between position 115 and the object 105. The angle (θ) between the first viewing direction 111 and the second viewing direction 112 increases from 0 to θ. The smallest viewing direction is when the angle=0. The largest viewing direction is when the angle=θ.

The phrase ‘equivalent viewing direction’ means the same viewing direction in the absence of physiologic changes in the tissue or an equivalent viewing direction when physiologic changes have occurred or a comparable tissue is utilized, where the equivalent viewing direction can compensate for changes in the tissue in the body with time and/or can compensate for the symmetry and asymmetry of different tissue in the body. The equivalent viewing direction can be used to ascertain the presence or absence of physiologic changes in the tissue with time, or when physiologic changes have occurred based on the inspection of a comparable tissue. The equivalent viewing direction can compensate for changes in the tissue in the body with time and/or can compensate for the symmetry and asymmetry of viewing projection images of different tissues in the body.

The term ‘identifies’ refers to a 3D or 2D image corresponding to a view that is displayed and/or compared with other views that reveals or more clearly elucidates a microcalcification or obstruction through one or more processes selected from the group consisting of: observation by the human eye, identification by a segmentation algorithm, identification by a bricking algorithm. The phrase ‘improves the visual clarity of identification’ means a process or technique that compares or changes one or more projection images to allow an obstruction including a micro calcification to be identified in the one or more projection images.

Receiving a Volumetric Image

A computed tomography (CT) scan can generate many 2-D images taken from different angles around a scanned object to produce cross-sectional (tomographic) images ('virtual slices') of the scanned object. Alternatively, positron emission tomography (PET), single photon emission computed tomography (SPECT), computer assisted tomography (CAT) scanners or tomosynthesis systems can produce ‘measured projection images’. These measured 2-D projection images can be used to reconstruct a ‘volumetric image’, where the virtual slices form a volumetric image or 3-D image of the scanned object. The phrase ‘volumetric image’ refers to a 3-D representation reconstructed from the data produced by forward-projecting or back-projecting medical imaging equipment. Measured projection images can be measured by medical technologists, and can be used to reconstruct a volumetric image and then the volumetric image can be received by a physician in order to diagnose a patient.

In an embodiment of the invention, using the reconstructed 3-D image it is possible to form a generated 2-D projection image, that is, a representation can be generated from a volumetric image by identifying a point source at a distinct focus and thereby a ‘projection direction’ through the volume to a plane at which the respective generated 2-D projection image can be formed.

Computing a Plurality of Projection Images

One or more generated 2-D projection images can be generated from a volumetric image. Computing a plurality of generated 2-D projection images of the volumetric image using a plurality of viewing directions between a first viewing direction and a second viewing direction can be used to produce generated 2-D projection images required by a physician but otherwise not revealed by a measured 2-D projection image. Alternatively, by generating a plurality of generated 2-D projection images, a dynamic view of the volumetric image can be generated, which allows for better diagnosis than a single or static measured 2-D projection image or a single or static generated 2-D projection image.

Comparing a First Projection Image a Second Projection Image

The phrase ‘time comparison’ means comparing a projection image obtained at a specific viewing direction with an earlier in time projection image of a tissue obtained at an equivalent viewing direction of the same tissue. In an embodiment of the invention, a time comparison compares one or more projection images of a right breast with one or more projection images of the same right breast measured at an earlier time point, where the projection images are generated at equivalent viewing directions. In an embodiment of the invention, a time comparison compares one or more measured 2-D projection images of a right breast with one or more generated 2-D projection images of the same right breast generated from a volumetric image reconstructed from a plurality of measured 2-D projection images from an earlier time point, where the projection images are generated at equivalent viewing directions. In an alternative embodiment of the invention, a time comparison compares one or more generated 2-D projection images of a right breast with one or more measured 2-D projection images of the same right breast measured at an earlier time point, where the projection images are generated at equivalent viewing directions.

The phrase ‘structural comparison’ means comparing a projection image obtained at a specific viewing direction with a projection image of a tissue obtained at an equivalent viewing direction of a different but comparable tissue. In an embodiment of the invention, a structural comparison compares one or more projection images of a right breast with one or more projection images of a left breast both viewed at equivalent viewing directions. In an embodiment of the invention, a structural comparison compares one or more generated 2-D projection images of a right breast with one or more generated 2-D projection images of a left breast, where each of the generated 2-D projection images are viewed at equivalent viewing directions. In an alternative embodiment of the invention, a structural comparison compares one or more measured 2-D projection images of a right breast with one or more generated 2-D projection images of a left breast, where each of the measured and generated 2-D projection images are viewed at equivalent viewing directions. In another embodiment of the invention, a structural comparison compares one or more generated 2-D projection images of a right breast with one or more measured 2-D projection images of a left breast, where each of the measured and generated 2-D projection images are viewed at equivalent viewing directions.

The phrase ‘dynamic comparison’ means comparing a series of projection images obtained at a variety of viewing directions. In an embodiment of the invention, a dynamic comparison compares one or more DBT projection images of a right breast that change in time as the viewing direction is scanned as a video. In an embodiment of the invention, the change in viewing direction can adjust for the type of tissue being scanned.

The phrase ‘visual comparison’ means time comparing, structurally comparing, and/or dynamically comparing one or more projection images with the naked eye.

The phrase ‘direct comparison’ means one or more of time comparing, structurally comparing, and dynamically comparing one or more projection images using a computer to analyze changes in the intensity density of a voxel matrix represented by the projection images. In an embodiment of the invention, one or more generated 2-D projection images are compared with one or more measured 2-D projection images using one or more of time comparing, structurally comparing, and dynamically comparing, wherein a computer is used to analyze changes in the intensity density of a voxel matrix represented by the one or more generated 2-D projection images and the one or more measured 2-D projection images.

A first viewing direction 111 corresponds with the line between position 110 and the object 105. A second viewing direction 116 corresponds with line between position 115 and the object 105 (see FIG. 1A). The increment 112 is the angle between the first viewing direction 111 and the second viewing direction 112 (see FIG. 1A). By selecting a first viewing direction, a first generated 2-D projection image can be formed. Similarly, selecting a second viewing direction allows a second generated 2-D projection image at the second viewing direction to be formed. In an embodiment of the invention a first generated 2-D projection image can be dynamically compared with one or more second generated 2-D projection images. In an alternative embodiment of the invention a measured 2-D projection image can be dynamically compared with one or more generated 2-D projection images. In an alternative embodiment of the invention, a first projection image can be time compared with a second projection image measured at an earlier time. In another embodiment of the invention, a generated 2-D projection image can be time compared with a measured 2-D projection image measured at an earlier time. In another embodiment of the invention, a first projection image can be structurally compared with a second projection image of a control tissue. In another embodiment of the invention, a generated 2-D projection image can be structurally compared with a measured 2-D projection image of a control tissue. In an embodiment of the invention, a density map for the first projection image is visually compared with a density map of the second projection image. In an embodiment of the invention, a density map for a generated 2-D projection image is visually compared with a density map of a measured 2-D projection image. In an alternative embodiment of the invention, a computer program is used to directly compare the density map for the first projection image with a density map of the second projection image. In another alternative embodiment of the invention, a computer program is used to directly compare the density map for a generated 2-D projection image with a density map of a measured 2-D projection image.

Volume Rendering

Volume rendering, or reconstructing a volume, includes a variety of standard visualization methods including volume rendering techniques (VRT), shaded volume rendering techniques (sVRT), maximum intensity projection (MIP), oblique slicing or multiplanar reformats (MPR), axial/sagittal and coronal slice display, and thick slices (also called slabs). Within the scope of the invention, other methods and apparatus of forward-projection and back-projection can be used for generating a series of measured 2-D projection images with which to reconstruct 3-D volumetric image representations, as described in U.S. Pat. No. 9,019,287, which is herein expressly incorporated by reference in its entirety.

In an embodiment of the invention, a computer chip, chip set, computer board and/or computer processor can be configured as a ‘graphics processing unit’ (GPU) to perform volume rendering and or to generate one or more reconstructed 2-D projection views from a volumetric image. In an embodiment of the invention, volume rendering includes initializing to arbitrary values the volume density distribution in a voxel matrix, iteratively estimating and comparing with a measured projection, and then correcting each pixel based on the comparison as described in U.S. Pat. No. 7,876,944.

Intensity Values

Image segmentation is an automated technique that facilitates distinguishing objects and other features in digital images. The technique can be used, for example, to simplify digitized images so that they can be more readily interpreted by computers (e.g., image analysis software) and/or by their users. An image can be made up of pixels containing a wide range of undifferentiated intensity values that although, possibly recognizable to the human eye as skeletal bones and digestive tract are largely uninterpretable by a computer. In an embodiment of the invention, a comparison between a first projection image with a second projection image that reveals an area of increased intensity values in the second projection image can indicate that the second viewing direction which generated the second projection image reveals an unobstructed projection image. In an alternative embodiment of the invention, a comparison between a generated 2-D projection image with a measured 2-D projection image that reveals an area of increased differentiated intensity values in the measured 2-D projection image can indicate that the viewing direction which formed the generated 2-D projection image reveals an unobstructed viewing direction. In an alternative embodiment of the invention, a comparison between a first projection image with a second projection image that reveals an area of increased differentiated intensity values in the second projection image can indicate that the second viewing direction which generated the second projection image reveals an increased clarity projection image, as described in U.S. Pat. No. 8,548,215, which is herein expressly incorporated by reference in its entirety. In an alternative embodiment of the invention, a comparison between a generated 2-D projection image with a measured 2-D projection image that reveals an area of increased differentiated intensity values in the measured 2-D projection image can indicate that the viewing direction which formed the generated 2-D projection image reveals an advantageous viewing direction.

Primary Study Versus Secondary Study

A primary study is a study carried out at a specified time point. A secondary study is a study carried out at a subsequent time point. In an embodiment of the invention, a computer chip, chip set, computer board and/or computer processor can be configured as a ‘digita data processor’ to perform volume rendering, to generate one or more projection views from a volume and or to compare two or more projection views. The digital data is generated by forward-projecting or back-projecting medical imaging equipment used to generate measured projection images or other 2-D representations. In an embodiment of the invention, a comparison between a generated 2-D projection image from a secondary study with a generated 2-D projection image from a primary study that reveals an area of increased differentiated intensity values can be used to assess the development or changes occuring over time. In an embodiment of the invention, a comparison between a generated 2-D projection image from a secondary study with a measured 2-D projection image from a primary study that reveals an area of increased differentiated intensity values in the measured 2-D projection image can indicate that the viewing direction which formed the generated 2-D projection image reveals an unobstructed viewing direction.

In the following description, various aspects of the present invention will be described. However, it will be apparent to those skilled in the art that the present invention may be practiced with only some or all aspects of the present invention. For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without the specific details. In other instances, well-known features are omitted or simplified in order not to obscure the present invention.

Parts of the description will be presented in data processing terms, such as data, selection, retrieval, generation, and so forth, consistent with the manner commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art. As is well understood by those skilled in the art, these quantities (data, selection, retrieval, generation) take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, and otherwise manipulated through electrical, optical, and/or biological components of a processor and its subsystems.

Various operations will be described as multiple discrete steps in turn, in a manner that is most helpful in understanding the present invention; however, the order of description should not be construed as to imply that these operations are necessarily order dependent.

Various embodiments will be illustrated in terms of exemplary classes and/or objects in an object-oriented programming paradigm. It will be apparent to one skilled in the art that the present invention can be practiced using any number of different classes/objects, not merely those included here for illustrative purposes. Furthermore, it will also be apparent that the present invention is not limited to any particular software programming language or programming paradigm.

Due to the physical constraints of the acquisition setup, the possible angular range of the acquisition is often limited. Typically the angular range 112 is less than 180° in digital breast tomosynthesis (DBT) (see FIG. 1A). For mathematical reasons, this results in volumetric images with a non-isotropic resolution. More precisely, the resolution in the plane perpendicular to the average projection direction is much higher, than the reconstructed resolution in the average direction of the X-Ray beam.

This aspect has to be taken into account when designing viewing methods for such images. Given the reconstructed volumetric image, in the following the direction of the lowest resolution will be referred to as the z-direction, or z. The vectors defining the average detector orientation, i.e. the plane with the highest resolution are denoted as x, and y. The x, y, and z directions are mutually perpendicular to each other.

In order to display a volumetric image on a standard computer screen, which is two dimensional, a transformation has to be applied in order to compute a 2-D representation of the volumetric image.

For DBT viewing, a slicing transformation can be used, where a single slice perpendicular to the z-direction is shown on the screen. Typically a user interface, such as a slider or text input field, allows the user to select which slice can be shown. In the following this will be referred to as ‘xy-slicing’ or ‘slicing’. While xy-slicing is an important viewing tool, it has some limitations. In particular it only takes into account a small subset of the information present in the volumetric data set.

The present invention overcomes the limitation of using only a small subset of the information by using a projection method to incorporate the entirety of the volumetric information. In an embodiment of the present invention, time is used as a third dimension to resolve ambiguities in a comprehensible and intuitive way.

From the volumetric image a projection can be computed. Let

I: R³→R be the volumetric image.

Let v e R³be a three dimensional vector defining a first viewing direction.

Let i_xand i_ybe two vectors spanning a projection plane perpendicular to v and perpendicular to each other.

Then a projection P(v,.) can be defined as follows:

P(v,.): R²→R

P(v,p)=max(I(r))|r e R³, where v dot i_x=p₁, v dot i_y=p₂and P(v,.) is a 2D image that can be displayed on a computer screen using standard methods.

Displaying P(v,.) as defined above provides the user with additional diagnostic information as it takes into account the whole data set. For example if there was a lesion in the examined specimen and the volumetric image was viewed using xy-slicing then that lesion would only be visible in a subset of the slices at or around the z-position of the lesion. If the wrong z-position was chosen, the lesion can be missed. Therefore the user would have to examine each slice to be certain there was no lesion present, or alternatively risk overlooking a lesion. In an unexpected result, viewing a dynamic comparison in the form of a video can allow the information to be quickly and efficiently compared.

FIG. 1A shows a specimen 105 imaged from two positions 110, 115 spanning an angular range 112 which generate viewing directions 111, 116 respectively. FIG. 1B shows the principle of DBT. In FIG. 1B, the specimen 105 (e.g. a human breast) can be imaged using an X-Ray source and an X-Ray detector from a multitude of positions that lie on the arc beginning at position 110 and ending at position 115 and which are detected at detector positions 120 and 125, respectively. The average acquisition direction is indicated by the dotted line 130. The positions span a certain angular range that is defined by the physical constraints of the machine and the patient's position. The z vector (z) 130 denotes the middle projection direction in that angular range. FIG. 2 illustrates the calculation of a projection P from the volumetric image I, 235. The projection is defined by the viewing direction v245, which defines the projection plane 240. In general v245 is not necessarily identical to the average acquisition direction z250. The two vectors i_x252 and i_y254 are the x-direction and y-direction of the projection image P, respectively. The vectors i_x252 and i_y254 are perpendicular to the viewing direction v245. The vector i_x252 is perpendicular to the vector i_y254, and can be chosen according to the users viewing preferences or automatically specified according to automated rules, as described in more detail in “Method and System for Rule-Based Display of Sets of Images” issued as U.S. Pat. No. 8,976,190. Digital Imaging and Communication in Medicine (DICOM) parameters for making rule based decisions include the time of generation of the measured projection images, the type of tissue measured and whether the tissue has an equivalent control that can be used as a control. For example for mediolateral acquisition directions, the y-axis will typically be chosen such that it aligns with the projection of the patient's head-foot axis.

Instead, when looking at the projection image P(v,.), an area of increased density, such as a lesion or calcification will appear as a brighter spot, irrespective of its z-position, making it possible to detect in many cases. FIG. 7A shows an atrists impression of an image of a human breast computed from a number of 2D X-Ray images procuced by a DBT device taken from a specific angle where a micro calcification is occluded by denser breast tissue. In FIG. 7A a region 774 is identified. FIG. 7B shows an atrists impression of an image of a human breast taken from a different angle to that shown in FIG. 7A. Comparison of FIG. 7A and FIG. 7B show a micro calcification is visible in FIG. 7B when the tissue is not occluded by denser breast tissue. Unexpectedly, in FIG. 7B the region 774 which was identified in FIG. 7A shows a micro calcification is visible and not occluded by the denser breast tissue.

FIGS. 8A-8D show an artists impression of four (4) images which make up time points in a mp3 video of a dynamic comparison of DBT of a right breast while the viewing direction changes, according to an embodiment of the invention. The mp3 video used to generate FIGS. 8A-8D had a duration of approximately 13 seconds. Unexpectedly, the mp3 video is an excellent means of inspecting DBTs to identify micro calcifications. FIG. 8A shows the artists impression of a screen dump from the mp3 video at approximately the two (2) second time point. FIG. 8B shows the artists impression of a screen dump from the mp3 video at approximately the five (5) second time point. FIG. 8C shows the artists impression of a screen dump from the mp3 video at approximately the nine (9) second time point. FIG. 8D shows the artists impression of a screen dump from the mp3 video at approximately the twelve (12) second time point. The dynamic comparison illustrates the differences between the intensity of the voxel matrix from which the projection image is calculated. Unexpectedly, when viewing the video a spot becomes apparent, which is shown in FIG. 8B and FIG. 8C but is not present in FIG. 8A or FIG. 8D. FIG. 9A shows an enlarged version of FIG. 8B, the artists impression of the screen dump from the mp3 video at approximately the five (5) second time point where the region 774 is identified. FIG. 9B shows an enlarged version of FIG. 8C, the artists impression of the screen dump from the mp3 video at approximately the nine (9) second time point where the region 774 is again identified. The spot seen in FIG. 8B (FIG. 9A) and FIG. 8C (FIG. 9B) reduces in intensity between the observation in FIG. 8B (FIG. 9A) and the observation in FIG. 8C (FIG. 9B). Unexpectedly, the emergence and dimunition of a relatively bright spot in the same position when viewing a video, can also be used to confirm a microcalcification rather than an artifact of the imaging system. Thus, based on the mp3 video a microcalcification 774 jumps to the viewer's attention by way of the nature of the dynamic comparison, as shown in the difference between FIG. 8A where no microcalcification is present and FIG. 8B (FIG. 9A) where the microcalcification, 774 is present. Viewing the mp3 video improves the visual clarity of identification of a micro calcification.

FIG. 10A shows the image of a human breast generated from a volumetric image reconstructed from a number of 2D X-Ray images produced by a DBT device, where the generated 2-D projection image was formed at a specific angle, where a micro calcification is occluded by denser breast tissue, as represented in FIG. 7A. FIG. 10B shows the generated 2-D projection image of the human breast formed from a different angle to that shown in FIG. 10A, where the micro calcification is visible and not occluded by the denser breast tissue, as represented in FIG. 7B. FIGS. 11A-11D show four (4) images which make up time points in the mp3 video of the dynamic comparison of the DBT of the right breast. FIG. 11A shows a screen dump from the mp3 video at a two (2) second time point, as represented in FIG. 8A. FIG. 11B shows a screen dump from the mp3 video at the five (5) second time point, as represented in FIG. 8B. FIG. 11C shows a screen dump from the mp3 video at the nine (9) second time point, as represented in FIG. 8C. FIG. 11D shows a screen dump from the mp3 video at the twelve (12) second time point, as represented in FIG. 8D. FIG. 12A shows an enlarged version of FIG. 11B, the screen dump from the mp3 video at the five (5) second time point where the region 774 is identified, as represented in FIG. 9A. FIG. 12B shows an enlarged version of FIG. 11C, the screen dump from the mp3 video at the nine (9) second time point where the region 774 is identified, as represented in FIG. 9B.

A draw-back of any projection method, is that there can be an occlusion or overlay effect. In the case of a maximum intensity projection as defined above, consider the case where two (2) separate areas of increased density are at different z positions on approximately the same viewing ray v. FIG. 3A shows a specimen 356 with two separate areas 353 and 355 of increased density. In the projection they will appear as one, potentially larger spot. That is, for one viewing direction (v1) 358 the two separate areas 353 and 355 can be projected to the same spot in the projection image (projection 1) shown in FIG. 3B.

In an embodiment of the present invention, this ambiguity can be resolved by making the projection dynamic. Instead of choosing a fixed viewing direction v, a dynamic viewing direction can be used. Using an alternative viewing direction (v2) 360, the two separate areas 353 and 355 project to different spots in the projection image, making it obvious that there are two areas of interest. FIG. 3C shows for the second viewing direction (v2) 360 the two separate areas 353 and 355 can be projected to different spots in the generated 2-D projection image (projection 2).

In an alternative embodiment of the invention, different dynamic functions can be used to generate dynamic projection viewing directions. Given the non-isotropy of the input data mentioned above, the most useful dynamic functions are continuous periodic functions around the z direction. Two non-limiting examples of dynamic functions include:

v(t)=normalize(z+A sin(ωt)x) Equation 1
v(t)=normalize(z+A sin(ωt)x+A cos(ωt)y) Equation 2

where normalize(v)=v/|v|; t: time; ω=2 π f; f: frequency of the dynamic movement and A: Amplitude of the dynamic movement, e.g. A=0.05.

In other embodiments of the invention, different alternative dynamic functions can be used to generate dynamic projection viewing directions. In an embodiment of the invention, a linear function can be used in which the angle can be changed linearly. In an alternative embodiment of the invention, a z direction can be chosen and either the x or the y direction can be incremented. FIG. 6 illustrates the dynamic variation of the viewing direction v according to Equation 2. The viewing direction at two different points v_t1666 and v_t2668 at time t₁and t₂is shown, as well as the corresponding projection planes projection plane (t₁) 670 and projection plane (t₂) 672. Over time the viewing direction v can be varied around the main acquisition direction z130.

In another embodiment of the invention, the viewing direction can be determined by the user. In another alternative embodiment of the invention, the viewing direction can be determined by the user with an appropriate input device, such as a mouse. In an embodiment of the present invention, let (m_x1, m_y1) be the position of the mouse (or appropriate input device) at a starting time t₁. The starting time can then be defined by a mouse click (or appropriate input device). In an alternative embodiment of the present invention, the starting time can be triggered by the user entering a certain window with the mouse (or appropriate input device), or other graphical or non graphical criteria.

Assuming the user is moving the mouse, let (m_x2, m_y2) be the position of the mouse (or appropriate input device) at time t₂. Let s_widthand s_heightbe the width and height of the screen.

Then v(t2)=normalize (z+2 A x(m_x2−m_x1)/s_width+2 A y(m_y2−m_y1)/s_height) can be the interactively controlled viewing direction at time t₂. A person of ordinary skill in the art will appreciate that alternative mappings from the mouse coordinates to viewing directions can be used. In various embodiments of the present invention, alternative input methods or devices can be used including, a slider, a trackball, a head tracking device or an eye tracking device.

The above projection is a maximum intensity projection.

In various other embodiments of the present invention, other projection functions can be used, including emission absorption models or minimum intensity projections. The above projection is equivalent to an orthographic projection, where a 3-D object is represented in two dimensions through parallel projection, where all the projection lines are orthogonal to the projection plane. A person of ordinary skill in the art will appreciate that alternative projections including perspective projections can be used.

In an embodiment of the invention, an optimal viewing direction can be selected by comparing the resulting projection images at a plurality of viewing directions. In an embodiment of the invention, the criterion used for determining the optimal viewing direction can be a viewing direction that identifies an unobserved obstruction. In an alternative embodiment of the invention, the criterion used for determining the optimal viewing direction can be a viewing direction that improves the visual clarity of an initial projection image. In another alternative embodiment of the invention, the criterion used for determining the optimal viewing direction can be a viewing direction that improves the visual clarity of an improved projection image compared with an initial projected image. In a different embodiment of the invention, the criterion used for determining the optimal viewing direction can be a viewing direction that improves the visual clarity of identification of an obstruction. In another different embodiment of the invention, the criterion used for determining the optimal viewing direction can be a viewing direction that identifies an obstruction using direct comparison. In another embodiment of the invention, the criterion used for determining the optimal viewing direction can be a viewing direction that improves the direct comparison clarity of an initial projection image. In an embodiment of the invention, the criterion used for determining the optimal viewing direction can be a viewing direction that improves the direct comparison clarity of an improved projection image compared with an initial projected image. In another embodiment of the invention, the criterion used for determining the optimal viewing direction can be a viewing direction that improves the direct comparison clarity of identification of an obstruction.

Volumetric images in DBT are quite large data sets, as the xy-resolution is an order of magnitude larger than for example a standard CT scan. In order to render such large images at interactive speeds graphics processing units (GPU) can be utilized, see for example Methods and Apparatus for Visualizing Three-Dimensional and Higher-Dimensional Image Data Sets, U.S. Pat. No. 8,189,002, which is herein expressly incorporated by reference in its entirety. In typical DBT images, only a subset of the voxels of the volumetric image contain tissue, while other voxels are background pixels that are irrelevant for the diagnosis. In an embodiment of the present invention, by using a threshold segmentation these background pixels can be identified. FIG. 4 shows how only a subset of the acquisition volume is covered by the specimen 462, while other areas (hatched) 464 only contain background pixels. These background pixels can be identified using threshold segmentation. FIG. 5 shows the volume can be subdivided into sub-volumes. In one embodiment of the invention, an octree decomposition scheme can be used for this subdivision. In another embodiment of the invention, a binary space partitioning (BSP) scheme can be used for the subdivision. A person with ordinary skills in the art will appreciate that other subdivision schemes can be used. Sub-volumes that contain only background voxels (shown as hatched in FIG. 4 and FIG. 5) 464 can be skipped during the rendering process. Sub-volumes 463 that contain both, background voxels and tissue voxels can be further subdivided until a configured minimum size containing specimen 462 or background 464 can be reached. In an embodiment of the invention, bricking can be used to display only those sub-volumes that are not background-only. The technique of bricking for GPU based rendering is described in U.S. Pat. No. 8,189,002.

In an embodiment of the present invention, for the effective use of dynamic projection images, a sufficiently high frame rate is required in order to allow for a smooth rendering that appears natural to the user. This can be achieved in many cases by using GPU hardware combined with the bricking technique.

In an alternative embodiment of the present invention, a periodic dynamic viewing direction function can be used, and a sequence of projections covering one full period (1/f) can be pre-rendered, and then be played back in a loop. In case of a client server visualization system, the pre-rendered images can be computed on the server side and cached on the client side thereby making optimal use of the bandwidth and allowing for smooth playback even on slow networks.

In radiological diagnostics, comparison to prior images is relevant to detect change, e.g. tumor growth. In an embodiment of the present invention, a projection of a current image and a projection of a corresponding prior image can be displayed side-by-side and used to determine the change in tumor characteristics. The comparison can include the user visually comparing with the naked eye. The comparison can also be undertaken by a direct comparison program where equivalent viewing directions are used for the direct comparison. In various embodiments of the present invention, the user can choose the same dynamic viewing direction function for both, the current and the prior image, thereby allowing for direct comparison.

Another aspect of the invention is to combine the projection display of the volumetric image with conventional 2D mammograms or other X-Ray or radiological images, by dividing the available computer screens into virtual view ports and using one or more of the virtual view ports to display the one or more projection images, and one or more of the virtual viewports to display the other radiological images.

A method for displaying one or more optimal projection images generated from a volumetric image comprising the steps of receiving the volumetric image, computing a plurality of projection images of the volumetric image using a plurality of viewing directions, where at least an initial projection image of the plurality of projection images is computed using a first viewing direction, where a second viewing direction of the plurality of viewing directions is not equal to the first viewing direction, one or more of time comparing, structurally comparing and dynamically comparing the plurality of projection images to determine one or more optimal projection images and displaying the optimal projection images.

A method for identifying one or more optimal projection images generated from a volumetric image comprising the steps of receiving the volumetric image, computing a plurality of projection images of the volumetric image using a plurality of viewing directions, where at least an initial projection image of the plurality of projection images is computed using a first viewing direction, where a second viewing direction of the plurality of viewing directions is not equal to the first viewing direction, and one or more of time comparing, structurally comparing and dynamically comparing the plurality of projection images to determine one or more optimal projection images, where the optimal viewing direction is selected from the group consisting of a viewing direction that identifies an unobserved obstruction, a viewing direction that improves the visual clarity of the first projection image, a viewing direction that improves the visual clarity of the second projection image, a viewing direction that improves the visual clarity of identification of an obstruction, a viewing direction that identifies an obstruction using direct comparison, a viewing direction that increases the differentiated intensity values of the first projection image, a viewing direction that increases the differentiated intensity values of the second projection image, a viewing direction that improves the direct comparison clarity of the first projection image, a viewing direction that improves the direct comparison clarity of the second projection image, and a viewing direction that improves the direct comparison clarity of identification of an obstruction.

A method to determine one or more optimal projection images from a volumetric image comprising the steps of receiving a volumetric image, computing a plurality of projection images based on the volumetric image using a plurality of viewing directions, one or more of time comparing, structurally comparing and dynamically comparing the plurality of projection images to determine one or more optimal viewing directions, and correlating the optimal viewing directions with one or more projection images of the plurality of projection images to determine one or more optimal projection images, where the optimal viewing direction is selected from the group consisting of a viewing direction that identifies an unobserved obstruction, a viewing direction that improves the visual clarity of the first projection image, a viewing direction that improves the visual clarity of the second projection image, a viewing direction that improves the visual clarity of identification of an obstruction, a viewing direction that identifies an obstruction using direct comparison, a viewing direction that improves the direct comparison clarity of the first projection image, a viewing direction that improves the direct comparison clarity of the second projection image, and a viewing direction that improves the direct comparison clarity of identification of an obstruction.

A method to determine one or more optimal projection images from a volumetric image comprising the steps of receiving a volumetric image, computing a plurality of projection images based on the volumetric image using a plurality of viewing directions, one or more of time comparing, structurally comparing and dynamically comparing the plurality of projection images to determine one or more optimal viewing directions, and correlating the optimal viewing directions with one or more projection images of the plurality of projection images to determine one or more optimal projection images, where at least one of the plurality of viewing directions is determined using a periodic continuous mathematical function, where the plurality of projection images is a sequence of projections images spanning one period of the periodic continuous mathematical function, further comprising rendering and caching a sequence of projection images, where the rendered and cached sequence of projections are played back one or more times.

A method to determine one or more optimal projection images from a volumetric image comprising the steps of receiving a volumetric image, computing a plurality of projection images based on the volumetric image using a plurality of viewing directions, one or more of time comparing, structurally comparing and dynamically comparing the plurality of projection images to determine one or more optimal viewing directions, and correlating the optimal viewing directions with one or more projection images of the plurality of projection images to determine one or more optimal projection images, where at least one of the plurality of viewing directions is determined using a periodic continuous mathematical function, where the plurality of projection images is a sequence of projections images spanning one period of the periodic continuous mathematical function, further comprising rendering and caching a sequence of projection images, where the rendering is carried out on a server and one or both the caching and play back is carried out on a client computer.

A system that displays a first projection image and a second projection image of a volumetric image comprising computing a first projection image of the volumetric image using a first viewing direction, computing a second projection image of the volumetric image using a second viewing direction, where the first viewing direction is not equal to the second viewing direction, and displaying the first projection image and the second projection image, further comprising computing a third projection image using a third viewing direction, and displaying one or both the first projection image and the second projection image with the third projection image.

A system that compares a first projection image and a second projection image of a volumetric image comprising computing a first projection image of the volumetric image using a first viewing direction, computing a second projection image of the volumetric image using a second viewing direction, and one or more of time comparing, structurally comparing and dynamically comparing the first projection image and the second projection image, further comprising computing a third projection image using a third viewing direction, and comparing one or both the first projection image and the second projection image with the third projection image.

A method for identifying an object in a projection image comprising the steps of receiving a three dimensional volumetric image of a tissue, computing a plurality of projection images of the three dimensional volumetric image of the tissue using a plurality of viewing directions, one or more of time comparing, structurally comparing and dynamically comparing the plurality of projection images, and identifying an object that is present in a projection image selected from the plurality of projection images that is not present in one or more of the one or more projection images selected from the plurality of projection images, where at least one of the plurality of viewing directions is determined using a periodic continuous mathematical function, where the plurality of projection images is a sequence of projection images spanning one period of the periodic continuous mathematical function.

A method for identifying an object in a projection image comprising the steps of receiving a three dimensional volumetric image of a tissue, computing a plurality of projection images of the three dimensional volumetric image of the tissue using a plurality of viewing directions, one or more of time comparing, structurally comparing and dynamically comparing the plurality of projection images, and identifying an object that is present in a projection image selected from the plurality of projection images that is not present in one or more of the one or more projection images selected from the plurality of projection images, where at least one of the plurality of viewing directions is determined using a periodic continuous mathematical function, where a graphics processing unit is used to compute one or more of the plurality of projection images.

A method for identifying an object in a projection image comprising the steps of receiving a three dimensional volumetric image of a tissue, computing a plurality of projection images of the three dimensional volumetric image of the tissue using a plurality of viewing directions, one or more of time comparing, structurally comparing and dynamically comparing the plurality of projection images, and identifying an object that is present in a projection image selected from the plurality of projection images that is not present in one or more of the one or more projection images selected from the plurality of projection images, where at least one of the plurality of viewing directions is determined using a periodic continuous mathematical function, where a graphics processing unit is used to compute one or more of the plurality of projection images, where bricking is used to accelerate computation of one or more of the plurality of projection images.

A method for identifying an optimal projection image comprising the steps of receiving a volumetric image, computing the plurality of projection images based on the volumetric image using a plurality of viewing directions, and comparing the plurality of projection images to determine an optimal viewing direction corresponding to an optimal projection image, where the optimal viewing direction is selected from the group consisting of a viewing direction that identifies an unobserved obstruction, a viewing direction that improves the visual clarity of the first projection image, a viewing direction that improves the visual clarity of the second projection image, a viewing direction that improves the visual clarity of identification of an obstruction, a viewing direction that identifies an obstruction using direct comparison, a viewing direction that improves the direct comparison clarity of the first projection image, a viewing direction that improves the direct comparison clarity of the second projection image, and a viewing direction that improves the direct comparison clarity of identification of an obstruction.

A method for displaying a plurality of projection images comprising the steps of receiving a volumetric image, computing the plurality of projection images based on the volumetric image using a plurality of viewing directions and displaying the plurality of projection images.

A method for comparing a first projection image and a second projection image comprising the steps of receiving a volumetric image, computing the first projection image based on the volumetric image using a first viewing direction, computing the second projection image based on the volumetric image using a second viewing direction, where the first viewing direction is not equal to the second viewing direction and one or more of time comparing, structurally comparing and dynamically comparing the first projection image and the second projection image, further comprising one or more steps selected from the group consisting of identifying visually an obstruction, improving the visual clarity of the first projection image, improving the visual clarity of the second projection image, improving the visual clarity of identification of an obstruction, identifying an obstruction using direct comparison, improving the direct comparison clarity of the first projection image, improving the direct comparison clarity of the second projection image, and improving the direct comparison clarity of identification of an obstruction.

A method for displaying one or more unobstructed projection images comprising the steps of receiving a volumetric image, computing a plurality of projection images based on the volumetric image using a plurality of viewing directions, where at least a first projection image of the plurality of projection images is computed using a first viewing direction and at least a second projection image of the plurality of projection images is computed using a second viewing direction, where the first viewing direction is not equal to the second viewing direction, one or more of time comparing, structurally comparing and dynamically comparing the first projection image and the second projection image to determine if one or both of the first projection image and the second projection image are unobstructed, and displaying based on the comparison one or both the first projection image and the second projection image.

A method for displaying an unobstructed projection image of a breast comprising the steps of receiving a volumetric image of the breast, computing a first projection image of the breast based on the volumetric image using a first viewing direction and a second projection image of the breast based on the volumetric image using a second viewing direction, where the first viewing direction is not equal to the second viewing direction, one or more of time comparing, structurally comparing and dynamically comparing the second projection image of the breast with the first projection image of the breast to determine if one or both the first projection image of the breast and second projection image of the breast is unobstructed, and based on the comparison displaying one or both the first projection image of the breast and second projection image of the breast.

A system for displaying unobstructed breast projection images comprising receiving a plurality of volumetric images of a breast, where a first volumetric image of the plurality of projection images is measured at a first time and a second volumetric image of the plurality of projection images is measured at a second time, where the first time differs from the second time by a time interval, computing a first projection image from the first volumetric image measured at the first time using a first viewing direction, computing one or more projection images from the first volumetric image measured at the first time using one or more viewing directions, one or more of time comparing, structurally comparing and dynamically comparing the first projection image and the one or more projection images to determine an unobstructed viewing direction, where a second projection image corresponds with the one or more projection images at the unobstructed viewing direction, computing a third projection image from the second volumetric image measured at the second time using the unobstructed viewing direction, and displaying the second projection image and the third projection image.

A method for identifying additional lesions in a tissue comprising the steps of computing a plurality of projection images of the tissue using a plurality of viewing directions, where a first projection image is computed using a first viewing direction and a second projection image is computed using a second viewing direction, displaying the first projection image and the second projection image, one or more of time comparing, structurally comparing and dynamically comparing the first projection image and the second projection image, visually identifying an intense spot that separates in the second projection image from the first projection image.

A system that displays a first projection image and a second projection image of a volumetric image comprising a processor responsive to a command to select a volumetric image one or more digital data processors capable of carrying out the steps including, computing a first projection image of the volumetric image using a first viewing direction, computing a second projection image of the volumetric image using a second viewing direction, and graphics resources for displaying the first projection image and the second projection image.

A system that compares a first projection image and a second projection image of a volumetric image comprising a processor responsive to a command to select a volumetric image, one or more digital data processors capable of carrying out the steps including computing a first projection image of the volumetric image using a first viewing direction, computing a second projection image of the volumetric image using a second viewing direction, and graphics resources for comparing the first projection image and the second projection image.

A method of visualizing a dynamic comparison of a volumetric image comprising the steps of receiving the volumetric image, computing a plurality of projection images of the volumetric image using a plurality of viewing directions between a smallest viewing direction and a largest viewing direction, and displaying a video showing the plurality of projection images, where the viewing direction changes with time.

The term ‘brick’ or ‘bricking’ means partitioning a 3D image or a portion of the 3D image. Bricking is an iterative process involving determining the intensity of pixels in the 2D image based on the rule that all points in the 3D image data that are required for evaluating the intensities of the sample points along a ray passing through a brick are located within that brick. That is in an imaging apparatus having a CPU and a GPU with a plurality of programmable vertex shaders coupled to a plurality of programmable pixel shaders, the CPU partitions the 3D image into a plurality ‘bricks’ based on the vertex shaders and pixel shaders determining the intensities of one or more pixels in the 2D image as an iterative function of intensities of sample points in one or more bricks in the 3D image through which viewing rays associated with those pixels are passed, and where any two adjacent bricks preferably have a sufficient overlap such that all points in the 3D image data that are required for evaluating the intensities of the sample points along a ray passing through a brick are located within that brick.

The term ‘view’ or ‘viewing’ means a display of a 3D or 2D image.

The phrases ‘viewing position’ or ‘viewing ray’ refer to a display of a 3D or 2D image as observed from the viewing position or along a line defined by the viewing ray.

The term ‘microcalcification’ refers to small deposits of calcium typically seen in a breast mammogram which depending on shape, number, pattern and/or relative position can be used as an early and/or presenting sign of breast cancer.

The term ‘obstruction’ means a filling defect or other ductal abnormality, such as ductal ectasia, fibrocystic changes or a ductal irregularity such as can be observed with ductography of the breast including galactography and ductogalactography.

The foregoing description of embodiments of the methods, systems, and components of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations will be apparent to one of ordinary skill in the relevant arts. For example, steps performed in the embodiments of the invention disclosed can be performed in alternate orders, certain steps can be omitted, and additional steps can be added. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications that are suited to the particular used contemplated. Other embodiments are possible and are covered by the invention. Such embodiments will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. The breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents. The invention is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to ‘an’ or ‘one’ embodiment in this disclosure are not necessarily to the same embodiment, and such references mean at least one.

Claims

1. A method comprising: (A) receiving at a server from a client computer, a plurality of measured 2-D projection images of an object, the server comprising a server digital data processor with access to one or more graphics processing units, andthe client computer comprising a client digital data processor with access to a graphics processing unit to display a viewport;(B) reconstructing a 3-D volumetric data set at the server of the object from the plurality of measured 2-D projection images;(C) computing at the server a first 2-D projection image from the 3-D volumetric data set based on a first viewing direction generated by a first line drawn between a first position and the object;(D) computing at the server a second 2-D projection image from the 3-D volumetric data set based on a second viewing direction generated by a second line drawn between a second position and the object, where a first angle is subtended between the first line and the second line;(E) computing at the server a third 2-D projection image from the 3-D volumetric data set based on a third viewing direction generated by a third line drawn between a third position and the object, where a second angle is subtended between the first line and the third line, where the second angle is greater than the first angle; and(F) sending from the server to the client computer the first 2-D projection image, the second 2-D projection image and the third 2-D projection image to be displayed as a video on the viewport.
2. The method of claim 1, where the server has a first memory, where one or more of the first 2-D projection image, the second 2-D projection image and the third 2-D projection image are stored in the first memory.
3. The method of claim 2, where the client computer has a second memory, where one or more of the first 2-D projection image, the second 2-D projection image and the third 2-D projection image are stored in the second memory.
4. The method of claim 1, where the video enhances one or more features selected from the group consisting of: i) a microcalcification;ii) an obstruction;iii) a microcalcification using direct comparison;iv) an obstruction using direct comparison;v) an artifact; andvi) an artifact using direct comparison.
5. The method of claim 1, where the plurality of measured 2-D projection images are Digital Breast Tomosynthesis images.
6. The method of claim 1, where two or more of the first viewing direction, the second viewing direction and the third viewing direction are selected according to a periodic continuous mathematical function.
7. The method of claim 1, where the first viewing direction, the second viewing direction and the third viewing direction comprise less than one period of a periodic continuous mathematical function.
8. A method comprising: providing a server including a graphics processing unit, where the server:(A) receives a 3-D volumetric data set of a tissue of a selected patient reconstructed from a plurality of 2-D projection images;(B) computes a first 2-D projection image from the 3-D volumetric data set based on a first viewing direction;(C) computes a second 2-D projection image from the 3-D volumetric data set based on a second viewing direction, where the second viewing direction is not the first viewing direction;(D) computes a third 2-D projection image from the 3-D volumetric data set based on a third viewing direction, where the third viewing direction is neither the first viewing direction nor the second viewing direction; and(E) sends to a client computer a video for display on a virtual viewport comprising the first 2-D projection image, the second 2-D projection image and the third 2-D projection image.
9. The method of claim 8, where the client computer includes a cache, where the video is stored in the cache.
10. The method of claim 8, further comprising the server sending a viewing instruction to the client computer to determine an order of displaying two or more of the first 2-D projection image, the second 2-D projection image and the third 2-D projection image.
11. The method of claim 8, where the video enhances one or more features selected from the group consisting of: i) a microcalcification;ii) an obstruction;iii) a microcalcification using direct comparison;iv) an obstruction using direct comparison;v) an artifact; andvi) an artifact using direct comparison.
12. The method of claim 8, where the plurality of 2-D projection images are Digital Breast Tomosynthesis images.
13. The method of claim 8, where two or more of the first viewing direction, the second viewing direction and the third viewing direction are selected according to a periodic continuous mathematical function.
14. A method comprising: providing a server including a graphics processing unit, where the server:(A) receives a plurality of measured 2-D projection images;(B) reconstructs a 3-D volumetric data set of a tissue of a selected patient from the plurality of measured 2-D projection images;(C) computes a first 2-D projection image from the 3-D volumetric data set at a first viewing direction;(D) computes a second 2-D projection image from the 3-D volumetric data set based on a second viewing direction, where the second viewing direction is not the first viewing direction; and(E) computes a third 2-D projection image from the 3-D volumetric data set based on a third viewing direction, where the third viewing direction is neither the first viewing direction nor the second viewing direction; and(F) sends to a client computer the first 2-D projection image, the second 2-D projection image and the third 2-D projection image; anddisplays the first 2-D projection image, the second 2-D projection image and the third 2-D projection image as a video on a virtual viewport.
15. The method of claim 14, where the client computer includes a cache, where one or both the first 2-D projection image and the second 2-D projection image are stored in the cache.
16. The method of claim 14, a comparison of the first 2-D projection image and the second 2-D projection image enhances one or more features selected from the group consisting of: i) a microcalcification;ii) an obstruction;iii) a microcalcification using direct comparison;iv) an obstruction using direct comparison;v) an artifact; andvi) an artifact using direct comparison.
17. The method of claim 14, where the tissue is a breast tissue.
18. The method of claim 14, where the first viewing direction and the second viewing direction are selected according to a periodic continuous mathematical function.
19. The method of claim 6, where the periodic continuous mathematical function (v(t)) is selected from the group consisting of normalize (z+A sin(ω t)x), normalize (z+A sin(ω t)x+A cos(ω t)y), and normalize (z+2 A x(mx2−mx1)/swidth+2 A y(my2−my1)seight), where the vectors defining the average detector orientation, i.e. the plane with the highest resolution are denoted as x and y, where the z vector (z) denotes the middle projection direction in an angular range, where x, y, and z directions are mutually perpendicular to each other, where A is the amplitude of the dynamic movement, where ω=2 π f, where f is the frequency of the dynamic movement, where swidth and sheight be the width and height of the screen, where mxn, myn are the x and y coordinate positions defined by an input device at time tn, where t is time and n is an integer 1, 2, 3.
20. The method of claim 13, where the periodic continuous mathematical function (v(t)) is selected from the group consisting of normalize (z+A sin(ω t)x), normalize (z+A sin(ω t)x+A cos(ω t)y), and normalize (z+2 A x(mx2−mx1)/swidth+2 A y(m2−my1)sheight), where the vectors defining the average detector orientation, i.e. the plane with the highest resolution are denoted as x and y, where the z vector (z) denotes the middle projection direction in an angular range, where x, y, and z directions are mutually perpendicular to each other, where A is the amplitude of the dynamic movement, where ω=2 π f, where f is the frequency of the dynamic movement, where swidth and sheight be the width and height of the screen, where mxn, myn are the x and y coordinate positions defined by an input device at time tn, where t is time and n is an integer 1, 2, 3.

PRIORITY CLAIM

The present application is a continuation of (1) U.S. patent application Ser. No. 15/988,542, entitled “APPARATUS AND METHOD FOR VISUALIZING DIGITAL BREAST TOMOSYNTHESIS AND OTHER VOLUMETRIC IMAGES” filed May 24, 2018, which claims priority to (2) U.S. patent application Ser. No. 15/218,972, entitled “APPARATUS AND METHOD FOR VISUALIZING DIGITAL BREAST TOMOSYNTHESIS AND OTHER VOLUMETRIC IMAGES” filed Jul. 25, 2016 and issued May 29, 2018 as U.S. Pat. No. 9,984,478, and which claims priority to (3) U.S. Provisional application No. 62/197,956 filed Jul. 28, 2015, the specification and drawings of each of (1)-(3) are herein expressly incorporated by reference in their entireties.

US Referenced Citations (364)

Number	Name	Date	Kind
2658310	Cook	Nov 1953	A
3431200	Davis	Mar 1969	A
3645040	Ort	Feb 1972	A
4137868	Pryor	Feb 1979	A
4235043	Harasawa et al.	Nov 1980	A
4258661	Margen	Mar 1981	A
4267038	Thompson	May 1981	A
4320594	Raymond	Mar 1982	A
4746795	Stewart et al.	May 1988	A
4905148	Crawford	Feb 1990	A
4910912	Lowrey, III	Mar 1990	A
4928250	Greenberg et al.	May 1990	A
4958460	Nielson et al.	Sep 1990	A
4984160	Saint Felix et al.	Jan 1991	A
5031117	Minor et al.	Jul 1991	A
5091960	Butler	Feb 1992	A
5121708	Nuttle	Jun 1992	A
5128864	Waggener et al.	Jul 1992	A
5218534	Trousset et al.	Jun 1993	A
5235510	Yamada	Aug 1993	A
5241471	Trousset et al.	Aug 1993	A
5253171	Hsiao et al.	Oct 1993	A
5274759	Yoshioka	Dec 1993	A
5280428	Wu et al.	Jan 1994	A
5287274	Saint Felix et al.	Feb 1994	A
5293313	Cecil	Mar 1994	A
5307264	Waggener et al.	Apr 1994	A
5355453	Row et al.	Oct 1994	A
5368033	Moshfeghi	Nov 1994	A
5375156	Kuo-Petravic et al.	Dec 1994	A
5412703	Goodenough et al.	May 1995	A
5412764	Tanaka	May 1995	A
5442672	Bjorkholm et al.	Aug 1995	A
5452416	Hilton	Sep 1995	A
5488700	Glassner	Jan 1996	A
5560360	Filler	Oct 1996	A
5594842	Kaufman et al.	Jan 1997	A
5602892	Llacer	Feb 1997	A
5633951	Moshfeghi	May 1997	A
5633999	Clowes et al.	May 1997	A
5640436	Kawai et al.	Jun 1997	A
5671265	Andress	Sep 1997	A
5744802	Muehllehner et al.	Apr 1998	A
5774519	Lindstrom et al.	Jun 1998	A
5790787	Scott et al.	Aug 1998	A
5793374	Guenter et al.	Aug 1998	A
5793879	Benn et al.	Aug 1998	A
5813988	Alfano et al.	Sep 1998	A
5821541	Tumer	Oct 1998	A
5825842	Taguchi	Oct 1998	A
5838756	Taguchi et al.	Nov 1998	A
5841140	Mc Croskey et al.	Nov 1998	A
5909476	Cheng et al.	Jun 1999	A
5930384	Guillemaud et al.	Jul 1999	A
5931789	Alfano et al.	Aug 1999	A
5950203	Stakuis	Sep 1999	A
5960056	Lai	Sep 1999	A
5963612	Navab	Oct 1999	A
5963613	Navab	Oct 1999	A
5963658	Klibanov et al.	Oct 1999	A
6002739	Heumann	Dec 1999	A
6018562	Willson	Jan 2000	A
6032264	Beffa et al.	Feb 2000	A
6044132	Navab	Mar 2000	A
6049390	Notredame	Apr 2000	A
6049582	Navab	Apr 2000	A
6072177	Mccroskey et al.	Jun 2000	A
6088423	Krug et al.	Jul 2000	A
6091422	Ouaknine et al.	Jul 2000	A
6104827	Benn et al.	Aug 2000	A
6105029	Maddalozzo, Jr. et al.	Aug 2000	A
6108007	Shochet	Aug 2000	A
6108576	Alfano et al.	Aug 2000	A
6123733	Dalton	Sep 2000	A
6175655	George	Jan 2001	B1
6205120	Packer et al.	Mar 2001	B1
6219061	Lauer et al.	Apr 2001	B1
6226005	Laferriere	May 2001	B1
6236704	Navab et al.	May 2001	B1
6243098	Lauer et al.	Jun 2001	B1
6249594	Hibbard	Jun 2001	B1
6255655	Mc Croskey et al.	Jul 2001	B1
6264610	Zhu	Jul 2001	B1
6268846	Georgiev	Jul 2001	B1
6278460	Myers et al.	Aug 2001	B1
6282256	Grass et al.	Aug 2001	B1
6289235	Webber et al.	Sep 2001	B1
6304771	Yodh et al.	Oct 2001	B1
6320928	Vaillant et al.	Nov 2001	B1
6324241	Besson	Nov 2001	B1
6377257	Borrel	Apr 2002	B1
6377266	Baldwin	Apr 2002	B1
6384821	Borrel	May 2002	B1
6404843	Vaillant	Jun 2002	B1
6415013	Hsieh et al.	Jul 2002	B1
6470067	Harding	Oct 2002	B1
6470070	Menhardt	Oct 2002	B2
6473793	Dillon et al.	Oct 2002	B1
6475150	Haddad	Nov 2002	B2
6507633	Elbakri et al.	Jan 2003	B1
6510241	Vaillant et al.	Jan 2003	B1
6519355	Nelson	Feb 2003	B2
6526305	Mori	Feb 2003	B1
6557102	Wong et al.	Apr 2003	B1
6559958	Motamed	May 2003	B2
6591004	VanEssen et al.	Jul 2003	B1
6615063	Ntziachristos et al.	Sep 2003	B1
6633688	Nixon	Oct 2003	B1
6636623	Nelson et al.	Oct 2003	B2
6654012	Lauer et al.	Nov 2003	B1
6658142	Kam et al.	Dec 2003	B1
6664963	Zatz	Dec 2003	B1
6674430	Kaufman et al.	Jan 2004	B1
6697508	Nelson	Feb 2004	B2
6707878	Claus et al.	Mar 2004	B2
6718195	Van Der Mark et al.	Apr 2004	B2
6731283	Navab	May 2004	B1
6740232	Beaulieu	May 2004	B1
6741730	Rahn et al.	May 2004	B2
6744253	Stolarczyk	Jun 2004	B2
6744845	Harding et al.	Jun 2004	B2
6745070	Wexler et al.	Jun 2004	B2
6747654	Laksono et al.	Jun 2004	B1
6754299	Patch	Jun 2004	B2
6765981	Heumann	Jul 2004	B2
6768782	Hsieh et al.	Jul 2004	B1
6770893	Nelson	Aug 2004	B2
6771733	Katsevich	Aug 2004	B2
6778127	Stolarczyk et al.	Aug 2004	B2
6785409	Suri	Aug 2004	B1
6798417	Taylor	Sep 2004	B1
6807581	Starr et al.	Oct 2004	B1
6825840	Gritz	Nov 2004	B2
6825843	Allen et al.	Nov 2004	B2
6923906	Oswald et al.	Aug 2005	B2
6947047	Moy et al.	Sep 2005	B1
6978206	Pu	Dec 2005	B1
7003547	Hubbard	Feb 2006	B1
7006101	Brown et al.	Feb 2006	B1
7031022	Komori et al.	Apr 2006	B1
7034828	Drebin et al.	Apr 2006	B1
7039723	Hu	May 2006	B2
7050953	Chiang et al.	May 2006	B2
7054852	Cohen	May 2006	B1
7058644	Patchet et al.	Jun 2006	B2
7076735	Callegari	Jul 2006	B2
7098907	Houston et al.	Aug 2006	B2
7120283	Thieret	Oct 2006	B2
7133041	Kaufman et al.	Nov 2006	B2
7154985	Dobbs	Dec 2006	B2
7167176	Sloan et al.	Jan 2007	B2
7184041	Heng et al.	Feb 2007	B2
7185003	Bayliss et al.	Feb 2007	B2
7219085	Buck et al.	May 2007	B2
7242401	Yang et al.	Jul 2007	B2
7262770	Sloan et al.	Aug 2007	B2
7274368	Keslin	Sep 2007	B1
7299232	Stakutis et al.	Nov 2007	B2
7315926	Fridella et al.	Jan 2008	B2
7324116	Boyd et al.	Jan 2008	B2
7339585	Verstraelen et al.	Mar 2008	B2
7472156	Philbrick et al.	Dec 2008	B2
7502869	Boucher et al.	Mar 2009	B2
7506375	Kanda et al.	Mar 2009	B2
7552192	Carmichael	Jun 2009	B2
7609884	Stalling	Oct 2009	B1
7693318	Stalling	Apr 2010	B1
7701210	Ichinose	Apr 2010	B2
7778392	Bergman	Aug 2010	B1
7876944	Stalling	Jan 2011	B2
7889895	Nowinski	Feb 2011	B2
7899516	Chen et al.	Mar 2011	B2
7907759	Hundley	Mar 2011	B2
7956612	Sorensen	Jun 2011	B2
7983300	Vaughan et al.	Jul 2011	B2
7991837	Tahan	Aug 2011	B1
7995824	Yim	Aug 2011	B2
8041094	Bernard	Oct 2011	B2
8107592	Bergman	Jan 2012	B2
8189002	Westerhoff	May 2012	B1
8233690	Ng	Jul 2012	B2
8319781	Westerhoff	Nov 2012	B2
8369600	Can et al.	Feb 2013	B2
8386560	Ma	Feb 2013	B2
8392529	Westerhoff	Mar 2013	B2
8508539	Vlietinck	Aug 2013	B2
8538108	Shekhar	Sep 2013	B2
8542136	Owsley et al.	Sep 2013	B1
8548215	Westerhoff	Oct 2013	B2
8634622	Woods	Jan 2014	B2
8775510	Westerhoff	Jul 2014	B2
8976190	Westerhoff	Mar 2015	B1
9019287	Westerhoff	Apr 2015	B2
9167027	Westerhoff	Oct 2015	B2
9299156	Zalis	Mar 2016	B2
9355616	Westerhoff	May 2016	B2
9454813	Westerhoff	Sep 2016	B2
9509802	Westerhoff	Nov 2016	B1
9524577	Westerhoff	Dec 2016	B1
9531789	Westerhoff	Dec 2016	B2
9595242	Westerhoff	Mar 2017	B1
9728165	Westerhoff	Aug 2017	B1
9749245	Stalling	Aug 2017	B2
9836872	Erhard	Dec 2017	B2
9860300	Westerhoff	Jan 2018	B2
9898855	Westerhoff	Feb 2018	B2
9904969	Westerhoff	Feb 2018	B1
9984460	Westerhoff	May 2018	B2
9984478	Westerhoff	May 2018	B2
10038739	Westerhoff	Jul 2018	B2
10043482	Westerhoff	Aug 2018	B2
10070839	Westerhoff	Sep 2018	B2
10296199	Gkanatsios	May 2019	B2
10311541	Westerhoff	Jun 2019	B2
10320684	Stalling	Jun 2019	B2
10373368	Westerhoff	Aug 2019	B2
10380970	Westerhoff	Aug 2019	B2
10395398	Westerhoff	Aug 2019	B2
10430914	Westerhoff	Oct 2019	B2
10540803	Westerhoff	Jan 2020	B2
10614543	Westerhoff	Apr 2020	B2
10631812	Westerhoff	Apr 2020	B2
10686868	Westerhoff	Jun 2020	B2
10706538	Westerhoff	Jul 2020	B2
10820877	Westerhoff	Nov 2020	B2
20010026848	Van Der Mark	Oct 2001	A1
20020016813	Woods et al.	Feb 2002	A1
20020034817	Henry et al.	Mar 2002	A1
20020049825	Jewett et al.	Apr 2002	A1
20020080143	Morgan et al.	Jun 2002	A1
20020089587	White et al.	Jul 2002	A1
20020099290	Haddad	Jul 2002	A1
20020099844	Baumann et al.	Jul 2002	A1
20020120727	Curley et al.	Aug 2002	A1
20020123680	Vailant	Sep 2002	A1
20020138019	Wexler	Sep 2002	A1
20020150202	Harding	Oct 2002	A1
20020150285	Nelson	Oct 2002	A1
20020180747	Lavelle et al.	Dec 2002	A1
20020184238	Chylla	Dec 2002	A1
20020184349	Maukyan	Dec 2002	A1
20030001842	Munshi	Jan 2003	A1
20030031352	Nelson et al.	Feb 2003	A1
20030059110	Wilt	Mar 2003	A1
20030065268	Chen et al.	Apr 2003	A1
20030086599	Armato	May 2003	A1
20030103666	Edie et al.	Jun 2003	A1
20030120743	Coatney et al.	Jun 2003	A1
20030123720	Launav et al.	Jul 2003	A1
20030149812	Schoenthal et al.	Aug 2003	A1
20030158786	Yaron	Aug 2003	A1
20030176780	Arnold	Sep 2003	A1
20030179197	Sloan et al.	Sep 2003	A1
20030194049	Claus et al.	Oct 2003	A1
20030220569	Dione	Nov 2003	A1
20030220772	Chiang et al.	Nov 2003	A1
20030227456	Gritz	Dec 2003	A1
20030234791	Boyd et al.	Dec 2003	A1
20040010397	Barbour et al.	Jan 2004	A1
20040012596	Allen et al.	Jan 2004	A1
20040015062	Ntziachristos et al.	Jan 2004	A1
20040022348	Heumann	Feb 2004	A1
20040059822	Jiang	Mar 2004	A1
20040066384	Ohba	Apr 2004	A1
20040066385	Kilgard	Apr 2004	A1
20040066891	Freytag	Apr 2004	A1
20040078238	Thomas et al.	Apr 2004	A1
20040102688	Walker	May 2004	A1
20040125103	Kaufman	Jul 2004	A1
20040133652	Miloushev et al.	Jul 2004	A1
20040147039	Van Der Mark	Jul 2004	A1
20040162677	Bednar	Aug 2004	A1
20040170302	Museth et al.	Sep 2004	A1
20040210584	Nir et al.	Oct 2004	A1
20040215858	Armstrong et al.	Oct 2004	A1
20040215868	Solomon et al.	Oct 2004	A1
20040239672	Schmidt	Dec 2004	A1
20040240753	Hu	Dec 2004	A1
20050012753	Karlov	Jan 2005	A1
20050017972	Poole et al.	Jan 2005	A1
20050066095	Mullick et al.	Mar 2005	A1
20050088440	Sloan et al.	Apr 2005	A1
20050128195	Houston et al.	Jun 2005	A1
20050152590	Thieret	Jul 2005	A1
20050165623	Landi et al.	Jul 2005	A1
20050225554	Bastos et al.	Oct 2005	A1
20050231503	Heng et al.	Oct 2005	A1
20050239182	Berzin	Oct 2005	A1
20050240628	Jiang et al.	Oct 2005	A1
20050256742	Kohan et al.	Nov 2005	A1
20050259103	Kilgard et al.	Nov 2005	A1
20050270298	Thieret	Dec 2005	A1
20050271302	Khamene et al.	Dec 2005	A1
20060010438	Brady et al.	Jan 2006	A1
20060010454	Napoli et al.	Jan 2006	A1
20060028479	Chun	Feb 2006	A1
20060034511	Verstraelen	Feb 2006	A1
20060066609	Iodice	Mar 2006	A1
20060197780	Watkins et al.	Sep 2006	A1
20060214949	Zhang	Sep 2006	A1
20060239540	Serra	Oct 2006	A1
20060239589	Omernick	Oct 2006	A1
20060282253	Buswell et al.	Dec 2006	A1
20070005798	Gropper et al.	Jan 2007	A1
20070038939	Challen	Feb 2007	A1
20070046966	Mussack	Mar 2007	A1
20070067497	Craft et al.	Mar 2007	A1
20070092864	Reinhardt	Apr 2007	A1
20070097133	Stauffer et al.	May 2007	A1
20070116332	Cai et al.	May 2007	A1
20070127802	Odry	Jun 2007	A1
20070156955	Royer, Jr.	Jul 2007	A1
20070165917	Cao et al.	Jul 2007	A1
20070185879	Roublev et al.	Aug 2007	A1
20070188488	Choi	Aug 2007	A1
20070226314	Eick et al.	Sep 2007	A1
20070255704	Baek et al.	Nov 2007	A1
20070280518	Nowinski	Dec 2007	A1
20080009055	Lewnard	Jan 2008	A1
20080042923	De Laet	Feb 2008	A1
20080086557	Roach	Apr 2008	A1
20080115139	Inglett et al.	May 2008	A1
20080137929	Chen et al.	Jun 2008	A1
20080147554	Stevens et al.	Jun 2008	A1
20080155890	Oyler	Jul 2008	A1
20080174593	Ham	Jul 2008	A1
20080208961	Kim et al.	Aug 2008	A1
20080224700	Sorensen	Sep 2008	A1
20080281908	McCanne et al.	Nov 2008	A1
20080317317	Shekhar	Dec 2008	A1
20090005693	Brauner et al.	Jan 2009	A1
20090043988	Archer et al.	Feb 2009	A1
20090077097	Lacapra et al.	Mar 2009	A1
20090147793	Hayakawa et al.	Jun 2009	A1
20090208082	Westerhoff et al.	Aug 2009	A1
20090210487	Westerhoff et al.	Aug 2009	A1
20090225076	Vlietinck	Sep 2009	A1
20090245610	Can et al.	Oct 2009	A1
20090313170	Goldner et al.	Dec 2009	A1
20100054556	Novatzky	Mar 2010	A1
20100060652	Karlsson	Mar 2010	A1
20100123733	Zaharia	May 2010	A1
20100174823	Huang	Jul 2010	A1
20100272342	Berman et al.	Oct 2010	A1
20100278405	Kakadiaris et al.	Nov 2010	A1
20110044524	Wang et al.	Feb 2011	A1
20110112862	Yu	May 2011	A1
20120078088	Whitestone et al.	Mar 2012	A1
20120226916	Hahn	Sep 2012	A1
20120233153	Roman et al.	Sep 2012	A1
20130176319	Westerhoff	Jul 2013	A1
20130195329	Canda	Aug 2013	A1
20140282008	Verard	Sep 2014	A1
20150131881	Gnanamani	May 2015	A1
20150201910	Zhao	Jul 2015	A1
20150213288	Bilodeau et al.	Jul 2015	A1
20160012181	Massey	Jan 2016	A1
20170011514	Westerhoff	Jan 2017	A1
20170065370	Nakai	Mar 2017	A1
20170346883	Westerhoff	Mar 2017	A1
20170098329	Westerhoff	Apr 2017	A1
20170104811	Westerhoff	Apr 2017	A1
20170178593	Westerhoff	Jun 2017	A1
20190318512	Westerhoff	Oct 2019	A1

Foreign Referenced Citations (44)

Number	Date	Country
10317384	Apr 2004	DE
0492897	Jul 1992	EP
0502187	Sep 1992	EP
0611181	Aug 1994	EP
0476070	Aug 1996	EP
0925556	Jun 1999	EP
0953943	Nov 1999	EP
0964 366	Dec 1999	EP
187340	Mar 2001	EP
2098895	Sep 2009	EP
2098994	Sep 2009	EP
2405344	Jan 2012	EP
WO9016072	Dec 1990	WO
WO9102320	Feb 1991	WO
WO9205507	Apr 1992	WO
WO9642022	Dec 1996	WO
WO9810378	Mar 1998	WO
WO9812667	Mar 1998	WO
WO9833057	Jul 1998	WO
WO0120546	Mar 2001	WO
WO0134027	May 2001	WO
WO0163561	Aug 2001	WO
WO0174238	Oct 2001	WO
WO0185022	Nov 2001	WO
WO0241760	May 2002	WO
WO02067201	Aug 2002	WO
WO02082065	Oct 2002	WO
WO03061454	Jul 2003	WO
WO03088133	Oct 2003	WO
WO03090171	Oct 2003	WO
WO03098539	Nov 2003	WO
WO04019782	Mar 2004	WO
WO04020996	Mar 2004	WO
WO04020997	Mar 2004	WO
WO04034087	Apr 2004	WO
WO04044848	May 2004	WO
WO04066215	Aug 2004	WO
WO04072906	Aug 2004	WO
WO05071601	Aug 2005	WO
WO09029636	Mar 2009	WO
WO09067675	May 2009	WO
WO09067680	May 2009	WO
WO11065929	Jun 2011	WO
WO2015063188	May 2015	WO

Non-Patent Literature Citations (88)

Entry
JP2018-524544, Office Action, dated Jun. 2, 2020, 4 pages (& English translation).
EP3329405, Office Action, dated Apr. 6, 2020, 5 pages.
ATI Website Index, http://www.ati.com/developer/index.html, Dec. 20, 2002, 2 pages.
Boone et al., Recognition of Chest Radiograph Orientation for Picture Archiving and Communications Systems Display Using Neural Networks, J. Digital Imaging, 1992, 5(3), 190-193.
Boone et al., Automated Recognition of Lateral from PA Chest Radiographs: Saving Seconds in a PACS Environment, J. Digital Imaging, 2003, 16(4), 345-349.
Luo et al., Automatic Image Hanging Protocol for Chest Radiographs in a PACS, IEEE Transactions on Information Technology in Biomedicine, 2006, 10(2), 302-311.
Cabral et al., Accelerated Volume Rendering and Tomographic Reconstruction Using Texture Mapping Hardware⋅, Silicon Graphics Computer Systems, 1995 IEEE, DD. 91-97.
Carr, Nathan A., Jesse D. Hall, John C. Hart, The ray engine, Proceedings of the ACM Siggraph/Eurographics conference on Graphics hardware, Sep. 1-2, 2002, pp. 37-46.
Chidlow, et al, Rapid Emission Tomography Reconstruction, Proceedings of the 2003 Eurographics/IEEE TVCG Workshop on Volume Graphics, Tokyo, Japan, Jul. 7-8, 2003, 13 pages.
Cohen, Michael, et al., A Progressive Refinement Approach to Fast Radiosity Image Generation, Computer Graphics, vol. 22, No. 4, Aug. 1988, pp. 75-84.
Corner, B., University of Nebraska-Lincoln, MatLab.txt, 2003, 1 page.
Dachille, et al., High-Quality Volume Rendering Using Texture Mapping Hardware, Siggraph/Eurographics Hardware Workshop (1998) (8 pages).
Dempster, et al., Maximum Likelihood From Incomplete Data Via the EM Algorithm, Harvard University and Educational Testing Service, Dec. 8, 1976, pp. 1-38.
Dennis, C, et al.,, Overview of X-Ray Computed Tomography, http://www.howstuffworks.com/framed.htm?parent=c . . . tm&url=http://www.ctlab.geo.utexas.edu/overview/, Dec. 26, 2002, 5 pages.
Dobbins, et al., Digital X-Ray Tomosynthesis: Current State of the Art and Clinical Potential, Physics in Medicine and Biology, vol. 48, pp. R65-R106 (2003).
Doggett, Michael, ATI, Programmability Features of Graphics Hardware, (paper) Apr. 23, 2002, pp. C1-C22.
Doggett, Michael, ATI, Programmability Features of Graphics Hardware, (slideshow) slides 1-62 31 pages.
Du, H., Sanchez-Elez, M., Tabrizi, N., Bagherzadeh, N., Anido, M. L., and Fernandez, M. 2003. Interactive ray tracing on reconfigurable SIMD MorphoSys. In Proceedings of the 2003 Conference on Asia South Pacific Design Automation (Kitakyushu, Japan, Jan. 21 - 24, 2003). ASPDAC. ACM, New York, NY, 471-476.
Eldridge Matthew, Homan Igehy, Pat Hanrahan, Pomegranate: a fully scalable graphics architecture, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p. 443-454, Jul. 2000.
Fang, L., et al., Fast Maximum Intensity Projection Algorithm Using Shear Warp Factorization and Reduced Resampling, Mangetic Resonance in Medicine 47:696-700 (2002).
Filtered Backprojection Reconstruction, http://www.physics.ubd.ca/-mirg/home/tutorial/fbDrecon.html, 216/2003, 5 pages.
Goddard et al., High-speed cone-beam reconstruction: an embedded systems approach, 2002, SPIE vol. 4681, pp. 483-491.
Grass et al., Three-dimensional reconstruction of high contrast objects using C-arm image intensifier projection data, 1999, Computerized Medical Imaging and Graphics, 23, pp. 311-321.
Hadwiger, Markus, et al., Hardware-Accelerated High-Quality Reconstruction of Volumetric Data on PC Graphics Hardware, VRVis Research Center, Vienna, Austria, and Institute of Computer Graphics and Algorithms, Vienna University of Technology, Austria, 9 pages.
Hastreiter et al. (Integrated registration and visualization of medical image data, Proc. Computer Graphics International, Jun. 22-26, 1998, pp. 78-85).
Hopf, M., Ertl, T., Accelerating 3d Convolution Using Graphics Hardware, Proc. IEEE Visualization, 1999, 5 pages.
Hudson, et al., Accelerated Image Reconstruction Using Ordered Subsets of Projection Data, IEEE Transactions on Medical Imaging, vol. 13, No. 4, Dec. 1994, pp. 601-609.
Image Registration Slideshow, 105 pages.
Iterative definition, Merriam-Webster on-line dictionary, printed Aug. 26, 2010, 3 pages.
Jain, Anju, A Programmable Graphics Chip, pcquest.com, Jun. 18, 2001.
Jones et al., Positron Emission Tomographic Images and Expectation Maximization: A VLSI Architecture for Multiple Iterations Per Second, Computer Technology and Imaging, Inc., 1988 IEEE, pp. 620-624.
Kajiya, J. T., Ray tracing volume densities, Proc. Siggraph, Jul. 1984, Computer Graphics, vol. 18, No. 3, pp. 165-174.
Karlsson, Filip; Ljungstedt, Carl Johan; Ray tracing fully implemented on programmable graphics hardware, Master's Thesis, Chalmers University of Technology, Dept. of Computer Engineering, Goteborg, Sweden, copyright © 2004, 29 pages.
Kruger J. and R. Westermann, Acceleration Techniques for GPU-based Volume Rendering, Proceedings of IEEE Visualization, 2003, 6 pages.
Lange et al., EM Reconstruction Algorithms for Emission and Transmission Tomography, J Computer Assisted Tomography 8, DD. 306, et seq. (1984).
Lange et al., Globally Convergent Algorithms for Maximum a Posteriori Transmission Tomography, IEEE Transactions on Image Processing, Vo. 4, No. 10, Oct. 1995, pp. 1430-1438.
Li et al., Tomographic Optical Breast Imaging Guided by Three-Dimensional Mammography, Applied Optics, Sep. 1, 2003, vol. 42, No. 25, pp. 5181-5190.
Li, et al., A Brick Caching Scheme for 30 Medical Imaging, Apr. 15-18, 2004, IEEE International Symposium on Biomedical Imaging: Macro to Nano 2004, vol. 1, pp. 563-566.
Maes, et al. Multimodality Image Registration by Maximization of Mutual Information, IEEE Tran. on Medical Imaging, vol. 16, No. 2, Apr. 1997. pp. 187-198).
Max, N., Optical Models for Direct Volume Rendering, IEEE Transactions on Visualization and Computer Graphics, Jun. 1995, 1(2): pp. 99-108.
McCool, M. et al., Shader Algebra, 2004, pp. 787-795.
McCool, Michael J., Smash: A Next-Generation API for Programmable Graphics Accelerators, Technical Report CS-200-14, Computer Graphics Lab Dept. of Computer Science, University of Waterloo, Aug. 1, 2000.
Microsoft, Architectural Overview Direct for 3D, http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dx8_c/directx_cpp/Graphics/ProgrammersGuide/GettingStarted/ Architecture, 12120/2002, 22 pages.
Mitchell, Jason L., RadeonTM 9700 Shading, Siggraph 2002—State of the Art in Hardware Shading Course Notes, DD.3.1-1-3.1-39, 39 pages.
Mitschke et al., Recovering the X-ray projection geometry for three-dimensional tomographic reconstruction with additional sensors: Attached camera versus external navigation system, 2003, Medical Image Analysis, vol. 7, pp. 65-78.
Mueller, K., and R. Yagel, Rapid 3-D Cone Beam Reconstruction With the Simultaneous Algebraic Reconstruction Technique (SART) Using 2-D Texture Mapping Hardware, IEEE Transactions on Medical Imaging, Dec. 2000, 19(12): pp. 1227-1237.
Navab, N., et al., 3D Reconstruction from Projection Matrices in a C-Arm Based 3D-Angiography System, W.M. Wells e al., eds., MICCAI'98, LNCS 1496, pp. 119-129, 1998.
Parker, S., et al., Interactive Ray Tracing for Isosurface rendering, IEEE, 1998, pp. 233-258.
PCT/US2008/084282, Preliminary and International Search Reports, dated May 11, 2011, 7 pages.
PCT/US2005/000837, Preliminary and International Search Reports, dated May 11, 2005, 7 pages.
PCT/US2008/74397, Preliminary and International Search Reports, dated Dec. 3, 2008 , 7 pages.
PCT/US2008/84368, Preliminary and International Search Reports, dated Jan. 13, 2009, 7 pages.
PCT/EP2016/067886, Preliminary and International Search Reports, dated Jan. 17, 2017, 18 pages.
PCT/EP2018/075744, Preliminary and International Search Reports, dated Feb. 1, 2019, 17 pages.
PCT/US2008/84376, Preliminary and International Search Reports, dated Jan. 12, 2009, 6 pages.
Pfister, H., et. al., The VolumePro real-time ray-casting System, Computer Graphics Proceedings of Siggraph), Aug. 1999, No. 251-260.
Phong, B. T. Illumination for Computer Generated Pictures, Communications of the ACM, 18(6), Jun. 1975, pp. 311-317.
Porter, D. H. 2002. Volume Visualization of High Resolution Data using PC-Clusters. Tech. rep., University of Minnesota. Available at http://www.lcse.umn.edu/hvr/pc_vol_rend_L.pdf.
Potmesil, M. and Hoffert, E. M. 1989. The pixel machine: a parallel image computer. In Proceedings of the 16th Annual Conference on Computer Graphics and interactive Techniques Siggraph '89. ACM, New York, NY, 69-78.
Purcell, T., et al., Real-time Ray Tracing on Programmable Graphics Hardware, Department of Computer Science, Stanford University, Stanford, CA, Submitted for review to Siggraph 2002, 2002. http://graphics.stanford.edu/papers/rtongfx/rtongfx_submit.pdf.
Purcell, T., et. al., Ray tracings on Programmable Graphics Hardware, Computer Graphics (Proceedings of Siggraph), 1998, pp. 703-712.
Purcell, Timothy J., Craig Donner, Mike Cammarano , Henrik Wann Jensen , Pat Hanrahan, Photon mapping on programmable graphics hardware, Proceedings of the ACM Siggraph/Eurographics conference on Graphics hardware, Jul. 26-27, 2003, 11 pages.
Ramirez et al. (Prototypes stability analysis in the design of a binning strategy for mutual information based medical image registration, IEEE Annual Meeting of the Fuzzy Information, Jun. 27-30, 2004, vol. 2, pp. 862-866.
Rib Cage Projection, downloaded from http://www.colorado.edu/physics/2000/tomography/final_rib_cage.html on Dec. 26, 2002, 3 pages.
Roettger, Stefan, et al., Smart Hardware-Accelerated Volume Rendering, Joint Eurographics—IEEE TCVG Symposium on Visualization, 2003, pp. 231-238, 301.
Sandborg, Michael, Computed Tomography: Physical principles and biohazards, Department of Radiation Physics, Faculty of Health Sciences, Linkoping University, Sweden, Report 81 ISSN 1102-1799, Sep. 1995 ISRN ULI-RAD-R--81--SE, 18 pages.
Sarrut et al. (Fast 30 Image Transformations for Registration Procedures, Proc. Int. Conf. on Image Analysis and Processing, Sep. 27-29, 1999, pp. 446-451.
Selldin, Hakan, Design and Implementation of an Application Programming Interface for Volume Rendering, Linkopings Universitet.
Shekhar, R.; Zagrodsky, V., Cine MPR: interactive multiplanar reformatting of four-dimensional cardiac data using hardware-accelerated texture mapping, IEEE Transactions on Information Technology in Biomedicine, vol. 7, No. 4, pp. 384-393, Dec. 2003.
Silver, et al., Determination and correction of the wobble of a C-arm gantry, Medical Imaging 2000: Image Processing, Kenneth M. Hanson, ed., Proceedings of SPIE vol. 3970 (2000).
Stevens, Grant, et al., Alignment of a Volumetric Tomography System, Med. Phys., 28 (7), Jul. 2001.
Tao, W., Tomographic mammography using a limited number of low dose cone beam projection images, Medical Physics, AIP, Melville, NY vol. 30, pp. 365-380, Mar. 2003, ISSN: 0094-2405.
Tasdizen, T. , Ross Whitaker, Paul Burchard , Stanley Osher, Geometric surface processing via normal maps, ACM Transactions on Graphics (TOG), v.22 n.4, p. 1012-1033, Oct. 2003.
Tasdizen, T.; Whitaker, R.; Burchard, P.; Osher, S.; Geometric surface smoothing via anisotropic diffusion of normals, IEEE Visualization, VIS 2002, Nov. 2002, pp. 125-132.
Technical Brief: Nvidia nfiniteFX Engine: Programmable Pixel Shaders, Nvidia Corporation, 5 pages.
Technical Brief: Nvidia nfiniteFX Engine: Programmable Vertex Shaders, Nvidia Corporation, 12 pages.
Viola, I, et al., Hardware Based Nonlinear Filtering and Segmentation Using High Level Shading Languages, Technical Report TR-186-2-03-07, May 2003, 8 pages.
Viola, P., Alignment by Maximization of Mutual Information, PhD Thesis MIT (Also Referred to as—A1 Technical report No. 1548), MIT Artificial Intelligence Lab, Jun. 1, 1995, pp. 1-29.
Weiler, M, M. Kraus and T. Ertl, Hardware-Based View-Independent Cell Projection, Proceedings IEEE Symposium on Volume Visualization 2002, pp. 13-22.
Weiler, M. et al., Hardware-based ray casting for tetrahedral meshes, IEEE Visualization, VIS 2003, Oct. 24-24, 2003, pp. 333-340.
Weiler, M. et al., Hardware-Based view-Independent Cell Projection, IEEE, 2002, pp. 13-22.
Weiskopf, D., T. Schathitzel, T. Ertl, GPU-Based Nonlinear Ray Tracing, Eurographics, vol. 23, No. 3, Aug. 2004.
Wen, Junhai; Zigang Wang; Bin Li; Zhengrong Liang; An investigation on the property and fast implementation of a ray-driven method for inversion of the attenuated Radon transform with variable focusing fan-beam collimators, 2003 IEEE Nuclear Science Symposium Conference Record, vol. 3, Oct. 19-25, 2003, pp. 2138-2142.
Wikipedia, Anonymous, ‘Volume Rendering’ May 30, 2015, retrieved Nov. 4 2016, https://en.wikipedia.org/w/index.php?title=Volume_rendering&oldid=664765767.
Wikipedia, Anonymous, ‘Tomographic Reconstruction’ Dec. 6 2014, retrieved Nov. 4 2016, https://en.wikipedia.org/w/index.php?title=Tomographic_Reconstruction&oldid=636925688.
Wu et al., Tomographic Mammography Using a Limited Number of Low-dose Conebeam Projection Images, Med. Phys., pp. 365-380 (2003).
Xu et al., Toward a Unified Framework for Rapid 30 Computed Tomography on Commodity GPUs, Oct. 19-25, 2003, IEEE Nuclear Science Symposium Conference 2003, vol. 4, pp. 2757-2759.
Xu et al., Ultra-fast 30 Filtered Backprojection on Commodity Graphics Hardware, Apr. 1-18, 2004, IEEE International symposium on Biomedical Imaging: Macro to Nano, vol. 1, pp. 571-574 and corresponding power point presentation.

Related Publications (1)

	Number	Date	Country
	20190318512 A1	Oct 2019	US

Provisional Applications (1)

	Number	Date	Country
	62197956	Jul 2015	US

Continuations (2)

	Number	Date	Country
Parent	15988542	May 2018	US
Child	16446562		US
Parent	15218972	Jul 2016	US
Child	15988542		US

Apparatus and method for visualizing digital breast tomosynthesis and other volumetric images

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

Field of Search

US

International Classifications

Disclaimer

Term Extension

Abstract