Patent Application
20050285853
This invention relates to three dimensional (3D) display systems and more particularly, to a 3D volumetric display system and method of assisting medical diagnostic interpretation of images and data in a virtual-reality environment.
There are many medical imaging systems used to acquire medical images suitable for diagnosing disease or injury. These include X-ray, CT scanner, magnetic resonance imaging (MRI), ultrasound, and nuclear medicine systems. These medical imaging systems are capable of acquiring large amounts of image data during a patient scan. The medical imaging devices are generally networked with a central image management system, such as Picture Archiving and Communication System (PACS).
In most cases, the image data is acquired as a series of contiguous two-dimensional (2D) slice images for diagnostic interpretation. For example, 100 to 1000 2D images may be acquired and viewed one at a time by scrolling through all the 2D images by the physician to diagnose the disease or injury. As a result, the physician is faced with the formidable task of viewing all the acquired 2D images to locate the region of interest where the disease or injury has occurred and then to select the diagnostically most useful images. As the image data sets get larger, this method of scrolling through the 2D images using a computer mouse by the physician and viewing each image becomes very time consuming and monotonous.
What is needed therefore is a system and method to improve diagnostic process and workflow through advanced visualization and user-interface technologies. What is also needed is a system and method of conducting diagnostic interpretation of the image data in a virtual-reality environment. What is also needed is a system and method of interacting with a patient's anatomy to conduct diagnostic interpretation of the image data by using tactile feedback on a variety of anatomical structures. What is also needed is a system and method of enabling a physician to contact and to manipulate the images for diagnosing anomalies in the virtual-reality environment. What is also needed is a graphical user interface (GUI) to permit an operator to use his/her hands to interactively manipulate virtual objects. These improvements would give physicians an ability to quickly navigate through a large image data set and would provide more efficient workflow. It should be understood, of course, that embodiments of the invention may also be used to meet other needs in addition to and/or instead of those set forth above.
In accordance with a preferred first aspect of the invention, an apparatus configured to display 3D volumetric data acquired from a patient by an imaging system is provided. The apparatus comprises a 3D volumetric display system configured to generate a real-time 3D diagnostic display of the 3D volumetric data. The 3D volumetric display system includes a graphical user interface (GUI) configured to permit a user to access, view, and manipulate the 3D volumetric data. The GUI includes a plurality of 3D computer-aided diagnosis (CAD) markers. Each of the 3D CAD markers has a delineator configured to navigate through the 3D volumetric data to locate pathology and to permit a user to compile and to prepare a report containing diagnosis information in a virtual-reality environment.
In accordance with another preferred aspect of the invention, a diagnostic apparatus comprises a display system configured to generate a stereoscopic image acquired from a patient by an imaging system. The display system includes a GUI configured to access simultaneously in a picture archiving and communication system (PACS) and an image workstation and to navigate through the stereoscopic image. The GUI comprises a 3D CAD marker having a delineator generated by a software program. The delineator is configured to navigate through the stereoscopic image to indicate likelihood of an anomaly and to compile and to prepare a report containing diagnosis information in a virtual-reality environment.
In accordance with a further preferred aspect of the invention, a method of assisting diagnostic interpretation of a stereoscopic image in a virtual-reality environment is provided. The method comprises navigating a 3D CAD marker through the stereoscopic image responsive to operator inputs, indicating likelihood of an anomaly in the stereoscopic image of a patient by using the delineator of the 3D CAD marker, displaying diagnosis information about the anomaly in the status bar, receiving an operator input using one of the plurality of command buttons, and generating a report containing the diagnosis information in the virtual-reality environment. The 3D CAD marker has a delineator and a status bar indicator including a plurality of command buttons.
In accordance with yet a further preferred aspect of the invention, a system configured to display a stereoscopic image in a virtual-reality environment is provided. The system comprises means for navigating a 3D CAD marker through the stereoscopic image responsive to operator inputs, means for locating an anomaly in the stereoscopic image of a patient by using the 3D CAD marker, means for displaying diagnosis information of the anomaly in the virtual-reality environment, and means for compiling and preparing a report containing the diagnosis information in the virtual-reality environment.
The haptics-enhanced virtual-reality system 14 is driven by the workstation 16 to display stereoscopic images 52 so that a user can touch and interact with a virtual object 36, i.e., an anatomical structure of a patient's body. The images may be received by the workstation 16 from the PACS 28, which stores images received from the imaging systems 34. Alternatively, the images may be received directly from one of the imaging systems 34, e.g., to allow a virtual examination of the patient's anatomy during a minimally-invasive surgical procedure. Haptic feedback is provided to the operator using the haptic actuators 18 and which apply forces to a user's hands and fingers. The haptic feedback may assist and inform the user of interactions and events within the virtual reality environment 12. The plurality of haptic actuators 18 and the plurality of position sensors or trackers 20 are connected to the workstation 16 to permit interaction in the virtual-reality environment 12. The actuators 18 and the trackers 20 may be mounted to a common user interface device, such as one or more haptic gloves 58 (see
Each imaging system 34 may include an acquisition workstation (not shown) which acts as a gateway between the imaging systems 34 and the network 22. To that end, the acquisition workstation may accept raw image data from the imaging systems 34 and optionally perform pre-processing on image data in preparation for delivering image data to the PACS network 28 for storage in a PACS image database (not shown). In operation, the acquisition workstation (not shown) may convert the image data into DICOM, DEFF, or other suitable format.
The display system 10 is configured to generate 3D diagnostic displays of 3D volumetric medical data network 22 acquired from a patient by one or more of the imaging systems 34. The 3D displays are generated in the virtual-reality environment 12. The display system 10 permits a user, such as a physician or radiologist, to conduct diagnostic interpretation of images in the virtual reality environment 12 and to interact with the 3D diagnostic displays. The imaging systems 34 may include, but are not limited to, magnetic resonance imaging devices, computed tomography (CT) devices, ultrasound devices, nuclear imaging devices, X-ray devices, and/or a variety of other types of imaging devices. It should be understood that imaging systems 34 are not limited to medical imaging devices and also include scanners and imaging devices from other fields.
As shown in
It will be appreciated that, although the interface tool (GUI) 38 is shown as being located in the virtual reality environment 12, the GUI is actually implemented by program logic stored and executed in the workstation 16. The workstation 16 receives feedback information from the position sensors 20 and processes the feedback information (in accordance with the stored program logic and in accordance with the stored image data received via the network 22) to drive the haptic actuators 18 and to drive the image projection system 46 (e.g., to alter the GUI display and/or to alter the displayed image data).
As shown best in
While the stereoscopic images 52 provide sufficient information to conduct diagnostic interpretation of the 3D images, many physicians or radiologists prefer to see 2D sectional images taken through the region of interest within the anatomical structure of the patient's body. Such 2D sectional images are often presented as three orthogonal planes including transverse, sagittal, and coronal images 56a, 56b, 56c respectively, depending on their orientation with respect to the patient. Thus, using the 3D diagnostic display to identify a region of interest in the patient, as shown in
As mentioned above, the display system 10 comprises the haptic actuators 18 which have robotic manipulators (not shown) that apply force to the user's hand corresponding to the environment that a virtual effector (i.e., muscles become active in response to stimulation) is in. The haptics feedback is used to indicate whether the user's hand is in contact with the anatomical structure of a patient's body 36. As previously mentioned, the display system 10 includes haptic glove 58 upon which the haptic actuators 18 are mounted and which is configured to be worn by the user to provide the tactile sensation to the hand of the user to simulate contact with the virtual object 36. The haptic glove 58 provides a sense of touch in the virtual reality environment 12. For example, if a user tries to grab the virtual object 36, the haptic glove 58 provides feedback to let the user know that the virtual object 36 is in contact with the user's hand. Also, the haptic glove 58 provides a mechanism to keep the user's hand from passing through the virtual object 36.
Referring to
During imaging of a subject of interest, such as a portion of an anatomical structure of a patient's body 36, one or more of the imaging systems 34 are used to acquire a plurality of 2D images of the subject interest. The PACS 28 archives the plurality of 2D images so they can be selectively retrieved and accessed. Other patient data may also be retrieved, such as electronic medical record data which may be retrieved from the EMR system 32. The plurality of 2D images and/or the patient's medical record is then displayed in the form of 2D viewports 64 in the virtual reality environment 12. The display system 10 is capable of displaying the 2D images 64, 3D planner images 56, and volumetric 3D diagnostic images 66 simultaneously as best shown in
The cubical model in
The haptic tool 72 comprises a virtual handle 78 attached to one of the plurality corners 76 of virtual lens 74 to permit a user to navigate through the anatomical structure of a patient's body 70. The virtual handle 78 is configured to be held by the user wearing the haptic glove 58 when the haptic tool 72 is navigated through the virtual anatomical structure of a patient's body 70. The haptic actuators 18 in
The haptic tool 72 further comprises a virtual tab 80 disposed on at least two of the plurality of corners 76 of the virtual lens 74. The virtual tabs 80 permit the user to change the dimensional size and orientation of the haptic tool 72 within the virtual anatomical structure of a patient's body 70. The haptic tool is capable of depicting the cross-sectional image 68 that is characterized by combination of three orthogonal planes including transverse, sagittal, and coronal planes as depicted by 56a, 56b, and 56c respectively. The haptic tool 72 is configured to be positioned at various orientations and angles with respect to the virtual anatomical structure of a patient's body 70 to generate the cross-sectional image 68 within stereoscopic image. For example, the haptic tool 72 is capable of displaying a cross-sectional image that is configured to be constructed from a combination of transverse, sagittal, and coronal images. In operation, when the haptic tool 72 is navigated through the stereoscopic image responsive to user inputs, coordinates of the haptic tool 72 are mapped with a boundary of the virtual anatomical structure of a patient's body 70 to generate the cross-sectional image 68 and then the cross-sectional image is displayed to permit a diagnostic interpretation of the image to be conducted.
The 3D CAD marker 82 includes a status indicator 86 which is associated with the delineator 84 to display diagnosis information in the virtual-reality environment. The status indicator 86 comprises a plurality of command buttons configured to receive a user input to compile the diagnostic report. The plurality of command buttons comprises first and second buttons (e.g., YES and NO command buttons) 88a & 88b, respectively. The YES command button 88a is configured to receive a user input to accept diagnosis information, e.g., responsive to the user pressing the YES command button 88a. The NO command button 88b is configured to receive a user input to discard unwanted diagnosis information, e.g., responsive to the user pressing the NO command button 88b. When the user wearing the haptic glove 58 contacts the YES or NO button, the position of the user's hand is detected using the position sensors 20 and in turn, the workstation 16 produces an activating signal to drive the haptic actuators 18 for outputting forces to the user's hand.
During operation, the user wears the haptic glove 58 while holding the 3D CAD marker 82 and the 3D CAD marker is navigated through the stereoscopic image or the virtual anatomical structure of a patient's body 70 by the workstation 16 responsive to the user inputs. The 3D CAD marker 82 indicates the likelihood of an anomaly in the stereoscopic image of a patient by using the delineator 84 and displays diagnosis information about the anomaly in the status bar 86. Finally, upon receiving the user input using the command buttons 88a, the display system generates a report containing the diagnosis information in the virtual-reality environment 12. The 3D CAD marker 82 includes a color code feature which enables a user to display diagnosis information in various colors within the GUI.
During operation, the display system 10 receives a user input associated with GUI to conduct measurement in the virtual-reality environment 12. The user wearing the haptic glove 58 clicks on the haptic toolbox icon 44, located in the tool palette window 40 shown in
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from its scope. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
