Auto-focus tool for multimodality image review

Abstract

Examples of the present disclosure describe systems and methods for an auto-focus tool for multimodality image review. In aspects, an image review system may provide for the display of a set of medical images representing one or more imaging modalities. The system may also provide an auto-focus tool that may be used during the review of the set of medical images. After receiving an instruction to activate the auto-focus tool, the system may receive a selection of an ROI in at least one of the images in the set of medical images. The auto-focus tool may identify the location of the ROI within the image and use the identified ROI location to identify a corresponding area or ROI in the remaining set of medical images. The auto-focus tool may orient and display the remaining set of medical images such that the identified area or ROI is prominently displayed.

Description

BACKGROUND

An ongoing tension is found in today's healthcare environments, such as radiology departments, between providing high-quality image review and maintaining adequate patient throughput to keep costs under control. Despite ongoing advances in imaging technology and related data processing systems, it is the radiologist who continues to bear the burden of the cost-quality tradeoff. As used herein, radiologist generically refers to a medical professional that analyzes medical images and makes clinical determinations therefrom.

Radiologists have expressed a clinical need for an automated solution to correlate corresponding regions of interests (ROIs) in medical images acquired from various imaging modalities. ROIs may be associated with breast abnormalities such as masses or microcalcification. ROIs could also be associated with specific focus areas of the breast that radiologist are interested reviewing in more detail. For any given patient, a radiologist may be able to review images from mammography, ultrasound, and MRI of the same patient. In an effort to determine the location of the ROI in each image, the radiologist will make a visual comparison, sometimes aided by a separate ruler or simply using the radiologist's hand or fingers. If these areas of interest appear in multiple images, it may lead to the conclusion that the region of interest is indeed a mass or microcalcification. If, on the other hand, there is no distinct region of interest in the second image at the appropriate location, it may lead to a conclusion that the region of interest in the first image is not a mass or microcalcification. Previously presented automated solutions have been proposed to provide for correlation between images of different modalities. However, such solutions proved unsatisfactory due to the vast amount of training data required to train machine learning (ML) mechanisms and the inaccuracy of the results provided by those ML mechanisms. In addition, ML mechanisms involve a substantial amount of processing resources and the automated correlation have resulted in a delay in presenting images to the radiologists. As a result, healthcare professionals have been forced to perform the ROI correlations manually with the aid of image manipulation tools, such as pan and zoom tools. This manual ROI correlation results in excessive mouse clicks and movements that impede image review workflow, add review time and associated costs, and cause fatigue to the radiologist.

It is with respect to these and other general considerations that the aspects disclosed herein have been made. Also, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving the specific problems identified in the background or elsewhere in the present disclosure.

SUMMARY

Examples of the present disclosure describe systems and methods for an auto-focus tool for multimodality image review. In aspects, an image review system may provide for the display of a set of medical images representing one or more imaging modalities. The system may also provide an auto-focus tool that may be used during the review of the set of medical images. After receiving an instruction to activate the auto-focus tool, the system may receive a selection of an ROI in at least one of the images in the set of medical images. The auto-focus tool may identify the location of the ROI within the image and use the identified ROI location to identify a corresponding area or ROI in the remaining set of medical images. The auto-focus tool may orient (e.g., pan, zoom, flip, rotate, align, center) and display the remaining set of medical images such that the identified area or ROI is prominently displayed in the set of medical images.

In one aspect, examples provided in the present disclosure relate to a system comprising: a processor; and memory coupled to the processor, the memory comprising computer executable instructions that, when executed, perform a method. The method comprises receiving a selection of a region of interest (ROI) in a first image of a plurality of images; identifying, using an auto-focus tool, a location of the ROI within the first image; identifying, using the auto-focus tool, an area corresponding to the location of the ROI in at least a second image of the plurality of images, wherein the first image and the second image are different imaging modality types and an automated determination mechanism is used to identify the area corresponding to the location of the ROI; and causing the auto-focus tool to automatically: focus a first field of view on the ROI in the first image; focus a second field of view on the area corresponding to the location of the ROI in the second image.

In a first alternative aspect, the method comprises receiving a selection of a bounding box in a mammography image of a plurality of images, the bounding box identifying an ROI; identifying, using an auto-focus tool, a location of the ROI within the mammography image; identifying, using the auto-focus tool, an area corresponding to the location of the ROI in at least a tomography slice image of the plurality of images and an MRI image of the plurality of images; and causing the auto-focus tool to automatically: magnify the ROI in a field of view of the tomography slice image; and pan to at least one of: a breast comprising the ROI or an image plane identifying the ROI in a field of view of the MRI image.

In a second alternative aspect, the method comprises receiving a selection of a bounding box in a mammography image of a plurality of images, the bounding box identifying an ROI; identifying, using an auto-focus tool, a location of the ROI within the mammography image; identifying, using the auto-focus tool, an area corresponding to the location of the ROI in at least a tomography slice image of the plurality of images, an ultrasound image of the plurality of images, and an MM image of the plurality of images; and causing the auto-focus tool to automatically: magnify the ROI in a field of view of the tomography slice image; pan to a breast comprising the ROI in a field of view of the ultrasound image; and pan to at least one of: a breast comprising the ROI or an image plane identifying the ROI in a field of view of the MRI image.

In an example, the system is an electronic image review system for reviewing medical images within a healthcare environment. In another example, focusing the first field of view on the ROI in the first image comprises centering the ROI within the first field of view and scaling the ROI to increase or decrease a size of the ROI within the first field of view. In another example, the imaging modality types include at least two of: mammography, MM, or ultrasound. In another example, the first image is generated during a current patient visit for a patient and the second image was generated during a previous patient visit for the patient. In another example, the plurality of images is arranged for viewing based on an image viewing layout specified by a user, the image viewing layout enabling the plurality of images to be concurrently presented to a user.

In another example, receiving the selection of the ROI comprises: receiving a selection of a point in the first image; and defining an area surrounding the point as the ROI, wherein in response to receiving the selection of the point in the first image, a bounding box comprising at least a portion of the ROI is automatically applied to the image such that at least a first object in the first image is delineated from at least a second object in the first image. In another example, receiving the selection of the ROI comprises: receiving a selection of a plurality of points in the first image; determining a centroid of the plurality of points; and defining an area surrounding the centroid as the ROI. In another example, the automated determination mechanism is at least one of: a rule set, a mapping algorithm, or an image or object classifier. In another example, a process for identifying the location of the ROI within the first image is based on the imaging modality type of the first image and the process includes the use of at least one of: image view information, spatial coordinate information, image header information, or image orientation data.

In another example, when the first image and a third image of a plurality of images are a same imaging modality type, a mapping function is used to map first ROI identification information of the first image to second ROI identification information of the third image such that the first ROI identification information and the second ROI identification information are a same type. In another example, a mapping function is used to convert first ROI identification information of the first image to second ROI identification information of the second image such that the first ROI identification information and the second ROI identification information are a different type. In another example, focusing the first field of view on the ROI in the first image comprises orienting the first image such that the ROI is at least one of horizontally or vertically centered in a viewport comprising the first image. In another example, wherein focusing the second field of view on the area corresponding to the location of the ROI in the second image comprises applying a scaling factor to the area corresponding to the location of the ROI.

In another aspect, examples provided in the present disclosure relate to a method comprising: receiving, at an image review device, a selection of a region of interest (ROI) in a first image of a plurality of images; identifying, using the auto-focus tool, a location of the ROI within the first image; identifying, using the auto-focus tool, an area corresponding to the location of the ROI in at least a second image of the plurality of images, wherein the first image and the second image are different imaging modality types and an automated determination mechanism is used to identify the area corresponding to the location of the ROI; and causing the auto-focus tool to automatically: focus a first field of view on the ROI in the first image; and focus a second field of view on the area corresponding to the location of the ROI in the second image. In an example, the plurality of images comprises at least a mammography image, an MRI image, and an ultrasound image.

In yet another aspect, examples provided in the present disclosure relate to a computing device comprising: a processor; and an image auto-focus tool configured to: receive a selection of a region of interest (ROI) in a first image of a plurality of images; identify a location of the ROI within the first image; identify an area corresponding to the location of the ROI in at least a second image of the plurality of images, wherein the first image and the second image are different imaging modality types and an automated determination mechanism is used to identify the area corresponding to the location of the ROI; focus a first field of view on the ROI in the first image; and focus a second field of view on the area corresponding to the location of the ROI in the second image, wherein focusing the second field of view comprises at least one of scaling the second image or panning the second image.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and/or advantages of examples will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive examples are described with reference to the following figures.

FIG. 1 illustrates an example input processing system for an auto-focus tool for multimodality image review, as described herein.

FIG. 2 illustrates an example method for utilizing an auto-focus tool for multimodality image review, as described herein

FIGS. 3A and 3B depict a set of multimodality images for illustrating the functionality of the auto-focus tool, as described herein.

FIG. 4 illustrates one example of a suitable operating environment in which one or more of the present embodiments may be implemented.

FIG. 5 illustrates an overview of an example system for an auto-focus tool for multimodality image review, as described herein.

DETAILED DESCRIPTION

Medical imaging has become a widely used tool for identifying and diagnosing ROIs and abnormalities, such as cancers or other conditions, within the human body. Medical imaging processes such as mammography and tomosynthesis are particularly useful tools for imaging breasts to screen for, or diagnose, cancer or other lesions within the breasts. Tomosynthesis systems are mammography systems that allow high resolution breast imaging based on limited angle tomosynthesis. Tomosynthesis, generally, produces a plurality of X-ray images, each of discrete layers or slices of the breast, through the entire thickness thereof. In contrast to conventional two-dimensional (2D) mammography systems, a tomosynthesis system acquires a series of X-ray projection images, each projection image obtained at a different angular displacement as the X-ray source moves along a path, such as a circular arc, over the breast. In contrast to conventional computed tomography (CT), tomosynthesis is typically based on projection images obtained at limited angular displacements of the X-ray source around the breast. Tomosynthesis reduces or eliminates the problems caused by tissue overlap and structure noise present in 2D mammography imaging.

In recent times, healthcare professionals have expressed a clinical need for an automated solution to correlate ROIs in medical images acquired from various imaging modalities, such as mammography, synthesized mammography, tomosynthesis, wide angle tomosynthesis, ultrasound, computed tomography (CT), and magnetic resonance imaging (MM). Proposed solutions have typically involved the use of various ML approaches, which require a vast amount of diverse training data that is generally not readily available for clinical use. Acquiring and using the training data to train an ML model or algorithm, thus, requires a substantial resource investment. This resource investment is further exacerbated by the substantial computing resources (e.g., central processing unit (CPU), memory, and file storage resources) demand required to operate the trained ML model or algorithm. In many cases, the analysis performed by a trained ML model or algorithm is slow (due to the computing resource demand) and the results of the trained ML model or algorithm are inaccurate or imprecise.

For the above reasons, the ROI correlation is still primarily performed manually by healthcare professionals. For example, healthcare professionals use image manipulation tools, such as pan and zoom tools, to focus on an ROI or narrow a field of view in an image. However, the use of such image manipulation tools often results in excessive mouse clicks and movements that impede image review workflow and cause healthcare professionals to fatigue. For instance, when using such image manipulation tools, a user must select multiple tools to accomplish a specific task. Each selected tool must be applied to each image or viewport in a set of medical images to enable the user to manually pan and/or zoom the image/viewport. Each instance of manual panning/zooming may result in multiple mouse clicks and movements while the user attempts to achieve an optimal or acceptable view of the image/viewport. Moreover, in some cases, the image manipulation tools are specific to a particular imaging modality or imaging system (e.g., the image manipulation tools are not multimodal). For instance, a first set of image manipulation tools of a first image review system may be used to view mammography images and a second set of image manipulation tools of a second image review system may be used to view MRI images. To compare the mammography images to the MM images, a healthcare professional may display the mammography and MRI images on separate display screens and manually orient the respective images on each display screen using the respective image manipulation tools. The use of different sets of image manipulation tools is cumbersome, complicated, and requires healthcare professionals to be proficient using multiple sets of image manipulation tools.

To address such issues with traditional methods for ROI correlation, the present disclosure describes systems and methods for an auto-focus tool for multimodality image review. In aspects, an image review system may provide for the display of a set of medical images representing one or more imaging modalities, such as mammography, tomosynthesis, MRI, and ultrasound, among others. As a specific example, the set of medical images may include a current mammography image of a patient's breast (collected during a current patient visit) and one or more prior mammography images of the patient's breast (collected during one or more previous patient visits). Displaying the set of medical images may include the use of one or more hanging protocols. A hanging protocol, as used herein, may describe what images to display (e.g., image attributes and conditions, including modality, anatomy, laterality, procedure, and reason) and how to display the images (e.g., viewport height or width; image zoom, pan, or rotation, image order, tiling). The hanging protocols may enable the simultaneous or concurrent display of multiple images within respective viewports of a display screen. A viewport, as used herein, may refer to a frame, a sub window, or a similar viewing area within a display area of a device. As used herein, a mammography image may refer to an image acquired on a conventional two-dimensional (2D) mammography system or a synthesized 2D image that is created from combining information from a tomosynthesis data set. For ease of reading, both can be referred to as a mammogram or a mammography image.

The system may also provide an auto-focus tool that may be used during the review of the set of medical images. The auto-focus tool may provide for automatically orienting (e.g., panning, zooming, centering, aligning) images in the set of medical images in accordance with a selected portion of an image in the set of medical images. Upon activation of the auto-focus tool, the system may enable a user, such as a healthcare professional (e.g., a radiologist, a surgeon or other physician, a technician, a practitioner, or someone acting at the behest thereof) to select an ROI in a displayed image. Alternatively, the selection of the ROI in a displayed image may cause the auto-focus tool to be activated or be part of the process for activating the auto-focus tool. The auto-focus tool may identify the location of the selected ROI within the image based on image attributes, such as view position information (e.g., bilateral craniocaudal (CC) view, mediolateral oblique (MLO) view, true lateral view), laterality (e.g., left, right), image coordinates and direction information (e.g., image position and image orientation with respect to the patient), and other information embedded in the DICOM header (e.g., pixel spacing, slice thickness, slice location) or information burned in the pixel data of the image. Additionally, information within an image, such as segmentation bounding box information (e.g., object-background differentiation data, skin-tissue demarcation data, and similar object classification data) may be used for identification.

The auto-focus tool may use the identified ROI location to associate a corresponding area or ROI in each of the other images in the set of medical images. The areas or ROIs in the other images may be identified using the image characteristics described above. As one example, a bounding box area of an identified ROI in a first mammography image may be used to identify the same bounding box area in a second mammography image of the same view position. As another example, a bounding box area of an identified ROI in a first mammography image may be used to set the viewing plane, set the display field of view, or set the size of an MRI image; the location may correspond to the selected ROI in a first mammography image and may be oriented and scaled according to the position, size, and location of the selected ROI.

After identifying the corresponding ROIs in the other images, the auto-focus tool may orient the set of medical images such that the identified ROI is prominently displayed in the set of medical images. For example, in each image in the set of medical images, the auto-focus tool may pan to, zoom in on, and/or center (within the respective viewport for the image) an area of the image corresponding to the identified ROI. As such, the auto-focus tool serves as a single tool that replaces (or minimizes) the need for several other tools (e.g., pan tool, zoom tool, centering/alignment tools, scroll tool, orientation tool) and enables healthcare professionals to quickly and efficiently focus on, for example, a patient's breast or a region within the patient's breast. These capabilities of the auto-focus tool improve image review workflow and decrease the fatigue of healthcare professionals (due to decreased mouse clicks and movements) during image review.

Accordingly, the present disclosure provides a plurality of technical benefits including but not limited to: automating ROI correlation in images having the same and/or different imaging modalities, consolidating multiple image manipulation tools into a single tool, enabling multimodal image review in a single system, improving image review workflow, and decreasing healthcare professional fatigue during image review, among others.

FIG. 1 illustrates an example input processing system for an auto-focus tool for multimodality image review. Although examples in FIG. 1 and subsequent figures will be discussed in the context of image content, the examples are equally applicable to other types of content, such as video content and text content. In some examples, one or more data and components described in FIG. 1 (or the functionality thereof) may be distributed across multiple devices. In other examples, a single device may comprise the data and components described in FIG. 1.

In FIG. 1, input processing system 100 comprises content selection component 102, content presentation component 104, ROI selection component 106, and auto-focus tool 108. One of skill in the art will appreciate that the scale of input processing system 100 may vary and may include additional or fewer components than those described in FIG. 1. As one example, input processing system 100 may comprise one or more additional content selection components 102, each of which may correspond to a different imaging system or imaging modality. As another example, the functionality of ROI selection component 106 may be integrated into auto-focus tool 108.

In examples, input processing system 100 may represent a content review and manipulation system, such as a medical image review system. Input processing system 100 may be implemented in a secure computing environment comprising sensitive or private information, such as a healthcare facility (e.g., a hospital, an imaging and radiology center, an urgent care facility, a medical clinic or medical offices, an outpatient surgical facility, or a physical rehabilitation center). Alternatively, one or more components of input processing system 100 may be implemented in a computing environment external to the secure computing environment.

Content selection component 102 may be configured to enable content to be selected from one or more data sources. For example, content selection component 102 may have access to multiple data stores comprising image data of a medical imaging technology, such as picture archiving and communication system (PACS) or radiology information system (RIS). A user, such as a healthcare professional, may use content selection component 102 to select images (and associated image content) of one or more imaging modalities, such as mammography, ultrasound, and MRI. The images may be selected using a user interface provided by content selection component 102. The user interface may enable the user to select images by various criterion, such as patient name/identifier, imaging modality type, image creation/modification date, image collection/generation location, etc.

Content presentation component 104 may be configured to present selected content to a user. For example, content presentation component 104 may enable a user to select or define a content presentation style or layout, such as a hanging protocol, using the user interface (or a separate user interface). Alternatively, a default content presentation style or layout may be applied to the content presentation style or layout. Based on the selected or applied content presentation style or layout, content presentation component 104 may arrange the selected content into one or more viewports. For instance, a patient's current mammography image may be arranged into a leftmost viewport, the patient's mammography image from a patient visit one year ago may be arranged into a center viewport, and the patient's mammography image from a patient visit two years ago may be arranged into a rightmost viewport. In examples, the images in each viewport may be manipulated independently from the other presented viewports. That is, a user may manipulate (e.g., orient, pan, scroll, apply window level, annotate, remove, or otherwise modify) an image in a first presented viewport without affecting images in other presented viewports.

ROI selection component 106 may be configured to enable a user to select an area or point in the presented content. For example, the user interface may comprise one or more area selection mechanisms for identifying an ROI within an image presented in a viewport. In some aspects, an area selection mechanism may be automated to automatically identify an ROI within an image. For instance, ROI selection component 106 may implement an image recognition algorithm or model. The algorithm/model may enable processing an image (and other content) to identify and analyze objects and attributes of the image. Examples of the algorithm/model may include convolution neural networks (CNN), bag-of-words, logistic regression, support vector machines (SVM), and k-nearest-neighbor (KNN). In examples, the algorithm/model may automatically overlay a segmentation bounding box (or a similar area selection utility) on an image. The bounding box may identify and/or delineate one or more objects in the image. As a specific example, in a mammography image, the bounding box may encompass a patient's breast such that the breast is delineated from the background of the image or from other content (e.g., annotations or embedded data) within the image. In other aspects, an area selection mechanism may be used by a user to manually select an ROI within an image. For instance, ROI selection component 106 may provide an input tool, such as a cursor or pointer object, an enclosure tool (e.g., elliptical ROI, rectangular ROI, freehand ROI), or a highlighting tool. A user may use the input tool to specify a point or region of the image.

Auto-focus tool 108 may be configured to enable multimodality image review of the presented content. For example, the user interface may comprise auto-focus tool 108 or may enable a means for activating auto-focus tool 108 (e.g., a command button, a menu item, a keyboard sequence, a voice command, an eye-gaze command). A user may activate auto-focus tool 108 after selection of an ROI in presented content. Alternatively, the user may activate auto-focus tool 108 prior to selection of the ROI. For instance, activation of auto-focus tool 108 may cause the activation of ROI selection component 106.

Upon selection of auto-focus tool 108 and/or the ROI, auto-focus tool 108 may identify the location of the ROI within the content from which the ROI was selected (“source content”). The process for identifying the location of the ROI may differ based on the type of imaging modality for the source content. As one example, for a mammography image, auto-focus tool 108 may identify the location of an ROI based on image view information, such as laterality (e.g., right breast or left breast) and view position (e.g., MLO, CC). For instance, an area of the mammography image corresponding to the upper region of a MLO view of a right breast may be selected as the ROI.

As another example, for an MM image, auto-focus tool 108 may identify the location of an ROI based on DICOM header information, which may include image position and slice information, for the MRI image. For instance, the header information for the MM image may contain image orientation and image position information that can be used to determine spatial coordinates of objects, boundaries, and/or landmarks within the image using the Reference Coordinate System (RCS). Using the information embedded in the header spatial coordinates corresponding to the ROI may be identified. In some instances, the spatial coordinates for the ROI may be converted to or defined in terms of coordinates relative to the patient, such as right/left, anterior/posterior, and feet/head positions.

As yet another example, for an ultrasound image, auto-focus tool 108 may identify the location of an ROI based on DICOM header information and/or pixel data embedded in the image. For instance, the header information for the ultrasound image may indicate image laterality (e.g., right or left) and/or the header information of objects associated with the ultrasound image, such as Grayscale Softcopy Presentation State (GSPS) objects, may contain information embedded in the annotation that captures the location. Alternatively, the annotations may be embedded in the pixel data of the ultrasound image and describes the location of the ROI. In at least one example, auto-focus tool 108 may also use movement data for the ultrasound device to identify the location of an ROI. For instance, position and movement data for an ultrasound transducer or probe may be recorded during the ultrasound imaging.

Auto-focus tool 108 may use the identified location of the ROI in the source content to identify a corresponding area in the other presented content. The method for identifying the corresponding areas may differ based on the type of imaging modality of the other presented content. In some aspects, when the source content and the other presented content is the same imaging modality type, auto-focus tool 108 may map the image view information, spatial coordinate information, header information, and/or pixel data of the ROI to the corresponding area in the other presented content. As one example, in a first mammography image (source content), an identified ROI may be in the upper inner quadrant of a CC view of a patient's right breast. Accordingly, in a second mammography image (other presented content), auto-focus tool 108 may identify the upper inner quadrant of a CC view of the patient's right breast. As another example, in a first MRI image for a patient (source content), the slice information (e.g., middle slice) and spatial coordinate information of an identified ROI may be mapped to the same image slice and coordinates in a second MRI image for the patient. As yet another example, in a first ultrasound image for a patient (source content), the laterality information associated with an identified ROI may be used to identify a second ultrasound image of the same laterality.

In other aspects, when the source content and the other presented content is a different imaging modality type, auto-focus tool 108 may convert the header information of the image, such as image view information or spatial information, or information contained in objects associated with the image or embedded in the pixel data into information corresponding to the other presented content type. As one example, auto-focus tool 108 may convert the image view position or laterality information associated with an ROI in a mammography image (source content) into a slice location or set of spatial coordinates approximating the corresponding location of the ROI in an MM image (other presented content). For instance, a mammography image of the CC view position and right laterality may correspond to the middle portion of the right breast from a bilateral breast MM image. As another example, the location of the ROI in the mammography image or information contained in associated objects (e.g., GSPS, CAD SR) may be mapped to set of MRI coordinates corresponding to the location of the ROI. Determining the spatial coordinates of the ROI or determining the laterality, quadrant, or region of the breast where the ROI resides, may include the use of one or more determination mechanisms, such as a rule set, decision logic, an ML component (e.g., algorithm/model), a mapping algorithm, etc.

As another example, auto-focus tool 108 may convert the image view information of an ROI in a mammography image (source content) into view laterality information identifying an ultrasound image (other presented content). As neither the mammography image nor the ultrasound image may comprise spatial coordinate information, the laterality information in the image view information may be used to identify a corresponding ultrasound image (e.g., an ultrasound image of similar laterality). Identifying the corresponding ultrasound image may include the use of at least one of the determination mechanisms.

As yet another example, auto-focus tool 108 may convert the spatial coordinates of an ROI in an MRI image (source content) into laterality information identifying an ultrasound image (other presented content). For instance, the sagittal midline of a patient's body may represent a patient's origin according to the Reference Coordinate System such that positive values in the X direction, or values to the left of the midline, indicate areas in or around the patient's left breast and negative values in the X direction, or values to the right of the midline, indicate areas in or around the patient's right breast. Accordingly, a determination mechanism, such as a coordinate mapping algorithm, may be used to map/convert the spatial coordinates into a laterality determination.

After identifying the corresponding area in each of the other presented content, auto-focus tool 108 may focus the field of view of each viewport such that the identified ROI and the corresponding areas are prominently displayed. For example, in the viewport comprising the source content, focus tool 108 may orient the identified ROI such that ROI is horizontally and/or vertically centered in the viewport. The orienting may occur automatically and in real-time in response to the selection of the auto-focus tool 108 and/or the ROI. Additionally, focus tool 108 may magnify the ROI to further focus the attention of a user on a particular region of the breast (e.g., upper inner quadrant, lower outer quadrant). In the viewports of the other presented content, focus tool 108 may similarly orient the areas corresponding to the ROI in the source content. As one example, an MRI image in a viewport may be oriented such that the center slice from the right side or left side of the patient (right breast or left breast) is displayed regardless of whether the ROI in the source content is located more internal (medial) or more external (lateral) for one breast. In such an example, the center slice may serve as a general or starting focus area for the user. As another example, an MM image in a viewport may have its focus set on the upper region of the breast based on the location of the ROI in the source content.

Having described a system and process flow that may employ the techniques disclosed herein, the present disclosure will now describe one or more methods that may be performed by various aspects of the present disclosure. In aspects, method 200 may be executed by a system, such as system 100 of FIG. 1. However, method 200 is not limited to such examples. In other aspects, method 200 may be performed by a single device comprising multiple computing environments. In at least one aspect, method 200 may be executed by one or more components of a distributed network, such as a web service/distributed network service (e.g., cloud service).

FIG. 2 illustrates an example method for utilizing an auto-focus tool for multimodality image review. Example method 200 may be executed by a user, such as a healthcare provider, in a computing environment comprising sensitive or private information associated with, for example, a healthcare facility, healthcare patients, and/or healthcare personnel. The computing environment may comprise an electronic image review system, such as input processing system 100. The image review system may enable the user to retrieve images from various data sources, such as data store(s) 106. The retrieved images may represent images of various imaging modalities, such as mammography, ultrasound, MRI, etc. The user may arrange the retrieved images for viewing and manipulating based on an image viewing layout, such as a hanging protocol. The image viewing layout may provide for the simultaneous display of images of varying (or similar) imaging modalities. As a specific example, the image viewing layout may comprise four viewports (each comprising an image) arranged in a left-to-right configuration.

Example method 200 begins at operation 202, where an image focus tool is selected. In aspects, the image review system may comprise or provide access to an image focus tool, such as auto-focus tool 108. For example, the image review system may provide a user interface component (e.g., graphical user interface (GUI), command line, microphone, haptic mechanism, camera) for selecting and/or activating the image focus tool. A user using the image review system to view one or more images may use the user interface component to select and activate the image focus tool. Alternatively, the user may select and activate the image focus tool prior to accessing, retrieving, or viewing the images.

At operation 204, an ROI in a first image may be selected. In aspects, a user may use an input tool (e.g., cursor, stylus, enclosure tool, highlighting tool, voice-based tool, eye-gaze tool) provided by the image focus tool or the image review system to select one or more points or portions of an image. For instance, the user may select a point in a first image of image viewing layout comprising four images. The selected points or portions may define a ROI. As one example, when a single point in an image is selected, an area surrounding the single point may be defined as the ROI. As another example, when multiple points in an image are selected, the image focus tool may determine a centroid (or approximate center point) of the multiple points. An area surrounding the centroid may be defined as the ROI. The amount and/or shape (e.g., ellipse, rectangle, freeform) of the area used to define the ROI may be determined automatically by the image focus tool or defined manually by the user. For instance, the image focus tool may automatically overlay the image with a bounding box that encompasses the selected/determined point. The bounding box may delineate one or more objects in the image from other objects or the background of the image.

At operation 206, the location of the ROI within the first image may be identified. The process for identifying the location of the ROI may include the use of one or more determination mechanisms (e.g., a rule set, decision logic, an ML component, a mapping algorithm, image or object classifier) and may differ based on the imaging modality type of the first image. As one example, the image focus tool may use a set of data extraction rules to identify ROI in a mammography image using image view information of the mammography image, such as laterality (e.g., right or left) and view position (e.g., MLO, CC). For instance, the data extraction rules may label or otherwise designate the ROI as “CC View, Right Breast” based on the information in a Digital Imaging and Communication in Medicine (DICOM) header of the mammography image. Alternatively, the ROI may be further labeled/designated using additional area information in the image, such as “Right Breast, Upper Outer Quadrant.”

As another example, the image focus tool may use a spatial mapping function to identify ROI in an MRI image using information available in the DICOM header of an MRI image and/or associated objects, such as Image Position (Patient), Image Orientation (Patient), Pixel Spacing, Slice Thickness, and Slice Location. The spatial mapping function may define the ROI using a 3D reference coordinate system (e.g., x, y, and z coordinates) in which the boundary of the ROI is defined by multiple sets of coordinate values or defined in terms of right/left, anterior/posterior, and feet/head coordinate values that are relative to the patient (e.g., L:52.2, A:5.5, H:10.6). Alternatively, the ROI may be defined by a single set of coordinate values (e.g., voxel (50, 100, 55)) representing the center (or centroid) of the ROI.

As another example, the image focus tool may use a text recognition algorithm to identify ROI in an ultrasound image using image header information, pixel data, orientation data, and/or imaging device data, such as laterality information (e.g., right or left), embedded image information (e.g., annotations and notes), and 2D/3D transducer/probe movement data. For instance, the text recognition algorithm may generally label or otherwise designate the ROI as “Right Breast” based on text-based laterality information extracted from the DICOM header of the ultrasound image. Alternatively, the ROI may be labeled/designated based on embedded annotations (e.g., handwritten notes, burned-in text) and/or an orientation map in the image data of the ultrasound image.

At operation 208, areas corresponding to the ROI may be identified in other images. In aspects, the image focus tool may use the identified location of the ROI within the first image to identify corresponding areas in the other images presented in the image viewing layout. The process for identifying the corresponding areas in the other images may include the use of one or more of the determination mechanisms described above (and/or additional determination mechanisms and may differ based on the imaging modality type of the other images. As one example, when the imaging modality type of the first image matches the imaging modality type of a second image, a mapping function may be used to map the ROI identification information of the first image (e.g., image view information, spatial coordinate information, header information) to the same (or similar) ROI identification information of the second image. For instance, the ROI label/designation “Right Breast, Upper Outer Quadrant” for a first mammography image may be used by the mapping function to map the same laterality (e.g., right breast) and region (e.g., Upper Outer Quadrant) in a second mammography image based on the image view information for the second mammography image.

As another example, when the imaging modality type of the first image does not match the imaging modality type of a second image, a mapping function may be used to map the ROI identification information of the first image (e.g., image view information, spatial coordinate information, header information) to different, but corresponding ROI identification information of the second image. For instance, the ROI label/designation “Right Breast, Axillary Region” for a mammography image may be converted to a set of MRI spatial coordinates. The set of MRI spatial coordinates may be predefined for one or more areas in each type of mammography image. As a specific example, spatial coordinates may be predefined for the various quadrants of the breast (e.g., Upper Outer, Upper Inner, Lower Outer, Lower Inner), regions (e.g., Central, Retroareolar, Axillary), and/or laterality (e.g., right, left) of the mammography image. Accordingly, the ROI label/designation for a mammography image may be used to select the corresponding MRI spatial coordinates in an MRI image.

At operation 210, the images may be automatically focused on the ROI and corresponding areas. In aspects, the image focus tool may focus the field of view of each image in the image viewing layout such that the ROI and corresponding areas are prominently displayed. For example, after (or prior to) identifying the areas corresponding to the ROI, the image focus tool may orient the identified ROI in the first image such that ROI is horizontally and/or vertically centered in the viewport comprising the first image. The image focus tool may also (simultaneously or subsequently) orient the other images such that the areas corresponding to the ROI are horizontally and/or vertically centered in their respective viewports and may also display the areas in an orientation that is different from the original image acquisition plane, for example displaying in one or more MRI images (presented content) the axial, sagittal, and/or coronal plane to provide different perspectives of the ROI. In some examples, the image focus tool may apply some degree of scaling, magnification, and/or filtering to one or more of the images. For instance, the image focus tool may apply a 2× scaling factor to a second image and a 4× scaling factor to a third image. As should be appreciated, the automatic orientation and scaling operations of the image focus tool reduces the amount of image manipulation tools, input device clicks (and other type of selections), and input device movements required to review images. The image focus tool also enables users to review images of differing imaging modality types using the same system and image review tool. Accordingly, the image focus tool improves the image review workflow, reduces the fatigue experienced by healthcare professional fatigue during image review, and may reduce the need to learn and operate multiple image review systems and image review tools.

FIGS. 3A and 3B depict a set of multimodality images for illustrating the functionality of the auto-focus tool described herein. The set of multimodality images depict images of a patient's breast. FIG. 3A comprises images 302, 304, 306, and 308. Image 302 is a full field digital mammography (FFDM) image of a right breast. Image 304 is a tomosynthesis reconstruction mammography image of the same laterality as image 302. Image 306 is a bilateral breast MRI image acquired and displayed in the axial plane. Image 308 is a bilateral breast MRI image displayed in the sagittal plane at a location that centers the patient. In at least one example, each of images 302, 304, 306, and 308 may represent a different image and one or more of images 302, 304, 306, and 308 may be collected/generated at different times. For instance, image 302 may represent a control image collected during a current (or recent) patient visit and images 304, 306, and 308 may represent images collected during one or more previous patient visits. In response to selecting an ROI of image 302, images 304, 306, and 308 may be manipulated to focus on (e.g., pan, zoom, flip, rotate, center, reconstructed) the ROI in the respective images while image 302 remains unchanged. In another example, each image may represent the same image, but rendered and presented in a different way, for example reconstructing an MRI image acquired in the axial plane to a sagittal or coronal image.

In FIG. 3B, ROI 310 has been selected in image 302. In response to the selection of ROI 310, the auto-focus tool described herein has automatically set the focus for images 302, 304, 306, and 308. For example, in image 302, the auto-focus tool has applied bounding box 312 to encompass ROI 310 while the alignment and magnification of the image 302 remains unchanged. In image 304, the auto-focus tool has magnified the image and vertically aligned the area corresponding to the ROI with the ROI in image 302. In image 306, the auto-focus tool has panned to the area corresponding to the ROI, magnified the area corresponding to the ROI, and centered the area corresponding to the ROI in the viewport comprising image 306. In image 308, the auto-focus tool has reconstructed the MRI image to the sagittal plane, scrolled to the center slice of the breast matching the laterality of the ROI, and vertically aligned the area corresponding to the ROI with the ROI in image 302.

FIG. 4 illustrates an exemplary suitable operating environment for the automating clinical workflow decision techniques described in FIG. 1. In its most basic configuration, operating environment 400 typically includes at least one processing unit 402 and memory 404. Depending on the exact configuration and type of computing device, memory 404 (storing, instructions to perform the techniques disclosed herein) may be volatile (such as RAM), nonvolatile (such as ROM, flash memory, etc.), or some combination of the two. This most basic configuration is illustrated in FIG. 4 by dashed line 406. Further, environment 400 may also include storage devices (removable, 408, and/or non-removable, 410) including, but not limited to, magnetic or optical disks or tape. Similarly, environment 400 may also have input device(s) 414 such as keyboard, mouse, pen, voice input, etc. and/or output device(s) 416 such as a display, speakers, printer, etc. Also included in the environment may be one or more communication connections 412, such as LAN, WAN, point to point, etc. In embodiments, the connections may be operable to facility point-to-point communications, connection-oriented communications, connectionless communications, etc.

Operating environment 400 typically includes at least some form of computer readable media. Computer readable media can be any available media that can be accessed by processing unit 402 or other devices comprising the operating environment. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium which can be used to store the desired information. Computer storage media does not include communication media.

Communication media embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, microwave, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

The operating environment 400 may be a single computer operating in a networked environment using logical connections to one or more remote computers. The remote computer may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above as well as others not so mentioned. The logical connections may include any method supported by available communications media. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

FIG. 5 illustrates an overview of an example system for an auto-focus tool for multimodality image review. Example system 500 as presented is a combination of interdependent components that interact to form an integrated system. System 500 may comprise hardware components and/or software components implemented on and/or executed by hardware components. System 500 may provide one or more operating environments for software components to execute according to operating constraints, resources, and facilities of system 500. In some examples, the operating environment(s) and/or software components may be provided by a single processing device, as depicted in FIG. 4. In other examples, the operating environment(s) and software components may be distributed across multiple devices. For instance, input may be entered on a user device and information may be processed or accessed using other devices in a network, such as one or more network devices and/or server devices.

In aspects, system 500 may represent a computing environment comprising sensitive or private information associated with, for example, a healthcare facility, healthcare patients, and/or healthcare personnel. Although specific reference to a healthcare environment is described herein, it is contemplated that the techniques of the present disclosure may be practiced in other environments. For example, system 500 may represent a software development environment or an alternative environment that does not comprise sensitive or private medical information.

In FIG. 5, system 500 comprises computing devices 502A, 502B, 502C, and 502D (collectively “computing device(s) 502”), application 504, data store(s) 506, and network 508. One of skill in the art will appreciate that the scale of systems such as system 500 may vary and may include more or fewer components than those described in FIG. 5. As one example, the functionality and/or one or more components of application 504 may be implemented on a server device, web-based device, or in a cloud computing environment. As another example, data store(s) 508 may be integrated into computing device(s) 502.

Computing device(s) 502 may be configured to collect, manipulate, and/or display input data from one or more users or devices. For example, computing device(s) 502 may collect input data from a healthcare professional, medical equipment (e.g., imaging devices, treatment devices, monitoring devices), medical workstations, data storage locations, etc. The input data may correspond to user interaction with one or more software applications or services implemented by, or accessible to, user device(s) 502. The input data may include, for example, voice input, touch input, text-based input, gesture input, video input, and/or image input. The input data may be detected/collected using one or more sensor components of user device(s) 502. Examples of sensors include microphones, touch-based sensors, geolocation sensors, accelerometers, optical/magnetic sensors, gyroscopes, keyboards, and pointing/selection tools. Examples of user device(s) 502 may include, but are not limited to, personal computers (PCs), medical workstations, server devices, cloud-based devices, mobile devices (e.g., smartphones, tablets, laptops, personal digital assistants (PDAs)), and wearable devices (e.g., smart watches, smart eyewear, fitness trackers, smart clothing, body-mounted devices, head-mounted displays).

Computing device(s) 502 may comprise or otherwise have access to application(s) 504. Application(s) 504 may enable users to access and/or interact with one or more types of content, such as images, text, audio, images, video, and animation. As one example, application(s) 504 may represent a multimodality image processing and/or review service that enables healthcare professionals to review medical images. Although specific reference to an image processing application/service, alternative implementations are contemplated. For example, application(s) 504 may represent word processing applications, spreadsheet application, presentation applications, document-reader software, social media software/platforms, search engines, media software/platforms, multimedia player software, content design software/tools, and database applications.

Application(s) 504 may comprise or have access to one or more data stores, such as data store(s) 506. Data store(s) 506 may comprise a corpus of content of various types (e.g., images, videos, documents, files, records). For example, data store(s) 506 may include image types, such as mammography images, MRI images, ultrasound images, etc. Data store(s) 506 may be stored and accessed locally on computing device(s) 502 or stored and accessed remotely via network 508. Examples of network 508 include, but are not limited to, personal area networks (PANs), local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). Application(s) 504 may retrieve content from data store(s) 506. Application(s) 504 may present the content using one or more display devices or components of Computing device(s) 502. In a particular example, application(s) 504 may present the content according to a hanging protocol or a similar content display format. The hanging protocol may provide for displaying a sequence of content items, such as images, in respective viewports of the display device or component.

In some examples, application(s) 504 implements or has access to an auto-focus tool (not pictured) for reviewing presented content. The auto-focus tool may provide for automatically orienting the presented content in accordance with a user-selected area within the content. For example, a healthcare professional may select an ROI in one image of a set of presented images that includes mammography images, MRI images, and ultrasound images. The auto-focus tool may orient each of the presented images such that the identified ROI is prominently displayed in each of the images. Accordingly, the auto-focus tool may enable the healthcare professional to automatically pan to, zoom in on, set the orientation, and/or center and align (within the respective viewport for the content) an area of the content corresponding to the identified ROI in various content items.

The embodiments described herein may be employed using software, hardware, or a combination of software and hardware to implement and perform the systems and methods disclosed herein. Although specific devices have been recited throughout the disclosure as performing specific functions, one of skill in the art will appreciate that these devices are provided for illustrative purposes, and other devices may be employed to perform the functionality disclosed herein without departing from the scope of the disclosure.

This disclosure describes some embodiments of the present technology with reference to the accompanying drawings, in which only some of the possible embodiments were shown. Other aspects may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments were provided so that this disclosure was thorough and complete and fully conveyed the scope of the possible embodiments to those skilled in the art.

Although specific embodiments are described herein, the scope of the technology is not limited to those specific embodiments. One skilled in the art will recognize other embodiments or improvements that are within the scope and spirit of the present technology. Therefore, the specific structure, acts, or media are disclosed only as illustrative embodiments. The scope of the technology is defined by the following claims and any equivalents therein.

Claims

1. A system comprising: a processor; andmemory coupled to the processor, the memory comprising computer executable instructions that, when executed, perform a method comprising: receiving a selection of a region of interest (ROI) in a first image of a plurality of images;identifying, using an auto-focus tool, a location of the ROI within the first image based on at least one of spacing of image pixel data, thickness of a slice image, and location of the slice image;converting, via a mapping function of the auto-focus tool, first ROI identification information of the first image to second ROI identification of at least a second image of the plurality of images, wherein the first ROI identification information and the second ROI identification information are a different type, the first ROI identification information includes image view information and the second ROI identification information includes image header information;identifying, using the auto-focus tool, an area corresponding to the location of the ROI in at least the second image of the plurality of images, wherein the first image and the second image are different imaging modality types and an automated determination mechanism is used to identify the area corresponding to the location of the ROI; andcausing the auto-focus tool to automatically: focus a first field of view on the ROI in the first image; andfocus a second field of view on the area corresponding to the location of the ROI in the second image.

2. The system of claim 1, wherein the system is an electronic image review system for reviewing medical images within a healthcare environment.

3. The system of claim 1, wherein focusing the first field of view on the ROI in the first image comprises centering the ROI within the first field of view and scaling the ROI to increase or decrease a size of the ROI within the first field of view.

4. The system of claim 1, wherein the imaging modality types include at least two of: mammography, MRI, or ultrasound.

5. The system of claim 1, wherein the first image is generated during a current patient visit for a patient and the second image was generated during a previous patient visit for the patient.

6. The system of claim 1, wherein the plurality of images is arranged for viewing based on an image viewing layout specified by a user, the image viewing layout enabling the plurality of images to be concurrently presented to a user.

7. The system of claim 1, wherein receiving the selection of the ROI comprises: receiving a selection of a point in the first image; anddefining an area surrounding the point as the ROI.

8. The system of claim 7, wherein, in response to receiving the selection of the point in the first image, a bounding box comprising at least a portion of the ROI is applied to the image.

9. The system of claim 8, wherein the bounding box is automatically applied such that at least a first object in the first image is delineated from at least a second object in the first image.

10. The system of claim 1, wherein receiving the selection of the ROI comprises: receiving a selection of a plurality of points in the first image;determining a centroid of the plurality of points; anddefining an area surrounding the centroid as the ROI.

11. The system of claim 1, wherein the automated determination mechanism is at least one of: a rule set, a mapping algorithm, or an image or object classifier.

12. The system of claim 1, wherein a process for identifying the location of the ROI within the first image is based on the imaging modality type of the first image.

13. The system of claim 12, wherein the process for identifying the location of the ROI within the first image includes the use of at least one of: image view information, spatial coordinate information, image header information, or image orientation data.

14. The system of claim 1, wherein, when the first image and a third image of the plurality of images are a same imaging modality type, the mapping function is used to map the first ROI identification information of the first image to third ROI identification information of the third image, wherein the first ROI identification information and the third ROI identification information are a same type.

15. The system of claim 1, wherein focusing the first field of view on the ROI in the first image comprises orienting the first image such that the ROI is at least one of horizontally or vertically centered in a viewport comprising the first image.

16. The system of claim 1, wherein focusing the second field of view on the area corresponding to the location of the ROI in the second image comprises applying a scaling factor to the area corresponding to the location of the ROI.

17. A method comprising: receiving, at an image review device, a selection of a region of interest (ROI) in a first image of a plurality of images;identifying, using an auto-focus tool, a location of the ROI within the first image based on at least one of spacing of image pixel data, thickness of a slice image, and location of the slice image;converting, via a mapping function of the auto-focus tool, first ROI identification information of the first image to second ROI identification of at least a second image of the plurality of images, wherein the first ROI identification information and the second ROI identification information are a different type, the first ROI identification information includes image view information and the second ROI identification information includes image header information;identifying, using the auto-focus tool, an area corresponding to the location of the ROI in at least the second image of the plurality of images, wherein the first image and the second image are different imaging modality types and an automated determination mechanism is used to identify the area corresponding to the location of the ROI; andcausing the auto-focus tool to automatically: focus a first field of view on the ROI in the first image; andfocus a second field of view on the area corresponding to the location of the ROI in the second image.

18. The method of claim 1, wherein the plurality of images comprises at least a mammography image, an MRI image, and an ultrasound image.

19. A computing device comprising: a processor; andan image auto-focus tool configured to: receive a selection of a region of interest (ROI) in a first image of a plurality of images;identify a location of the ROI within the first image based on at least one of spacing of image pixel data, thickness of a slice image, and location of the slice image;convert, via a mapping function of the auto-focus tool, first ROI identification information of the first image to second ROI identification of at least a second image of the plurality of images, wherein the first ROI identification information and the second ROI identification information are a different type, the first ROI identification information includes image view information and the second ROI identification information includes image header information;identify an area corresponding to the location of the ROI in at least the second image of the plurality of images, wherein the first image and the second image are different imaging modality types and an automated determination mechanism is used to identify the area corresponding to the location of the ROI;focus a first field of view on the ROI in the first image; andfocus a second field of view on the area corresponding to the location of the ROI in the second image, wherein focusing the second field of view comprises at least one of scaling the second image or panning the second image.