Patent Application
20230267618
Embodiments of the subject matter disclosed herein relate to ultrasound imaging, and more particularly, to an automated, ultrasound-based pelvic floor examination.
Medical ultrasound is an imaging modality that employs ultrasound waves to probe the internal structures of a body of a patient and produce a corresponding image. For example, an ultrasound probe comprising a plurality of transducer elements emits ultrasonic pulses which reflect or echo, refract, or are absorbed by structures in the body. The ultrasound probe then receives reflected echoes, which are processed into an image. Ultrasound images of the internal structures may be saved for later analysis by a clinician to aid in diagnosis and/or displayed on a display device in real time or near real time.
In an embodiment, a method includes identifying a view plane of interest based on one or more 3D ultrasound images, obtaining a view plane image including the view plane of interest from a 3D volume of ultrasound data of a patient, where the one or more 3D ultrasound images are generated from the 3D volume of ultrasound data, segmenting an anatomical region of interest (ROI) within the view plane image to generate a contour of the anatomical ROI, and displaying the contour on the view plane image.
The above advantages and other advantages, and features of the present description will be readily apparent from the following Detailed Description when taken alone or in connection with the accompanying drawings. It should be understood that the summary above is provided to introduce in simplified form a selection of concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure.
Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:
Pelvic floor examinations using ultrasound may be useful to evaluate the health of the pelvic floor, including but not limited to the bladder, levator ani muscle, urethra, and vagina. Ultrasound-based pelvic floor examinations may help determine the integrity of pelvic muscles and ascertain the necessity of corrective actions including surgical interventions. A complete pelvic floor examination of a patient may include a series of dynamic exams with both 2D and 3D acquisitions, which are highly dependent on the patient's involvement (e.g., commanded muscle maneuvers by the patient) and operator expertise. For example, one or more 3D renderings may be acquired to view the anatomical region of interest, and then a series of 3D renderings may be acquired while the patient is asked to push down and/or contract the pelvic floor muscles. Further, the examination includes several measurements on the acquired images. Accordingly, standard pelvic floor examinations require highly trained operators and may be time-consuming and mentally burdensome for both the patients and the operators.
For example, the measurements may include dimensions of the levator hiatus (e.g., area, and lateral as well as anterior-posterior diameters), which is an opening in the pelvic floor defined by the levator ani muscle and the inferior pubic rami. The dimensions of the levator hiatus may be measured during muscle contraction and expansion (e.g., during Valsalva) in order to assess the structural integrity of the levator ani muscle, possible pelvic organ prolapse, and proper functioning and strength of the pelvic floor muscles.
During a standard pelvic floor examination, an ultrasound operator may hold an ultrasound probe on a patient in a given position while the patient performs breath holds, contracts and/or pushes down the pelvic floor muscles, or performs other activities. As such, image quality may vary from exam to exam. Further, the presentation of the imaged pelvic floor muscles can vary from patient to patient, and thus highly trained operators may be necessary to ensure the correct image slice (e.g., showing the plane of minimal hiatal dimension) is selected for analysis (from among a plurality of image slices acquired as part of a 3D volume of ultrasound data). The operator may identify the appropriate initial volume image (e.g., an image frame from the 3D volume), identify the view plane of interest (e.g., the plane of minimal hiatal dimension) in the selected volume image, annotate the view plane of interest with the levator hiatus contour, and perform various measurements on the levator hiatus, such as area, circumference, lateral diameter, and anterior-posterior diameter. Each step may be time-consuming, which may be further exacerbated if low image quality results in the operator having to reacquire certain images or volumes of data.
Thus, according to embodiments disclosed herein, artificial-intelligence based methods may be applied to automate aspects of the pelvic floor examination. As explained in more detail below, a view plane of interest (e.g., the plane including the minimal hiatal dimension) may be automatically identified in a 3D ultrasound image. Once the view plane of interest is identified, a set of deep learning models may be deployed to automatically segment the levator hiatus boundary, as well as mark the two diameters (e.g., the lateral and anterior-posterior diameters) on the plane of the minimal hiatal dimension for the ascertainment of the various measurements and subsequently health/integrity of the levator muscles. This process may be repeated as the patient performs breath holds, contracts the pelvic floor muscles, etc. In doing so, clinical outcomes may be improved by increased accuracy and robustness of the pelvic exams, operator and patient experience may be improved due to reduced examination and analysis time, and reliance on highly trained operators may be reduced.
While the disclosure presented herein is directed to a pelvic floor examination where the plane of minimal hiatal dimension is identified within a volume of ultrasound data and the levator hiatus is segmented using a set of deep learning models to initially identify and then measure aspects of the levator hiatus, the framework provided herein may be applied to automate other medical imaging exams that rely on identification of a slice from a volume of data and/or segmentation of an anatomical region of interest.
An example ultrasound system including an ultrasound probe, a display device, and an imaging processing system are shown in
Referring to
After the elements 104 of the probe 106 emit pulsed ultrasonic signals into a body (of a patient), the pulsed ultrasonic signals reflect from structures within an interior of the body, like blood cells or muscular tissue, to produce echoes that return to the elements 104. The echoes are converted into electrical signals, or ultrasound data, by the elements 104 and the electrical signals are received by a receiver 108. The electrical signals representing the received echoes are passed through a receive beamformer 110 that outputs ultrasound data.
The echo signals produced by transmit operation reflect from structures located at successive ranges along the transmitted ultrasonic beam. The echo signals are sensed separately by each transducer element and a sample of the echo signal magnitude at a particular point in time represents the amount of reflection occurring at a specific range. Due to the differences in the propagation paths between a reflecting point P and each element, however, these echo signals are not detected simultaneously. Receiver 108 amplifies the separate echo signals, imparts a calculated receive time delay to each, and sums them to provide a single echo signal which approximately indicates the total ultrasonic energy reflected from point P located at range R along the ultrasonic beam oriented at the angle θ.
The time delay of each receive channel continuously changes during reception of the echo to provide dynamic focusing of the received beam at the range R from which the echo signal is assumed to emanate based on an assumed sound speed for the medium.
Under direction of processor 116, the receiver 108 provides time delays during the scan such that steering of receiver 108 tracks the direction θ of the beam steered by the transmitter and samples the echo signals at a succession of ranges R so as to provide the time delays and phase shifts to dynamically focus at points P along the beam. Thus, each emission of an ultrasonic pulse waveform results in acquisition of a series of data points which represent the amount of reflected sound from a corresponding series of points P located along the ultrasonic beam.
According to some embodiments, the probe 106 may contain electronic circuitry to do all or part of the transmit beamforming and/or the receive beamforming. For example, all or part of the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be situated within the probe 106. The terms “scan” or “scanning” may also be used in this disclosure to refer to acquiring data through the process of transmitting and receiving ultrasonic signals. The term “data” may be used in this disclosure to refer to either one or more datasets acquired with an ultrasound imaging system. A user interface 115 may be used to control operation of the ultrasound imaging system 100, including to control the input of patient data (e.g., patient medical history), to change a scanning or display parameter, to initiate a probe repolarization sequence, and the like. The user interface 115 may include one or more of the following: a rotary element, a mouse, a keyboard, a trackball, hard keys linked to specific actions, soft keys that may be configured to control different functions, and a graphical user interface displayed on a display device 118.
The ultrasound imaging system 100 also includes a processor 116 to control the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110. The processor 116 is in electronic communication (e.g., communicatively connected) with the probe 106. For purposes of this disclosure, the term “electronic communication” may be defined to include both wired and wireless communications. The processor 116 may control the probe 106 to acquire data according to instructions stored on a memory of the processor, and/or memory 120. The processor 116 controls which of the elements 104 are active and the shape of a beam emitted from the probe 106. The processor 116 is also in electronic communication with the display device 118, and the processor 116 may process the data (e.g., ultrasound data) into images for display on the display device 118. The processor 116 may include a central processor (CPU), according to an embodiment. According to other embodiments, the processor 116 may include other electronic components capable of carrying out processing functions, such as a digital signal processor, a field-programmable gate array (FPGA), or a graphic board. According to other embodiments, the processor 116 may include multiple electronic components capable of carrying out processing functions. For example, the processor 116 may include two or more electronic components selected from a list of electronic components including: a central processor, a digital signal processor, a field-programmable gate array, and a graphic board. According to another embodiment, the processor 116 may also include a complex demodulator (not shown) that demodulates the real RF (radio-frequency) data and generates complex data. In another embodiment, the demodulation can be carried out earlier in the processing chain. The processor 116 is adapted to perform one or more processing operations according to a plurality of selectable ultrasound modalities on the data. In one example, the data may be processed in real-time during a scanning session as the echo signals are received by receiver 108 and transmitted to processor 116. For the purposes of this disclosure, the term “real-time” is defined to include a procedure that is performed without any intentional delay. For example, an embodiment may acquire images at a real-time rate of 7-20 frames/sec and/or may acquire volumetric data at a suitable volume rate. The ultrasound imaging system 100 may acquire 2D data of one or more planes at a significantly faster rate. However, it should be understood that the real-time frame-rate may be dependent on the length of time that it takes to acquire each frame of data for display. Accordingly, when acquiring a relatively large amount of data, the real-time frame-rate may be slower. Thus, some embodiments may have real-time frame-rates or volume rates that are considerably faster than 20 frames/sec (or volumes/sec) while other embodiments may have real-time frame-rates or volume rates slower than 7 frames/sec (or volumes/sec). The data may be stored temporarily in a buffer (not shown) during a scanning session and processed in less than real-time in a live or off-line operation. Some embodiments of the invention may include multiple processors (not shown) to handle the processing tasks that are handled by processor 116 according to the exemplary embodiment described hereinabove. For example, a first processor may be utilized to demodulate and decimate the RF signal while a second processor may be used to further process the data, for example by augmenting the data as described further herein, prior to displaying an image. It should be appreciated that other embodiments may use a different arrangement of processors.
The ultrasound imaging system 100 may continuously acquire data at a frame-rate or volume rate of, for example, 10 Hz to 30 Hz (e.g., 10 to 30 frames per second). Images generated from the data (which may be 2D images or 3D renderings) may be refreshed at a similar frame-rate on display device 118. Other embodiments may acquire and display data at different rates. For example, some embodiments may acquire data at a frame-rate or volume rate of less than 10 Hz or greater than 30 Hz depending on the size of the frame and the intended application. A memory 120 is included for storing processed frames or volumes of acquired data. In an exemplary embodiment, the memory 120 is of sufficient capacity to store at least several seconds' worth of frames or volumes of ultrasound data. The frames or volumes of data are stored in a manner to facilitate retrieval thereof according to its order or time of acquisition. The memory 120 may comprise any known data storage medium.
In various embodiments of the present invention, data may be processed in different mode-related modules by the processor 116 (e.g., B-mode, Color Doppler, M-mode, Color M-mode, spectral Doppler, Elastography, TVI, strain, strain rate, and the like) to form 2D or 3D data. For example, one or more modules may generate B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, Elastography, TVI, strain, strain rate, and combinations thereof, and the like. As one example, the one or more modules may process color Doppler data, which may include traditional color flow Doppler, power Doppler, HD flow, and the like. The image lines, frames, and/or volumes are stored in memory and may include timing information indicating a time at which the image lines, frames, and/or volumes were stored in memory. The modules may include, for example, a scan conversion module to perform scan conversion operations to convert the acquired data from beam space coordinates to display space coordinates. A video processor module may be provided that reads the acquired images from a memory and displays an image in real time while a procedure (e.g., ultrasound imaging) is being performed on a patient. The video processor module may include a separate image memory, and the ultrasound images may be written to the image memory in order to be read and displayed by display device 118.
In various embodiments of the present disclosure, one or more components of ultrasound imaging system 100 may be included in a portable, handheld ultrasound imaging device. For example, display device 118 and user interface 115 may be integrated into an exterior surface of the handheld ultrasound imaging device, which may further contain processor 116 and memory 120. Probe 106 may comprise a handheld probe in electronic communication with the handheld ultrasound imaging device to collect raw ultrasound data. Transmit beamformer 101, transmitter 102, receiver 108, and receive beamformer 110 may be included in the same or different portions of the ultrasound imaging system 100. For example, transmit beamformer 101, transmitter 102, receiver 108, and receive beamformer 110 may be included in the handheld ultrasound imaging device, the probe, and combinations thereof.
After performing a two-dimensional or three-dimensional ultrasound scan, a block of data (which may be two-dimensional or three-dimensional) comprising scan lines and their samples is generated. After back-end filters are applied, a process known as scan conversion is performed to transform the data block into a displayable bitmap image with additional scan information such as depths, angles of each scan line, and so on. During scan conversion, an interpolation technique is applied to fill missing holes (i.e., pixels) in the resulting image. These missing pixels occur because each element of the block should typically cover many pixels in the resulting image. For example, in current ultrasound imaging systems, a bicubic interpolation is applied which leverages neighboring elements of the block. As a result, if the block is relatively small in comparison to the size of the bitmap image, the scan-converted image will include areas of less than optimal or low resolution, especially for areas of greater depth.
Referring to
Image processing system 202 includes a processor 204 configured to execute machine readable instructions stored in non-transitory memory 206. Processor 204 may be single core or multi-core, and the programs executed thereon may be configured for parallel or distributed processing. In some embodiments, the processor 204 may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and/or configured for coordinated processing. In some embodiments, one or more aspects of the processor 204 may be virtualized and executed by remotely-accessible networked computing devices configured in a cloud computing configuration.
Non-transitory memory 206 may store a view plane model 207, a segmentation model 208, a contour refinement model 210, ultrasound image data 212, and a training module 214.
Each of the view plane model 207, the segmentation model 208, and the contour refinement model 210 may include one or more machine learning models, such as deep learning networks, comprising a plurality of weights and biases, activation functions, loss functions, gradient descent algorithms, and instructions for implementing the one or more deep neural networks to process input ultrasound images. Each of the view plane model 207, the segmentation model 208, and the contour refinement model 210 may include trained and/or untrained neural networks and may further include training routines, or parameters (e.g., weights and biases), associated with one or more neural network models stored therein.
The view plane model 207 may thus include one or more machine learning models configured to process input ultrasound images (which may include 3D renderings) to identify a view plane of interest within a volume of ultrasound data. As will be explained in more detail below, during a pelvic examination, the view plane of interest may be a view plane that includes a minimal hiatal dimension (MHD), referred to as an MHD plane. The view plane model 207 may receive selected frames of a volume of ultrasound data and process the selected frames to identify the MHD plane within the volume of ultrasound data. The view plane model 207 may include a hybrid neural network (e.g., convolutional neural network (CNN)) architecture including 3D convolution layers, a flattening layer, and a 2D neural network (e.g., a CNN such as a UNet). The view plane model 207 may output a 2D segmentation mask that identifies the location of the view plane of interest within the volume of ultrasound data.
The segmentation model 208 may include one or more machine learning models, such as a neural network, configured to process an input ultrasound image to identify an anatomical ROI in the input ultrasound image. For example, as explained in more detail below, the segmentation model 208 may be deployed during a pelvic examination in order to identify the levator hiatus in an input ultrasound image. In some examples, the input ultrasound image may be an image including the view plane identified by the view plane model 207, e.g., the MHD plane. The segmentation model 208 may process the input ultrasound image to output a segmentation (e.g., mask) identifying the anatomical ROI in the input ultrasound image. However, some anatomical features, such as the levator hiatus, may be difficult to accurately identify in a precise manner given patient to patient variability in the size and shape of the anatomical feature. Thus, the initial segmentation output by the segmentation model 208 is used as a guide to map a predefined template to the anatomical ROI in the given ultrasound image to form an adjusted segmentation template that can be entered as input to the contour refinement model 210.
The contour refinement model 210 may include one or more machine learning models, such as a neural network, configured to process the input ultrasound image (e.g., the same image used as input to the segmentation model) and the adjusted segmentation template to identify the anatomical ROI in the input ultrasound image more accurately. The identified anatomical ROI (e.g., the segmentation output by the contour refinement model 210) may be used to generate a border/contour of the anatomical ROI that may then be assessed to measure aspects of the anatomical ROI.
The ultrasound image data 212 may include 2D images and/or 3D volumetric data, from which 3D renderings and 2D images/slices may be generated, captured by the ultrasound imaging system 100 of
The training module 214 may comprise instructions for training one or more of the deep neural networks stored in the view plane model 207, the segmentation model 208, and/or the contour refinement model 210. In some embodiments, training module 214 includes instructions for implementing one or more gradient descent algorithms, applying one or more loss functions, and/or training routines, for use in adjusting parameters of one or more deep neural networks of the view plane model 207, the segmentation model 208, and/or the contour refinement model 210. In some embodiments, training module 214 includes instructions for intelligently selecting training data pairs from ultrasound image data 212. In some embodiments, training data pairs comprise pairs of input data and ground truth data. The input data may include one or more ultrasound images. For example, to train the view plane model 207, the input data may include, for each pair of input data and ground truth data, a set of 3D ultrasound images (e.g., three or more 3D ultrasound images, such as nine 3D ultrasound images) selected from a volume of ultrasound data. For each set of 3D ultrasound images, the corresponding ground truth data used to train the view plane model 207 may include a segmentation mask (e.g., generated by an expert) indicating the location, within the volume of ultrasound data, of the view plane of interest. The view plane model 207 may be updated based on a loss function between each segmentation mask output by the view plane model and the corresponding ground truth segmentation mask.
To train the segmentation model 208, the input data may include, for each pair of input data and ground truth data, an ultrasound image of the view plane of interest. The corresponding ground truth data may include an expert-labeled segmentation of the anatomical ROI within the ultrasound image of the view plane of interest. The segmentation model 208 may be updated based on a loss function between each segmentation output by the segmentation model and the corresponding ground truth segmentation.
To train the contour refinement model 210, the input data may include, for each pair of input data and ground truth data, an ultrasound image of the view plane of interest and an adjusted segmentation template of an anatomical ROI within the ultrasound image (e.g., a template transformed using the segmentation output for model 208, as explained above), and the corresponding ground truth data may include an expert-labeled segmentation of the anatomical ROI within the ultrasound image of the view plane of interest. In some examples, the segmentation model may be trained and validated, and then the segmentation model may be deployed using the training images for training the contour refinement model to generate a plurality of segmentations that are each used to adjust the template segmentation. These adjusted segmentation templates may be used with the training images as input to train the contour refinement model. The contour refinement model 210 may be updated based on a loss function between each segmentation output by the contour refinement model and the corresponding ground truth segmentation. The output (segmentation) of the segmentation model 208 may be used as a guiding map to position a pre-defined (and fixed) template of the levator hiatus to the ultrasound image under consideration. The matched template is expected to act as an additional guidance input to the contour refinement model 210, which also takes in the original ultrasound image as an input. The corresponding ground truth data may include an expert-labeled segmentation of the anatomical ROI within the ultrasound image of the view plane of interest. The contour refinement model 210 may be updated based on a loss function between each segmentation output by the contour refinement model and the corresponding ground truth segmentation. Morphological operations are performed on the resulting segmentation output to further smoothen and refine the contour.
The segmentation model 208 and the contour refinement model 210 may be separate models/networks that are trained independently of one another. For example, the neural network of the segmentation model 208 may have different weights/biases than the neural network of the contour refinement model 210. Further, while in some examples the contour refinement model 210 may be trained using output of the segmentation model 208, the contour refinement model 210 may be trained independently from the segmentation model 208 because the contour refinement model 210 may use a different loss function than the segmentation model 208 and/or the loss function applied during training of the contour refinement model 210 does not directly take into account the output from the segmentation model 208.
In some embodiments, the non-transitory memory 206 may include components included in two or more devices, which may be remotely located and/or configured for coordinated processing. For example, at least some of the images stored as part of the ultrasound image data 212 may be stored in an image archive, such as a picture archiving and communication system (PACS). In some embodiments, one or more aspects of the non-transitory memory 206 may include remotely-accessible networked storage devices configured in a cloud computing configuration.
In some embodiments, the training module 214 is not disposed at the image processing system 202 and the view plane model 207, the segmentation model 208, and/or the contour refinement model 210 may be trained on an external device. The view plane model 207, the segmentation model 208, and/or the contour refinement model 210 on the image processing system 202 thus include trained and validated network(s).
User input device 232 may comprise one or more of a touchscreen, a keyboard, a mouse, a trackpad, a motion sensing camera, or other device configured to enable a user to interact with and manipulate data within image processing system 202. In one example, user input device 232 may enable a user to make a selection of a view plane thickness and initiate a workflow for automatically identifying a view plane via view plane model 207, segmenting a region of interest via the segmentation model 208 and contour refinement model 210, and performing automated measurements based on the segmentation.
Display device 234 may include one or more display devices utilizing virtually any type of technology. In some embodiments, display device 234 may comprise a computer monitor, and may display ultrasound images. Display device 234 may be combined with processor 204, non-transitory memory 206, and/or user input device 232 in a shared enclosure, or may be peripheral display devices and may comprise a monitor, touchscreen, projector, or other display device known in the art, which may enable a user to view ultrasound images produced by an ultrasound imaging system, and/or interact with various data stored in non-transitory memory 206.
It should be understood that image processing system 202 shown in
Each image of the plurality of 3D images 304 may correspond to a different slice of ultrasound data (with the slice extending in the elevation plane, which may be referred to as the sagittal plane), and each slice may be positioned at a different position along the azimuth direction, while the view plane of interest may extend in the azimuth direction (e.g., in the axial plane) and thus include ultrasound data from each of the plurality of 3D images. The plurality of 3D images 304 may be selected according to a suitable process. For example, the plurality of 3D images 304 may be selected automatically (e.g., by the computing device). In some examples, the plurality of 3D images 304 may be selected based on user input identifying an initial estimate of the location of the view plane within the volume 302. For example, the operator of the ultrasound probe may enter a user input indicating a location of the view plane of interest within a selected 3D ultrasound image. The computing device may then select the plurality of 3D images 304 based on the user-specified location of the view plane of interest and/or the 3D image selected by the user. For example, the plurality of 3D images 304 may include the 3D image selected by the user and one or more additional 3D images near the 3D image selected by the user (e.g., the slices adjacent to the 3D image selected by the user). While
The plurality of 3D images 304 are combined into a stacked set of 3D images 306. The plurality of 3D images may be concatenated or combined as multiple layers to form the stacked set of 3D images 306. The stacked set of 3D images 306 is entered as input to a view plane model 307, which is a non-limiting example of view plane model 207 of
In this way, the view plane model may identify a view plane of interest (e.g., the MHD plane) within a volume of ultrasound data. Once the view plane of interest is identified, a 3D image of that view plane may be rendered and used for further processing in the automated ultrasound exam. In contrast, prior manual ultrasound exams may demand the operator identify the location of the view plane of interest on a selected 3D image (e.g., first 3D image 406) by applying a rendering box to the image, such as by drawing a box that encompasses the location of the view plane. As appreciated from the view plane indicator 408 and the other view plane lines shown in
The corrected segmentation template may be entered as input along with the image 502 to a contour refinement model at 510 (e.g., the contour refinement model 210 of
At 702, method 700 includes acquiring ultrasound data of a patient. The ultrasound data may be acquired with an ultrasound probe (e.g., ultrasound probe 106 of
At 706, method 700 includes acquiring volumetric ultrasound data with the patient in a first condition. Some exam workflows, such as pelvic exams, may dictate that an anatomical neighborhood of the patient (e.g., the pelvis) be imaged while the patient performs muscle contractions, releases, breath holds, or other maneuvers. Thus, the operator may control the ultrasound probe to acquire a volumetric ultrasound dataset of the anatomical neighborhood while the patient is instructed to assume/hold the first condition, which may be a breath hold such as Valsalva, for example.
At 708, once the volumetric ultrasound dataset has been acquired, selected frames of the volumetric ultrasound data are entered as input into a view plane model, such as view plane model 207 of
At 714, the location of the view plane (as identified by the 2D segmentation mask) may be displayed as a view plane indicator that is overlaid on one of the selected ultrasound image frames. In this way, the operator may view the identified location of the view plane of interest. If the operator does not agree with the identified location of the view plane of interest, the operator may enter user input (e.g., moving the view plane indicator as desired) and method 700 may include adjusting the view plane based on the entered user input at 716.
At 718, method 700 determines if the exam workflow includes additional patient conditions. For example, after the first patient condition, the exam workflow may dictate that a new volume of ultrasound data be acquired with the patient in a second condition that is different than the first condition (e.g., muscle contraction). If the workflow includes additional patient conditions that have not been imaged, method 700 proceeds to 720 to acquire volumetric ultrasound data with the patient in the next condition. The acquisition of the volumetric data with the patient in the next condition may include, before the volumetric data is acquired, reception of user input specifying the view plane length and desired slice thickness on a selected image. The user input may trigger the next volumetric acquisition. Method 700 then loops back to 708 and repeats the identification of the view plane of interest in the newly-acquired volumetric ultrasound dataset. If it is instead determined at 718 that the workflow does not include additional patient conditions (e.g., all patient conditions have been imaged) and/or that the exam is complete, method 700 ends.
At 802, method 800 includes obtaining a view plane image. The view plane image may be obtained by extracting the view plane image from a volumetric ultrasound dataset based on a mask output by a view plane model (such as view plane model 207). The view plane image may be extracted based on one of the 2D segmentation masks output as part of method 700 described above. For example, the volumetric ultrasound dataset may be the volumetric ultrasound dataset acquired as part of method 700 with the patient in the first condition. The 2D segmentation mask may indicate the location of the view plane of interest within the volumetric ultrasound data, which may be the MHD plane as explained above. The view plane image may be extracted by extracting the ultrasound data from the volumetric ultrasound dataset that lies in the plane identified by the 2D segmentation mask, as well as ultrasound data adjacent to the plane (e.g., above and below) as dictated by the user-specified slice thickness (as explained above with respect to
At 804, the view plane image is entered as input to a segmentation model, such as segmentation model 208. The segmentation model may be a deep learning model (e.g., neural network) trained to output a segmentation of an anatomical ROI within the view plane image, such as the levator hiatus. In some examples, the deep learning model may be trained to segment additional structures to improve accuracy and/or model training, but the anatomical ROI may be the only segmented structure that is output to the user. Thus, at 806, method 800 includes receiving a segmentation of the anatomical ROI from the segmentation model. As explained previously, the anatomical ROI may exhibit patient to patient variation that may make an accurate segmentation of the anatomical ROI for each patient difficult for a deep learning model to perform. Accordingly, the segmentation output by the segmentation model (which may be an initial segmentation of the anatomical ROI) may be used to adjust a template of the anatomical ROI, as indicated at 808. The template of the anatomical ROI may be an average/mean shape and/or size of the anatomical ROI determined based on a plurality of patients. For example, the training data used to train the segmentation model may include ground truth data comprising expert-labeled images of a plurality of patients, where the labels indicate the boundaries of the anatomical ROI in each image. The labels/boundaries generated by the experts may be averaged using a suitable process, such as a Procrustes analysis, to identify the mean shape of the anatomical ROI. The initial segmentation may be used to adjust the pre-determined template with a transformation matrix. The template may be adjusted as dictated by the initial segmentation in the x- and y-directions (e.g., stretched and/or squeezed) but may not be rotated or have other more complex transformations applied. Once the template is adjusted based on the segmentation, an adjusted segmentation template is formed.
At 810, the view plane image and the adjusted segmentation template are entered as input to a contour refinement model (e.g., contour refinement model 210 of
At 814, a contour generated from the refined segmentation is displayed as an overlay on the view plane image. The contour may be the border/boundary of the refined segmentation. By displaying the contour as overlay on the view plane image (where the contour is aligned with the anatomical ROI within the view plane image so that the contour marks the border of the anatomical ROI), the operator of the ultrasound system or other clinician viewing the view plane image may determine if the contour is accurate and sufficiently defines the anatomical ROI within the view plane image.
At 816, one or more measurements may be performed based on the contour. For example, the area, circumference, diameter(s) of the anatomical ROI may be measured automatically based on the contour. To determine the diameter(s), one or more measurement lines may be placed across the contour, e.g., a first measurement line may be placed at the longest segment of the contour and a second measurement line may be placed at the widest segment of the contour. The measurements may be displayed for user review and/or saved as part of the patient exam.
At 818, method 800 determines if additional volumes are available for analysis. As explained previously, during pelvic exams, multiple volumes of ultrasound data may be acquired at different patient conditions. If an additional volume of ultrasound data is available for analysis (e.g., a second volumetric ultrasound dataset acquired during a second condition, as explained above with respect to
The first example GUI 900 further includes the first view plane image 910, which is a 3D rendering of an axial slice of data from the first volumetric ultrasound dataset, where the slice extends in the view plane as defined by the first view plane indicator 904 and has a thickness defined by the first slice thickness line 906. The first view plane image 910 includes, as an overlay, a first contour 912 showing the border of the anatomical ROI (herein the levator hiatus) as determined from the output of the segmentation model and the contour refinement model, as well as two measurement lines. The border and measurement lines of the anatomical ROI may be used to generate measurements of the anatomical ROI, which are shown in a first measurement box 914. As shown, the anatomical ROI in the first volumetric ultrasound dataset has a first area (e.g., 26.5 cm2), a first anterior-posterior (AP) diameter (e.g., 72.3 mm), and a first lateral (lat) diameter (e.g., 48.1 mm).
The second example GUI 920 further includes the second view plane image 930, which is a 3D rendering of an axial slice of data from the second volumetric ultrasound dataset, where the slice extends in the view plane as defined by the second view plane indicator 924 and has a thickness defined by the second slice thickness line 926. The second view plane image 930 includes, as an overlay, a second contour 932 showing the border of the anatomical ROI (herein the levator hiatus) as determined from the output of the segmentation model and the contour refinement model, as well as two measurement lines. The border and measurement lines of the anatomical ROI may be used to generate measurements of the anatomical ROI, which are shown in a second measurement box 934. As shown, the anatomical ROI in the second volumetric ultrasound dataset has a second area (e.g., 23.8 cm2), a second AP diameter (e.g., 63.7 mm), and a second lat diameter (e.g., 49.6 mm).
A technical effect of executing an automated ultrasound exam including automatically identifying a view plane of interest within a volume of ultrasound data using a view plane model is that the view plane of interest may be identified more accurately and quickly than manual identification of the view plane of interest. Another technical effect of executing the automated ultrasound exam including segmenting an anatomical ROI using two independent segmentation models and an adjusted segmentation template is that the anatomical ROI may be identified quickly and in a more accurate manner than relying on a standard, single segmentation model.
The disclosure also provides support for a method, comprising: identifying a view plane of interest based on one or more 3D ultrasound images, obtaining a view plane image including the view plane of interest from a 3D volume of ultrasound data of a patient, where the one or more 3D ultrasound images are generated from the 3D volume of ultrasound data, segmenting an anatomical region of interest (ROI) within the view plane image to generate a contour of the anatomical ROI, and displaying the contour on the view plane image. In a first example of the method, the view plane of interest includes a minimal hiatal dimension (MHD) plane and the anatomical ROI comprises a levator hiatus. In a second example of the method, optionally including the first example, the method further comprises: identifying a first diameter of the contour and a second diameter of the contour, and displaying the first diameter and the second diameter. In a third example of the method, optionally including one or both of the first and second examples, segmenting the anatomical ROI to generate the contour comprises entering the view plane image as input into a segmentation model trained to output an initial segmentation of the anatomical ROI. In a fourth example of the method, optionally including one or more or each of the first through third examples, segmenting the anatomical ROI to generate the contour further comprises adjusting a template segmentation of the anatomical ROI based on the initial segmentation to generate an adjusted segmentation template and entering the adjusted segmentation template and the view plane image as input to a contour refinement model trained to output a refined segmentation of the anatomical ROI, the contour based on the refined segmentation. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, the segmentation model and the contour refinement model are separate models and are trained independently of one another. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, the template segmentation represents an average segmentation of the anatomical ROI from a plurality of patients. In a seventh example of the method, optionally including one or more or each of the first through sixth examples, identifying the view plane of interest based on the one or more 3D ultrasound images comprises entering the one or more 3D ultrasound images as input to a view plane model trained to output a 2D segmentation mask indicating a location of the view plane of interest within the 3D volume of ultrasound data.
The disclosure also provides support for a system, comprising: a display device, and a computing device operably coupled to the display device and including memory storing instructions executable by a processor to: identify a view plane of interest based on one or more 3D ultrasound images, obtain a view plane image including the view plane of interest from a 3D volume of ultrasound data of a patient, where the one or more 3D ultrasound images are generated from the 3D volume of ultrasound data, segment an anatomical region of interest (ROI) within the view plane image to generate a contour of the anatomical ROI, and display the contour on the view plane image on the display device. In a first example of the system, the memory stores a view plane model trained to identify the view plane of interest using the one or more 3D ultrasound images as input. In a second example of the system, optionally including the first example, the view plane model comprises one or more 3D convolution layers, a flattening layer, and a 2D network. In a third example of the system, optionally including one or both of the first and second examples, the memory stores a segmentation model and a contour refinement model that are deployed to segment the anatomical ROI. In a fourth example of the system, optionally including one or more or each of the first through third examples, the segmentation model is trained to output an initial segmentation of the anatomical ROI using the view plane image as input, and the contour refinement model is trained to output a refined segmentation of the anatomical ROI using the view plane image and an adjusted segmentation template, the adjusted segmentation template including a template segmentation adjusted based on the initial segmentation, and wherein the contour of the anatomical ROI is generated from the refined segmentation. In a fifth example of the system, optionally including one or more or each of the first through fourth examples, the view plane of interest includes a minimal hiatal dimension (MHD) plane and the anatomical ROI comprises a levator hiatus.
The disclosure also provides support for a method for an automated pelvic ultrasound exam, comprising: identifying a minimal hiatal dimension (MHD) plane based on one or more 3D ultrasound images generated from a 3D volume of ultrasound data of a patient, displaying, on a display device, an indicator of a location of the MHD plane relative to one of the one or more 3D ultrasound images, obtaining an MHD image including the MHD plane from the 3D volume of ultrasound data, segmenting a levator hiatus within the MHD image to generate a contour of the levator hiatus, performing one or more measurements of the levator hiatus based on the contour, and displaying, on the display device, results of the one or more measurements and/or displaying the contour on the MHD image. In a first example of the method, the 3D volume of ultrasound data is a first 3D volume of ultrasound data acquired while the patient is in a first condition, and further comprising: identifying the MHD plane based on one or more second 3D ultrasound images generated from a second 3D volume of ultrasound data of the patient acquired while the patient is in a second condition, displaying, on the display device, a second indicator of a second location of the MHD plane relative to one of the one or more second 3D ultrasound images, obtaining a second MHD image including the MHD plane from the second 3D volume of ultrasound data, segmenting the levator hiatus within the second MHD image to generate a second contour of the levator hiatus, performing one or more second measurements of the levator hiatus based on the second contour, and displaying, on the display device, results of the one or more second measurements and/or the second contour on the second MHD image. In a second example of the method, optionally including the first example, identifying the MHD plane based on the one or more 3D ultrasound images comprises entering the one or more 3D ultrasound images as input to a view plane model trained to output a 2D segmentation mask indicating a location of the MHD plane within the 3D volume of ultrasound data. In a third example of the method, optionally including one or both of the first and second examples, segmenting the levator hiatus to generate the contour comprises entering the MHD image as input into a segmentation model trained to output an initial segmentation of the levator hiatus. In a fourth example of the method, optionally including one or more or each of the first through third examples, segmenting the levator hiatus to generate the contour further comprises adjusting a template segmentation of the levator hiatus based on the initial segmentation to generate an adjusted segmentation template and entering the adjusted segmentation template and the MHD image as input to a contour refinement model trained to output a refined segmentation of the levator hiatus, the contour based on the refined segmentation. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, the template segmentation represents an average segmentation of the levator hiatus from a plurality of patients.
When introducing elements of various embodiments of the present disclosure, the articles “a,” “an,” and “the” are intended to mean that there are one or more of the elements. The terms “first,” “second,” and the like, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. As the terms “connected to,” “coupled to,” etc. are used herein, one object (e.g., a material, element, structure, member, etc.) can be connected to or coupled to another object regardless of whether the one object is directly connected or coupled to the other object or whether there are one or more intervening objects between the one object and the other object. In addition, it should be understood that references to “one embodiment” or “an embodiment” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features.
In addition to any previously indicated modification, numerous other variations and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of this description, and appended claims are intended to cover such modifications and arrangements. Thus, while the information has been described above with particularity and detail in connection with what is presently deemed to be the most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that numerous modifications, including, but not limited to, form, function, manner of operation and use may be made without departing from the principles and concepts set forth herein. Also, as used herein, the examples and embodiments, in all respects, are meant to be illustrative only and should not be construed to be limiting in any manner.
