Patent Application
20210304410
The present disclosure relates to assessing cardiac function, specifically, methods and systems incorporating machine learning and/or artificial intelligence algorithms for determining ejection fraction.
Accurate assessment of cardiac function is crucial for the diagnosis of cardiovascular disease, screening for cardiotoxicity and decisions regarding the clinical management of patients with a critical illness. (See e.g., Ziaeian, B. & Fonarow, G. C. Epidemiology and aetiology of heart failure. Nat. Rev. Cardiol. 13, 368-378 (2016).; Shakir, D. K. & Rasul, K. I. Chemotherapy induced cardiomyopathy: pathogenesis, monitoring and management. J. Clin. Med. Res. 1, 8-12 (2009). Dellinger, R. P. et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med. 39, 165-228 (2013); the disclosures of which are hereby incorporated by reference in their entireties.) However, human assessment of cardiac function focuses on a limited sampling of cardiac cycles and has considerable inter-observer variability despite years of training. (See e.g., Farsalinos, K. E. et al. Head-to-head comparison of global longitudinal strain measurements among nine different vendors: The EACVI/ASE Inter-Vendor Comparison Study. J. Am. Soc. Echocardiogr. 28, 1171-1181 (2015); Lang, R. M. et al. Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging. Eur. Heart J. Cardiovasc. Imaging 16, 233-271 (2015); the disclosures of which are hereby incorporated by reference in their entireties.)
This summary is meant to provide examples and is not intended to be limiting of the scope of the invention in any way. For example, any feature included in an example of this summary is not required by the claims, unless the claims explicitly recite the feature.
In one embodiment, a method for analyzing images obtained from an echocardiogram includes obtaining a cardiac ultrasound video of an individual's heart illustrating at least one view of a heart of the patient, assessing at least one cardiac parameter based on the cardiac ultrasound video using a first machine learning model comprising spatiotemporal convolutions, and outputting the at least one cardiac parameter.
In a further embodiment, the spatiotemporal convolutions incorporate two dimensions of spatial information and temporal information as a third dimension.
In another embodiment, the assessing at least one cardiac parameter step further includes determining an ejection fraction of the individual's heart.
In a still further embodiment, determining the ejection fraction is accomplished by segmenting a left ventricle of the individual's heart via a second machine learning model trained for semantic segmentation of the left ventricle.
In still another embodiment, the second machine learning model incorporates atrous convolutions for frame level semantic segmentation.
In a yet further embodiment, the assessing at least one cardiac parameter step further includes determining at least one biomarker.
In yet another embodiment, the at least one biomarker is selected from B-type Natriuretic Peptide, troponin I, hemoglobin, and blood urea nitrogen.
In a further embodiment again, determining at least one biomarker is accomplished via a second machine learning algorithm trained using ablated data.
In another embodiment again, the cardiac ultrasound video is obtained in real-time or as a recording.
In a further additional embodiment, the cardiac ultrasound video is obtained from a large ultrasound device or a small ultrasound device.
In another additional embodiment, the cardiac ultrasound video is obtained across a network.
In a still yet further embodiment, an echocardiography system includes a processing device including a machine learning module for assessing at least one cardiac parameter based on the cardiac ultrasound video using a first machine learning model including spatiotemporal convolutions.
In still yet another embodiment, the spatiotemporal convolutions incorporate two dimensions of spatial information and temporal information as a third dimension.
In a still further embodiment again, assessing at least one cardiac parameter includes determining an ejection fraction of the individual's heart.
In still another embodiment again, determining the ejection fraction is accomplished by segmenting a left ventricle of the individual's heart via a second machine learning model trained for semantic segmentation of the left ventricle.
In a still further additional embodiment, the second machine learning model incorporates atrous convolutions for frame level semantic segmentation.
In still another additional embodiment, the assessing at least one cardiac parameter step further includes determining at least one biomarker.
In a yet further embodiment again, the at least one biomarker is selected from B-type Natriuretic Peptide, troponin I, hemoglobin, and blood urea nitrogen.
In yet another embodiment again, determining at least one biomarker is accomplished via a second machine learning algorithm trained using ablated data.
In a yet further additional embodiment, the system further includes an input device configured to obtain ultrasound video.
The foregoing and other objects, features, and advantages of the disclosed technology will become more apparent from the following detailed description, which proceeds with reference to the accompanying figures.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.
Cardiac function is essential for the maintenance of normal systemic tissue perfusion; cardiac dysfunction manifests as dyspnea, fatigue, exercise intolerance, fluid retention, and increased risk of mortality. (See e.g., Ziaeian, B. & Fonarow, G. C. Epidemiology and aetiology of heart failure. Nat. Rev. Cardiol. 13, 368-378 (2016).; Shakir, D. K. & Rasul, K. I. Chemotherapy induced cardiomyopathy: pathogenesis, monitoring and management. J. Clin. Med. Res. 1, 8-12 (2009). Dellinger, R. P. et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med. 39, 165-228 (2013); Lang, R. M. et al. Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging. Eur. Heart J. Cardiovasc. Imaging 16, 233-271 (2015); McMurray, J. J. et al. ESC guidelines for the diagnosis and treatment of acute and chronic heart failure 2012. Eur. J. Heart Fail. 14, 803-869 (2012); Loehr, L. R., Rosamond, W. D., Chang, P. P., Folsom, A. R. & Chambless, L. E. Heart failure incidence and survival (from the Atherosclerosis Risk in Communities study). Am. J. Cardiol. 101, 1016-1022 (2008); Bui, A. L., Horwich, T. B. & Fonarow, G. C. Epidemiology and risk profile of heart failure. Nat. Rev. Cardiol. 8, 30-41 (2011); the disclosures of which are hereby incorporated by reference in their entireties.) Impairment of cardiac function is described as cardiomyopathy or heart failure and is a leading cause of hospitalization in the United States and a growing global health issue. (See e.g., Roizen, M. F. Forecasting the future of cardiovascular disease in the United States: a policy statement from the American Heart Association. Yearbook Anesthesiol. Pain Manage. 2012, 12-13 (2012); Yancy, C. W. et al. 2013 ACCF/AHA guideline for the management of heart failure. Circulation 128, e240-e327 (2013); the disclosures of which are hereby incorporated by reference in their entireties.) A variety of methodologies have been used to quantify cardiac function and diagnose dysfunction. In particular, measurement of left ventricular ejection fraction, the ratio of change in the left ventricular end-systolic and end-diastolic volumes, is one of the most important metrics of cardiac function, as it identifies patients who are eligible for life-prolonging therapies. (See e.g., Huang, H. et al. Accuracy of left ventricular ejection fraction by contemporary multiple gated acquisition scanning in patients with cancer: comparison with cardiovascular magnetic resonance. J. Cardiovasc. Magn. Reson. 19, 34 (2017); the disclosure of which is hereby incorporated by reference in its entirety.) However, the assessment of ejection fraction is associated with considerable interobserver variability as well as inter-modality discordance based on methodology and modality. (See e.g., Pellikka, P. A. et al. Variability in ejection fraction measured by echocardiography, gated single-photon emission computed tomography, and cardiac magnetic resonance in patients with coronary artery disease and left ventricular dysfunction. JAMA Netw. Open 1, e181456 (2018); Malm, S., Frigstad, S., Sagberg, E., Larsson, H. & Skjaerpe, T. Accurate and reproducible measurement of left ventricular volume and ejection fraction by contrast echocardiography: a comparison with magnetic resonance imaging. J. Am. Coll. Cardiol. 44, 1030-1035 (2004); Cole, G. D. et al. Defining the real-world reproducibility of visual grading of left ventricular function and visual estimation of left ventricular ejection fraction: impact of image quality, experience and accreditation. Int. J. Cardiovasc. Imaging 31, 1303-1314 (2015); the disclosures of which are hereby incorporated by reference in their entireties.)
Human assessment of the ejection fraction has variance in part due to the common finding of irregularity in the heart rate and the laborious nature of a calculation that requires manual tracing of the size of the ventricle to quantify every beat. Although the American Society of Echocardiography and the European Association of Cardiovascular Imaging guidelines recommend tracing and averaging up to five consecutive cardiac cycles if variation is identified, the ejection fraction is often evaluated from tracings of only one representative beat or visually approximated if a tracing is deemed to be inaccurate (See e.g., Koh, A. S. et al. A comprehensive population-based characterization of heart failure with mid-range ejection fraction. Eur. J. Heart Fail. 19, 1624-1634 (2017); the disclosure of which is hereby incorporated by reference in its entirety.) This results in high variance and limited precision with inter-observer variation ranging from 7.6% to 13.9%. More-precise evaluation of cardiac function is necessary, as even patients with a borderline reduction in ejection fraction have been shown to have considerably increased morbidity and mortality. (See e.g., Chioncel, O. et al. Epidemiology and one-year outcomes in patients with chronic heart failure and preserved, mid-range and reduced ejection fraction: an analysis of the ESC Heart Failure Long-Term Registry. Eur. J. Heart Fail. 19, 1574-1585 (2017); Shah, K. S. et al. Heart failure with preserved, borderline, and reduced ejection fraction: 5-year outcomes. J. Am. Coll. Cardiol. 70, 2476-2486 (2017); Papolos, A., Narula, J., Bavishi, C., Chaudhry, F. A. & Sengupta, P. P. U.S. hospital use of echocardiography: insights from the nationwide inpatient sample. J. Am. Coll. Cardiol. 67, 502-511 (2016); the disclosures of which are hereby incorporated by reference in their entireties.)
With rapid image acquisition, relatively low cost and without ionizing radiation, echocardiography is the most widely used modality for cardiovascular imaging. (See e.g., Douglas, P. S. et al. ACCF/ASE/AHA/ASNC/HFSA/HRS/SCAI/SCCM/SCCT/SCMR 2011 Appropriate use criteria for echocardiography. J. Am. Soc. Echocardiogr. 24, 229-267 (2011); Zhang, J. et al. Fully automated echocardiogram interpretation in clinical practice: feasibility and diagnostic accuracy. Circulation 138, 1623-1635 (2018); the disclosures of which are hereby incorporated by reference in their entireties.) There is great interest in using deep learning techniques for echocardiography to determine the ejection fraction. (See e.g., Madani, A., Arnaout, R., Mofrad, M. & Arnaout, R. Fast and accurate view classification of echocardiograms using deep learning. NPJ Digit. Med. 1, 6 (2018); Ghorbani, A. et al. Deep learning interpretation of echocardiograms. NPJ Digit. Med. 3, 10 (2020); Behnami, D. et al. in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support 65-73 (Springer, 2018); the disclosures of which are hereby incorporated by reference in their entireties.) Previous attempts to algorithmically assess cardiac function with deep learning models relied on manually curated still images at systole and diastole instead of using the actual echocardiogram videos and these models had substantial error compared to human evaluation of cardiac function with R2 ranging between 0.33 and 0.50. Limitations in human interpretation, including laborious manual segmentation and the inability to perform beat-to-beat quantification may be overcome by sophisticated automated approaches. Recent advances in deep learning suggest that it can accurately and reproducibly identify human-identifiable phenotypes as well as characteristics that are not recognized by human experts. (See e.g., Ardila, D. et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat. Med. 25, 954-961 (2019); Poplin, R. et al. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2, 158-164 (2018); Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115-118 (2017); Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559-1567 (2018); Chen, L.-C., Papandreou, G., Schroff, F. & Adam, H. Rethinking atrous convolution for semantic image segmentation. Preprint at arxiv.org/abs/1706.05587 (2017); the disclosures of which are hereby incorporated by reference in their entireties.)
To overcome current limitations in the human assessment of cardiac function, many embodiments describe an end-to-end deep learning approach for labelling of the left ventricle and estimation of the ejection fraction from input echocardiogram videos alone. Many embodiments perform frame-level semantic segmentation of the left ventricle with weakly supervised learning from clinical expert labelling. In additional embodiments, a three-dimensional convolutional neural network (CNN) with residual connections predicts clip-level ejection fraction from the native echocardiogram videos. In further embodiments, the segmentations results are combined with clip-level predictions to produce beat-to-beat evaluation of the ejection fraction. This approach provides interpretable tracings of the ventricle, which facilitate human assessment and downstream analysis, while leveraging the three-dimensional CNN to fully capture spatiotemporal patterns in the video. (See e.g., Tran, D. et al. A closer look at spatiotemporal convolutions for action recognition. in Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018); Tran, D., Bourdev, L., Fergus, R., Torresani, L. & Paluri, M. Learning spatiotemporal features with 3D convolutional networks. In Proc. IEEE International Conference on Computer Vision 4489-4497 (2015); the disclosures of which are hereby incorporated by reference in their entireties.)
Many embodiments are directed to video-based deep-learning algorithms that achieve state-of-the-art assessment of cardiac function. Certain embodiments use expert human tracings for weakly supervised learning of left ventricular segmentation and spatiotemporal convolutions on video data to obtain a beat-to-beat cumulative evaluation of the ejection fraction across the entire video. The variance in predictions of numerous embodiments is comparable to or less than measurements of cardiac function by human experts. Moreover, its performance in predicting the ejection fraction was robustly accurate when used on a validation dataset of echocardiogram videos from an independent medical center without additional model training. With only one GPU, many embodiments are capable of completing these tasks in real time; each prediction task takes only 0.05 s per frame and is much more rapid than the human assessment of ejection fraction.
Many embodiments greatly decrease the labor of the cardiac function assessment by automating the segmentation task and provides the opportunity for more-frequent, rapid evaluations of cardiac function. The end-to-end approach of numerous embodiments generates beat- and clip-level predictions of the ejection fraction as well as the segmentation of the left ventricle throughout the cardiac cycle for visual interpretation of the modelling results. In settings in which the sensitive detection of change in cardiac function is critical, early detection of change can substantially affect clinical care.
In many embodiments, one or more machine learning models predict the ejection fraction for each cardiac cycle using spatiotemporal convolutions with residual connections and generates frame-level semantic segmentations of the left ventricle using weak supervision from expert human tracings. These outputs are combined to create beat-to-beat predictions of the ejection fraction and to predict the presence of heart failure with reduced ejection fraction.
Turning to
Some embodiments construct a first CNN model 104 with atrous convolutions for frame-level semantic segmentation of the left ventricle. The technique of atrous convolutions enables the model to capture larger patterns and has previously been shown to perform well on non-medical imaging datasets. The standard human clinical workflow for estimating the ejection fraction requires manual segmentation of the left ventricle during end systole and end diastole. Numerous embodiments generalize these labels in a weak supervision approach with atrous convolutions to generate frame-level semantic segmentation throughout the cardiac cycle in a 1:1 pairing with the original video. In numerous embodiments, the automatic segmentation is used to identify ventricular contractions. Additionally, further embodiments provide a clinician-interpretable intermediary that mimics the clinical workflow.
Second, many embodiments train a second CNN model 106 with residual connections and spatiotemporal convolutions across frames to predict the ejection fraction. In contrast to previous CNN architectures for machine learning of medical images, this approach integrates spatial as well as temporal information with temporal information across frames as the third dimension in network convolutions. Spatiotemporal convolutions, which incorporate spatial information in two dimensions as well as temporal information in the third dimension, have not previously been used for medical data given the relative scarcity of labelled medical videos.
Returning to
Based on a test dataset, the prediction of the ejection fraction by certain embodiments had a mean absolute error of 4.1%, root mean squared error of 5.3% and R2 of 0.81 compared with the annotations by human experts. These parameters are well within the range of typical measurement variation between different clinicians, which is usually described as inter-observer variation and can be as high as 13.9% (
Additionally, many embodiments show cross-healthcare system reliability. For example, one embodiment was tested on an external test dataset of 2,895 echocardiogram videos from 1,267 patients from an independent hospital system. The results indicated a robust prediction of the ejection fraction with a mean absolute error of 6.0%, root mean squared error of 7.7%, R2 of 0.77 and an area under the curve of 0.96 compared with the annotations by cardiologists at the independent hospital system.
Comparison with Human Variation
Embodiments also show reduce variability due to human-caused effects. For example, when an embodiment trained on the same ultrasound 202 as used for some patients and an embodiment trained on data from a different ultrasound 204 as used for some patients were compared to methods to determine ejection fraction, including Simpson's monoplane method 206, Simpson's biplane method 208, and global longitudinal strain 210, the exemplary embodiments less variance. As illustrated in
Many embodiments automatically generate segmentations of the left ventricle and/or right ventricle. Segmentation enables clinicians to better understand how these embodiments make predictions. Additionally, segmentation is also useful because it provides a relevant point for human interjection in the workflow and for physician oversight of the model in clinical practice.
To train the segmentation model, manual segmentations are obtained via standard clinical workflows during end systole and end diastole. Implicit in the echocardiogram videos is that, in all intermediate frames, the left ventricle is constrained in shape and size between the labels at end systole and end diastole. Certain embodiments used these sparse human labels to train many embodiments to generate frame-level segmentations for the entire video (
Variation in beat-to-beat model interpretation was seen in echocardiogram videos of patients with arrhythmias and ectopy. Turning to
Right ventricular assessment is challenging given more variation in image acquisition, complex anatomic structure limiting assessment in one 2-dimensional view, and more limited understanding of normal function. Compared with the left ventricle, the right ventricle is poorly annotated and often requires expensive, low-throughput advanced imaging. In embodiments that segment the right ventricle, the challenge of limited human expert annotations was overcome with fuzzy augmentation, in which training dataset size was augmented by using human labels to weakly supervise preceding and subsequent frames, and constrained loss, in which an additional loss function penalizes change in segmentation prediction across frames of the video. Trained on a small number of echocardiogram videos, these embodiments accurately predict right ventricular dysfunction, pulmonary hypertension, and/or provide clues for diseases causing right ventricular dysfunction.
Further embodiments train a machine learning model to identify additional markers, such as age, weight, gender/sex, and certain cardiac biomarkers. Biomarkers of interest include biomarkers relevant to cardiac function and myocyte damage, such as B-type Natriuretic Peptide (BNP), troponin I, as well as biomarkers of systemic physiology such as hemoglobin and blood urea nitrogen (BUN). Such embodiments can include this model in addition to a model of ventricular segmentation or in lieu of a model for segmentation. Embodiments comprising one or more models trained to recognize markers are trained using ablated data.
With only one GPU, many embodiments rapidly perform the predictions (less than 0.05 s per prediction) and enables the real-time segmentation of the left ventricle and prediction of the ejection fraction (Table 3).
Turning to
In various embodiments, the echocardiogram video is obtained locally (e.g., directly from an ultrasound machine), while some embodiments obtain the echocardiogram video remotely, such as across a network or on a cloud-based system. An advantage of remote setups is the ability for a central computing server across a hospital or clinical system to analyze videos obtained systemwide as well as centralize any updates or improvements to a system.
At 504, many embodiments assess cardiac parameters based on the echocardiogram video. Various embodiments identify cardiac function using one or more machine learning models described elsewhere herein. For example, many embodiments perform beat-to-beat evaluation based on spatiotemporal convolutions and semantic segmentation to identify ejection fraction and a heart failure prediction. Further embodiments provide output in the form of biomarker analysis, such as one or more of B-type Natriuretic Peptide (BNP), troponin I, hemoglobin, and blood urea nitrogen (BUN). Further embodiments provide demographic identifiers, such as age, weight, and/or sex/gender. In many embodiments, the output is provided for multiple beats within the video (e.g., 2 beats, 3 beats, 4 beats, 5 beats, 10 beats, etc.).
At 506, many embodiments output results of cardiac assessments from 504. In various embodiments, the output is qualitative, while other embodiments provide output as quantitative measures. For example, for hemoglobin measures, qualitative output could be displayed as “anemic” or “non-anemic,” while quantitative measures may provide a determination of hemoglobin concentration. Various embodiments provide both quantitative and qualitative output. Relevant qualitative and quantitative output for various aspects are known to one of skill in the art. Output can be provided as a printout or other hard copy, while some embodiments return the output as an entry into an electronic health record. Further embodiments further allow a doctor, physician, or other medical practitioner to automatically forward results to the individual with a simple option for forwarding upon receipt of the results.
Additional embodiments incorporate such methods as software or are embodied as machine readable media that can be performed by a processor. Additionally, some embodiments are directed to systems incorporating such methods into a system 550 for echocardiography. Turning to
Ultrasound images or video are sent to a processing device 554. The processing device 554 can include an image processing component 556 to allow certain annotations or input from a physician or other medical professional, including individual or patient information, such as name, date of birth, date of exam, or other information relevant to the examination. Additional embodiments include a machine learning module 558, such as described herein, to assess cardiac function and other features for the individual's cardiac health.
Further embodiments include an output device 560 to display or provide output from the echocardiography system 550 to medical professional or other individual.
Certain embodiments include a modem or other network connection device 562 to allow for communication between the echocardiography system 550 to other systems, such as a larger healthcare management system.
Although the following embodiments provide details on certain embodiments of the inventions, it should be understood that these are only exemplary in nature, and are not intended to limit the scope of the invention.
A standard full resting echocardiogram study consists of a series of 50-100 videos and still images visualizing the heart from different angles, locations and image acquisition techniques (two-dimensional images, tissue Doppler images, color Doppler images and others). Each echocardiogram video corresponds to a unique patient and a unique visit. In this dataset, one apical four-chamber two-dimensional greyscale video is extracted from each study. Each video represents a unique individual as the dataset contains 10,030 echocardiography videos from 10,030 unique individuals who underwent echocardiography between 2016 and 2018 as part of clinical care at Stanford Health Care. Videos were randomly split into 7,465, 1,277 and 1,288 patients, respectively, for the training, validation and test sets. A summary of the dataset is described in Table 4.
The randomly selected patients in the data have a range of ejection fractions representative of the patient population who visit the echocardiography laboratory. Images were acquired by skilled sonographers using iE33, Sonos, Acuson SC2000, Epiq 5G or Epiq 7C ultrasound machines and processed images were stored in a Philips Xcelera picture archiving and communication system. Video views were identified through implicit knowledge of view classification in the clinical database by identifying images and videos labelled with measurements done in the corresponding view. For example, apical four-chamber videos were identified by selecting videos from the set of videos in which a sonographer or cardiologist traced left ventricle volumes and labelled these for analysis to calculate ejection fraction. The apical four-chamber view video was thus identified by extracting the Digital Imaging and Communications in Medicine (DICOM) file linked to the measurements of the ventricular volume used to calculate the ejection fraction.
An automated preprocessing workflow was used to remove identifying information and eliminate unintended human labels. Each subsequent video was cropped and masked to remove text, electrocardiogram and respirometer information, and other information outside of the scanning sector. The resulting square images were either 600×600 or 768×768 pixels depending on the ultrasound machine and down sampled by cubic interpolation using OpenCV into standardized 112×112 pixel videos. Videos were spot-checked for quality control, to confirm view classification and to exclude videos with color Doppler.
This research was approved by the Stanford University Institutional Review Board and data privacy review through a standardized workflow by the Center for Artificial Intelligence in Medicine and Imaging (AIMI) and the University Privacy Office. In addition to masking of text, electrocardiogram information and extra data outside of the scanning sector in the video files as described above, the video data of each DICOM file was saved as an AVI file to prevent any leakage of identifying information through public or private DICOM tags. Each video was subsequently manually reviewed by an employee of the Stanford Hospital familiar with imaging data to confirm the absence of any identifying information before public release.
Model design and training was done in Python using the PyTorch deep learning library. Semantic segmentation was performed using the Deeplabv3 architecture. The segmentation model had a base architecture of a 50-layer residual net and minimized pixel-level binary cross-entropy loss. The model was initialized with random weights and was trained using a stochastic gradient descent optimizer. A model with spatiotemporal convolutions was initialized with pretrained weights from the Kinetics-400 dataset. Three model architectures with variable integration of temporal convolutions (R3D, MC3 and R2+1D) were tested and ultimately chose decomposed R2+1D spatiotemporal convolutions as the architecture with the best performance to use for various embodiments (
The models were trained to minimize the squared loss between the prediction and true ejection fraction using a stochastic gradient descent optimizer with an initial learning rate of 0.0001, momentum of 0.9 and batch size of 16 for 45 epochs. The learning rate was decayed by a factor of 0.1 every 15 epochs. For model input, video clips of 32 frames were generated by sampling every other frame (sampling period of 2) with both clip length and sampling period determined by hyperparameter search (
Test Time Augmentation with Beat-to-Beat Assessment
There can be variation in the ejection fraction, end-systolic and end-diastolic volumes during atrial fibrillation, and in the setting of premature atrial contractions, premature ventricular contractions and other sources of ectopy. The clinical convention is to identify at least one representative cardiac cycle and use this representative cardiac cycle to perform measurements, although an average of the measurements of up to five cardiac cycles is recommended when there is considerable ectopy or variation. For this reason, our final model used test time augmentation by providing individual estimates for each ventricular beat throughout the entire video and outputs the average prediction as the final model prediction. We use the segmentation model to identify the area of the left ventricle and threshold-based processing to identify ventricular contractions during each cardiac cycle. Each ventricular contraction (systole) was identified by choosing the frames of the smallest left ventricle size as identified by the segmentation arm of EchoNet-Dynamic. For each beat, a subsampled clip centered around the ventricular contraction was obtained and used to produce a beat-by-beat estimate of ejection fraction. The mean ejection fraction of all ventricular contractions in the video was used as the final model prediction.
The relationship between model performance and the quality of the echocardiogram video were evaluated. The dataset was not curated on clinical quality and no videos were excluded due to insufficient image quality. On the internal Stanford test dataset, the model performance was evaluated with variation in video saturation and gain, and the performance of this embodiment is robust to the range of the acquisition quality of the clinical images. To further test the effect of variable video quality, noise and degraded video quality were simulated by randomly removing a proportion of pixels from videos in the test dataset and evaluated model performance on the degraded images. This embodiment is also robust to a wide range of synthetic noise and image degradation.
Prospective validation was performed by two senior sonographers with advanced cardiac certification and more than 15 years of experience each. For each patient, measurements of cardiac function were independently acquired and assessed by each sonographer on the same day. Every patient was scanned using Epiq 7C ultrasound machines, the standard instrument in the Stanford Echocardiography Laboratory, and a subset of patients were also rescanned by the same two sonographers using a GE Vivid 95E ultrasound machine. Tracing and measurements were done on a dedicated workstation after image acquisition. For comparison, the independently acquired apical four-chamber videos were fed into the model and the variance in measurements assessed.
Transthoracic echocardiogram studies from November 2018 to December 2018 from an independent external healthcare system, Cedars-Sinai Medical Center, were used to evaluate the performance of EchoNet-Dynamic in predicting ejection fraction. The same automated preprocessing workflow was used to convert DICOM files to AVI files, mask information outside of the scanning sector and resize input to 112×112-pixel videos of variable length. Previously described methods were used to identify apical four-chamber view videos. After manual exclusion of incorrect classifications, long cine loops of bubble studies, videos with injection of ultrasonic contrast agents and videos with color Doppler, 2,895 videos from 1,267 patients were identified. These videos were used as the input for this embodiment trained on the Stanford dataset and model predictions were compared with human interpretations from physicians at Cedars-Sinai Medical Center. The input video sampling period was set to one as the frame rate of the external dataset was roughly half that of videos from the Stanford dataset. Model predictions from multiple videos of the same patient were averaged to produce a composite estimate of ejection fraction.
Recognizing the inherent variation in human assessment of ejection fraction, five expert sonographers and cardiologists who specialize in cardiovascular imaging performed a blinded review of the echocardiogram videos with the highest absolute difference between the initial human label and the prediction by this embodiment (mean absolute difference of 15.0%, s.d. of 3.79%). Each expert independently received the relevant echocardiogram video and a set of two blinded measurements of ejection fractions that corresponded to the initial human label and the prediction by this embodiment. The experts were asked to select which ejection fraction corresponded more closely to their evaluation of ejection fraction as well as to note any limitations in echocardiogram video quality that would hinder their interpretation. In the blinded review, experts noted that 38% (15 out of 40) of videos had considerable issues with video quality or acquisition and that 13% (5 out of 40) of videos had marked arrhythmia, limiting human assessment of ejection fraction (Table 3). In this setting, the consensus interpretation of the expert clinicians preferred the prediction by this embodiment over the initial human label in 43% (17 out of 40) of the echocardiogram videos.
No statistical methods were used to predetermine sample size. Confidence intervals were computed using 10,000 bootstrapped samples and obtaining 95 percentile ranges for each prediction. The performance of the semantic segmentation task was evaluated using the Dice similarity coefficient compared with the human labels from the held-out test dataset. The performance of the ejection fraction task was evaluated by calculating the mean absolute difference between the prediction of this embodiment and the human calculation of ejection fraction as well as calculating the R2 between the prediction by this embodiment and the human calculation. Prospective comparison with human readers was performed with the uniformly most powerful invariant equivalence test for two-sample problems.
Embodiments are capable of automating ventricular segmentation and outperform prior methods in terms of variance, specificity, and sensitivity for segmentation, cardiac function, and biomarker identification.
Having described several embodiments, it will be recognized by those skilled in the art that various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the invention. Additionally, a number of well-known processes and elements have not been described in order to avoid unnecessarily obscuring the present invention. Accordingly, the above description should not be taken as limiting the scope of the invention.
Those skilled in the art will appreciate that the foregoing examples and descriptions of various preferred embodiments of the present invention are merely illustrative of the invention as a whole, and that variations in the components or steps of the present invention may be made within the spirit and scope of the invention. Accordingly, the present invention is not limited to the specific embodiments described herein, but, rather, is defined by the scope of the appended claims.
indicates data missing or illegible when filed
This application claims priority to U.S. Provisional Application Ser. No. 63/002,098, entitled “Methods and Systems Video-Based AI for Beat-To-Beat Assessment of Cardiac Function” to Ouyang et al., filed Mar. 30, 2020; the disclosure of which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63002098 | Mar 2020 | US |
