NSF Award
2406231
Although healthcare Artificial Intelligence (AI) research has made notable progress, its integration into healthcare practice remains limited. Clinicians typically make decisions by reviewing electronic health records that include clinical notes, medical images, and lab results. However, most current healthcare AI tools are incapable to performing multimodal analysis. In addition, existing prediction models are inflexible and lack the interactive capabilities to address the complex and dynamic requirements of clinicians. A systematic and comprehensive evaluation framework is also missing in these healthcare AI tools. In this project, the team aims to develop an end-to-end AI system to overcome these limitations. The proposed framework brings a novel AI approach to multimodal data by training foundation models to simultaneously combine and analyze four data modalities. These modalities include: data over time (i.e., time-series), text from notes, image, and numerical data. The system then provides comprehensive evaluation and tailored feedback. The resulting technologies will facilitate seamless integration of AI systems into real-world clinical settings.<br/><br/>This project pursues three closely connected research thrusts: modeling multimodal clinical data, enhancing interactions between clinicians and AI, and enabling comprehensive evaluations of healthcare AI systems. Specifically, the team will develop new process for training foundation models to enable joint consideration of four data modalities including time-series, text, image, and tabular data. Clinicians will be able to interact with these models in natural language, and in return these models will generate contextualized clinical text. We will enhance usability, trustworthiness, and personalization by introducing three key features: (1) clinician intent elicitation, (2) trustworthy content generation grounded in multimodal input data, and (3) clinician-oriented computationally efficient content personalization. To make sure that the models can be effectively deployed in everyday routines of busy medical practitioners, we will develop standardized evaluation metrics in collaboration with clinicians, including a wide array of objective and subjective measures such as predictive accuracy, bias, toxicity, fairness, and robustness, and preferences of the clinicians. The outcomes will include innovative methodological contributions to AI, as well as datasets and benchmarks for evaluate these systems. Additionally, the project will develop key capabilities for enhancing patient safety during the high-risk transfer of care, and will make broader scientific and engineering contributions to computer science, information sciences, and statistics. This project’s findings will be promptly shared with the public and industrial partners to maximize the impact and facilitate the transition to practical applications.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
