NSF Award
2412195
The accurate prediction of clinical outcomes is a critical aspect of personalized medicine, offering vital information that can shape patient treatment plans and ultimately affect patient prognosis. The current pathological grading/classification system requires extensive information processing by a human brain to interpret highly complex data resources. Histopathology, as the cornerstone of disease diagnosis, has advanced significantly with technological innovations allowing for the capture of images at greater speed and resolution. However, most current histopathological image analysis methods often overlook the complex hierarchical structures of tissues. Understanding the intricate interactions among various cell types, which form the cellular components and, in turn, tissue architectures, is crucial for insights into biology and disease status. Analyzing pathological images is crucial, but effectively integrating associated biomedical text data, such as pathological reports, preliminary diagnosis reports, and clinical notes, poses additional challenges. This variety of text data is defined as medical captions, akin to extended image captions, which provide necessary context but also introduce further complexity in diagnostics. Enhanced computational methods that simultaneously leverage pathological slides and their captions could revolutionize the accuracy and efficiency of predicting clinical outcomes.<br/><br/>The goal of this project is to develop novel pathological image-text analysis tools for clinical outcome prediction. The project will focus on 1) developing algorithms for pathological image analysis, which include auto-prompting fine-tuning framework for subtype cell segmentation, cell-level graph learning, patch-level graph learning, and intelligent integration of cell-level graph and patch-level graph for clinical outcome prediction; 2) fine-tuning large language models using biomedical text data to obtain improved text embeddings, which include the development of algorithms for biomedical text data analysis, incorporating fine-tuning of deep pre-trained models for precise biomedical text data representation; 3) Integrating pathological image data with biomedical text data for clinical outcome prediction, which include novel algorithms for the intelligent integration of multi-modal data and cross-modal learning models to generate biomedical text data representation from the histopathological images of the same patient. The successful realization of these aims promises to provide healthcare professionals with powerful tools to enhance the decision-making process, personalize treatment plans, and improve overall patient outcomes. Additionally, the proposed study stands to offer broader insights into the integration of multi-modal medical data, setting a new standard for how medical informatics can be leveraged in the era of big data and precision medicine. The multidisciplinary nature of this project also provides unique opportunities for integrating its components into existing curricula, as well as inspiring scientific interests in K-12 students and underrepresented students. The results of this project will be disseminated in the form of peer-reviewed publications, open-source software, tutorials, seminars, and workshops.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
