Our project aims to enhance eye scan segmentation and analysis by leveraging secondary data from ophthalmology repositories. We will utilize the multimodal capabilities of LLaMA 3.2, a vision-enabled language model (LLM), fused with a U-Net architecture for precise pixel-level segmentation. By training on existing ophthalmology data from GREI repositories, including eye scans and clinical data, our model will not only perform accurate segmentation but also provide context-aware analysis linking visual findings with patient histories and medical knowledge. This secondary analysis has the potential to improve diagnostic accuracy, reduce interpretation time, and ultimately enhance patient outcomes in ophthalmology by providing clinicians with detailed, context-rich insights from existing data.
Research Aims
We propose to conduct a secondary analysis by developing a context-aware eye scan segmentation and analysis model using existing ophthalmology datasets. Our project aims to fuse LLaMA 3.2’s vision capabilities with a U-Net architecture to improve segmentation accuracy and provide detailed clinical insights.
Data Utilization
We will use pre-labeled and unlabeled eye scan images, including OCT and fundus images, along with associated clinical data such as patient histories, diagnoses, and treatments. These datasets will be sourced from GREI repositories, specifically from the Mendeley Data repository. The data types include image data and text-based clinical information, encompassing eye scan images and their metadata.
Incorporation of GREI Repository Data
By utilizing datasets from Mendeley Data, a GREI repository, we ensure that our project leverages openly available, high-quality datasets. These repositories provide the necessary scan images and clinical data required for training and validating our model. The scans need to be collected and organized into a complete dataset before use as they exist in case by case format in Mendeley where each case has extensive scans of the patient throughout the years.
Methods and Analysis
We propose a custom fusion of the U-Net architecture with LLaMA 3.2, leveraging both the language model’s vision comprehension and its text-based capabilities to guide and enhance image segmentation. The U-Net will handle segmentation, while LLaMA’s vision module analyzes the segmented scan and uses cross-attention layers to improve both segmentation and scan analysis. The textual understanding of LLaMA will further contribute to the model’s output for explainability and enhance segmentation quality through its deeper analysis of medical data.
Overview of LLaMA 3.2-Vision: LLaMA 3.2's vision module processes and analyzes images in conjunction with text. This multimodal model is ideal for ophthalmology, where the segmentation of retinal layers and blood vessels in eye scans can be linked to clinical and patient data to offer deeper insight into both segmentation and image interpretation.
U-Net for Segmentation: We utilize the U-Net architecture for precise segmentation, particularly for detecting subtle retinal damage. The U-Net encoder processes the scan, passing encoded features through the decoder and through LLaMA’s cross-attention layers for deeper text-image analysis. The U-Net decoder outputs pixel-level segmentation maps, enriched by LLaMA’s contextual understanding of patient data and clinical conditions.
Training Process on Ophthalmology Data: LLaMA 3.2 will be fine-tuned on ophthalmology text and eye scan datasets that include labeled scans and patient data. The U-Net and LLaMA will be trained on both image and text data, allowing the model to segment scans while considering the clinical context with cross-modal capabilities focusing on critical regions as signs of disease.
Expected Findings and Outcomes
Enhanced Segmentation Quality: By combining LLaMA 3.2’s vision capabilities with the U-Net, we expect to achieve higher segmentation accuracy and usability compared to traditional models. The use of ophthalmology-specific datasets during training will enable the model to generalize across different types of eye scans, improving its robustness in clinical settings. This increased precision is expected to be particularly impactful in cases where current methods struggle, such as low-contrast images or scans containing subtle lesions.
Context-Aware Analysis: Along with enhanced segmentation, LlaMA’s ability to process both visual and text information means that it can offer context-aware analysis. For example, it could provide an assessment of how the segmented regions of an eye scan compare with scans from similar cases or suggest next steps based on its understanding of the patient’s condition. This would enable clinicians to rely on the model not just for image segmentation but for a more holistic interpretation of the patient’s health.
Clinical Utility: The proposed solution is designed to be highly practical for clinical use. It can assist ophthalmologists in making faster and more accurate diagnoses, reducing the burden of manual interpretation. Furthermore, the model’s explainability—its ability to provide insights into why it segmented or analyzed a scan in a particular way—makes it a valuable tool in supporting clinical decision-making.
Sharing and Dissemination
Results will be published in peer-reviewed journals and/or conferences. The trained model and code will be made available on open-source platforms like GitHub. We will reference the datasets used, adhering to licensing agreements, to facilitate reproducibility.
FAIR and CARE Principles
Our project aligns with the FAIR principles by using data that is Findable, Accessible, Interoperable, and Reusable from GREI repositories. Outputs will be openly accessible with clear metadata and standardized formats for interoperability. Regarding CARE principles, we will respect data sovereignty, especially if working with indigenous data, ensuring appropriate governance and benefit-sharing.
Replicability and Reproducibility
We will provide detailed documentation of our methods, including data preprocessing steps, model architectures, and training procedures. To facilitate the reliability of our models and algorithms and the reproducibility of our results, we will disseminate our project instruction manuals, de-identified datasets, codes, and trained models. The data, codes, and documents will be shared on public repositories such as GitHub and Open Science Framework.
Contributions to Scientific Disciplines
Our project contributes to medical imaging and AI (Artificial Intelligence) by demonstrating how multimodal models can improve diagnostic tools. It advances the application of LLMs in medical image analysis, integrating textual and visual data in AI models for healthcare, and could inspire similar approaches in other medical fields.
Our proposed work is based on the synergistic research collaboration of an interdisciplinary team. As an impactful outcome of the proposed work, we expect AiScan, which we designed for interdisciplinary research focused on data science for vision research, and which can later be extended to other data-intensive domains, to address unmet clinical needs. We expect AiScan to be broadly distributed to produce a series of transformative outcomes in vision research.
Impact on Diagnosis, Treatment, and Prevention
By providing a more accurate and context-aware analysis of eye scans, our model can assist ophthalmologists in the early detection of diseases like glaucoma, diabetic retinopathy, and macular degeneration. Early and accurate diagnosis can lead to timely interventions, better treatment outcomes, and potentially prevent vision loss, thereby significantly impacting patient quality of life.
Our interdisciplinary team consists of ophthalmology researchers and data scientists. We came together through a shared interest in applying advanced AI techniques to improve ophthalmic care. Our data scientists have expertise in AI, machine learning, deep learning, and LLMs. The basic, translational and clinical researcher active in vision research and ophthalmology bring clinical insights to the project and help interpret medical data. We will collaborate closely through regular meetings, integrating clinical expertise with technical development to align our project goals with real-world healthcare needs.
Dr. Shu-Ching Chen is a fellow of IEEE, AAAS, AAIA, and SIRI, and is the inaugural Executive Director of the Data Science and Analytics Innovation Center (dSAIC). dSAIC is a multi-university center based at the University of Missouri-Kansas City (UMKC).
Dr. Mei-Ling Shyu is a fellow of IEEE, AAAS, AIMBE, AAIA, and SIRI, professor of Electrical and Computer Engineering at UMKC, and has extensive research experience in the areas of machine learning, deep learning, data mining, big data analytics, data integration and information fusion with successful applications in healthcare including ophthalmology.
Dr. Koulen is a fellow of ARVO, Professor of Ophthalmology and the Felix and Carmen Sabates Missouri Endowed Chair in Vision Research and serves as Vice Chair for Research, Department of Ophthalmology, at the UMKC School of Medicine.
Key considerations for success include:
