FSU researchers unveil study on how AI can improve accuracy in differential diagnosis for healthcare providers
Florida – Florida State University researchers are shedding new light on the powerful role artificial intelligence (AI) could play in transforming the way health care professionals approach diagnosis. In a newly published study, scientists at FSU’s School of Information’s eHealth Lab explored how AI, specifically large language models (LLMs), could help clinicians improve the accuracy and efficiency of differential diagnosis—a critical and often complex step in clinical decision-making.
The research, published in npj Digital Medicine, is already gaining traction in the academic and medical communities, having been accessed more than 3,000 times since its mid-March release. The study was led by Senior Author Zhe He, director of FSU’s Institute for Successful Longevity, and Visiting Assistant Professor Balu Bhasuran, with contributions from a team of researchers across several institutions, including the National Library of Medicine, Emory University, and Tampa General Hospital.
At its core, the study evaluated how five different AI models performed when asked to generate potential diagnoses from 50 patient vignettes — real-world cases often used to train medical professionals. The models tested included GPT-4, GPT-3.5, Llama-2-70B, Claude-2, and Mixtral-8x7B. These models, all examples of LLMs, were evaluated on how well they could identify a patient’s diagnosis using only the clinical case and, in a separate evaluation, how their accuracy improved when lab test results were added.
The results were striking, especially for GPT-4, which showed a 55% top-one accuracy rate when lab data was included and an 80% lenient accuracy rate — meaning the correct diagnosis was somewhere in the AI’s top recommendations. The study also found that across all models, including lab results significantly improved diagnostic performance.
“The AI generated differential diagnosis is very comprehensive in covering all possible diagnoses for patients,” He said. “What this study helps show is how AI can potentially be used as a tool to help practitioners make more informed decisions for their patients.”
Differential diagnosis — the process by which doctors narrow down potential conditions based on overlapping symptoms — can be especially challenging in complex or rare cases. By integrating AI into this process, the researchers believe health professionals could benefit from a powerful second opinion that not only speeds up decision-making but also reduces errors.
“When we asked the model for the top differential diagnosis, most of these models were able to produce the patient’s exact diagnosis,” Bhasuran said. “That’s very interesting because it implies that even in rare case diseases, the model is able to predict that.”
The study builds on FSU’s ongoing work with the LabGenie project, an initiative aimed at improving older adults’ understanding of lab test results. The researchers’ latest findings extend the project’s focus from patient communication to clinical decision-making, presenting a broader vision for how AI tools might enhance the entire health care experience—from diagnosis to treatment.
In addition to He and Bhasuran, the research team included coauthors from the University of South Florida, University of North Texas Health Science Center, and several undergraduate students from FSU’s Undergraduate Research Opportunity Program (UROP). Students Angelique Deville, Hailey Thompson, Maggie Awad and Yash Alva played a vital role in the study by extracting key details from the 50 medical case reports used in testing.
The project was supported by a grant from the Agency for Healthcare Research and Quality and received partial funding from the University of Florida–Florida State University Clinical and Translational Science Award and the National Library of Medicine.
The implications of this study are far-reaching. Beyond providing a clearer path to diagnosis, tools like these can help cut down on unnecessary testing, reduce hospital stays, and lower health care costs. In an era where health care systems are under increasing pressure to deliver accurate, efficient, and affordable care, AI offers a promising solution that could reshape the way clinicians make critical decisions.
With their work, the FSU researchers are paving the way for AI integration into mainstream medicine—not to replace physicians, but to empower them. The team is optimistic that this research could serve as a foundation for future clinical tools that are not only accurate but also accessible and user-friendly for providers across the health care spectrum.
To learn more about the project and the eHealth Lab’s ongoing research, visit ehealthlab.cci.fsu.edu.
