SHM Converge Scientific Abstract ePoster Gallery-Can Chatgpt Identify Diagnostic Uncertainty Upon Admission?

Pdf Summary

This document summarizes a study on whether ChatGPT can identify diagnostic uncertainty in hospital admission notes compared with human reviewers.<br /><br />The study focused on diagnostic uncertainty, which refers to a clinician’s inability to confidently explain a patient’s health problem. The authors argued that measuring this uncertainty could help detect diagnostic error risk earlier during hospitalization.<br /><br />Researchers reviewed admission notes from patients on a general internal medicine service at an academic medical center in Boston. Two clinicians independently rated each note’s diagnostic uncertainty using a 6-point Likert scale, then reached consensus. They created a prompt for ChatGPT based on the manual review process and supplied it with selected parts of the admission note, including the problem list, assessment/plan, diagnostic testing, and treatments. ChatGPT was tested at temperature 0.<br /><br />The study found moderate agreement between ChatGPT and human reviewers. For ChatGPT 3.5, Cohen’s kappa was 0.48 with 74.1% agreement; for ChatGPT 4, kappa was 0.44 with 71.3% agreement. Mean and median uncertainty ratings were similar overall, though ChatGPT and humans sometimes differed in interpretation.<br /><br />The authors concluded that ChatGPT shows promise as a tool for identifying diagnostic uncertainty, but its performance is limited by differences in how uncertainty is defined and by the fact that the model did not have access to all the clinical information available to human reviewers. They suggested that further prompt refinement and more consistent definitions are needed. If improved, LLMs could become useful cognitive aids for clinicians to flag high-risk situations early and potentially reduce diagnostic errors.

Asset Subtitle

Jorge A. Rodriguez

Meta Tag

Author List Abhishek Goyal, Anuj K. Dalal, Jorge A. Rodriguez, Marie H. Leeson, Savanna Plombon

Category Research

Concept Diagnostic Uncertainty

Concept Diagnostic Error Risk

Concept Uncertainty Rating

Concept Gold Standard

Concept Diagnostic Certainty

Distinguished Non-Finalist

Presenter Organization jarodriguez1@partners.org

Presenting Author Jorge A. Rodriguez

Track Technology in Hospital Medicine

Keywords

ChatGPT

diagnostic uncertainty

hospital admission notes

human reviewers

Cohen's kappa

clinical decision support

diagnostic error

internal medicine

large language models

medical AI

Diagnostic Uncertainty

Diagnostic Error Risk

Uncertainty Rating

Gold Standard

Diagnostic Certainty