SHM Converge Scientific Abstract ePoster Gallery-Performance of an Ehr-Embedded Prediction Algorithm for Identifying Diagnostic Errors

Pdf Summary

Researchers evaluated an EHR-embedded prediction algorithm designed to identify diagnostic errors (DEs) in hospitalized patients, since current surveillance methods likely miss many harmful DEs. The algorithm, integrated into Epic and running every 15 minutes, used real-time EHR data to assign patients to low, moderate, or high risk based on the proportion of time their flag was green, yellow, or red during the hospital stay.<br /><br />In a prospective study of general medicine patients at a community hospital in Boston (July–December 2022), 1,721 eligible cases were identified. From these, 175 cases were randomly sampled for detailed review. Two expert clinicians independently assessed diagnostic uncertainty (DU), likelihood of DE using the Safer Dx instrument, and associated harm. The risk category was also adjusted upward or downward depending on whether the admission or discharge diagnosis showed uncertainty or certainty.<br /><br />Among sampled cases, 33 DEs (18.9%) and 16 harmful DEs (9.1%) were found, corresponding to population-weighted estimates of 12.5% and 7.5%, respectively. For harmful DEs, using a high-risk vs. moderate/low-risk cutoff, the algorithm showed sensitivity of 0.63, specificity of 0.50, PPV of 0.11, and NPV of 0.93. After adjusting for diagnostic uncertainty, performance improved to sensitivity 0.75, specificity 0.64, PPV 0.17, and NPV 0.96. Similar analyses were performed for all DEs and using an alternative cutoff.<br /><br />Overall, the algorithm had low PPV but high NPV, suggesting it may be useful for ruling out patients unlikely to have DEs and narrowing the group needing more detailed Safer Dx review. Incorporating diagnostic uncertainty may improve performance further, and large language models may help detect DU and other DE risk factors from clinical notes. Limitations include the single-center design and exploratory nature of DU assessment.

Asset Subtitle

Abhishek Goyal

Meta Tag

Author List Abhishek Goyal, Anuj K. Dalal, Jorge A. Rodriguez, Marie H. Leeson, Savanna Plombon, Stuart Lipsitz

Category Research

Concept Prediction Algorithm

Concept Diagnostic Error

Concept Electronic Health Record

Concept Risk Screening

Concept Harmful Diagnostic Error

Distinguished Non-Finalist

Presenter Organization Brigham and Women's Hospital

Presenting Author Abhishek Goyal

Track Technology in Hospital Medicine

Keywords

electronic health record

diagnostic errors

prediction algorithm

hospitalized patients

diagnostic uncertainty

Safer Dx

harmful diagnostic errors

risk stratification

clinical decision support

Epic EHR

Prediction Algorithm

Diagnostic Error

Electronic Health Record

Risk Screening

Harmful Diagnostic Error