SHM Converge Scientific Abstract ePoster Gallery-An LLM Pipeline to Elevate the Clinician Voice: Lessons From an Inpatient Ai Chart Summarization Pilot

Pdf Summary

This Stanford Medicine pilot examined how a local, open-source LLM pipeline could help analyze clinician feedback on an inpatient AI chart-summarization tool. Health systems are rapidly deploying LLM-enabled EHR tools, but there is little guidance on how to evaluate them during early rollout. To address this, the team collected free-text feedback from 26 clinicians (physicians/APPs, nurses, and case managers) who generated 783 summaries over two months; 124 summaries included comments.<br /><br />The pipeline automatically split comments into 331 discrete segments, assigned each segment a label and sentiment, and organized them into three major categories: overall tool feedback, task-specific performance, and feature-specific feedback. Low-level codes were generated inductively by the LLM and refined through iterative clinician review. In total, 44 low-level codes were produced.<br /><br />Results showed predominantly positive sentiment about the tool’s overall utility. Clinicians especially valued care team identification and the citation feature. However, the analysis also identified important weaknesses, particularly in temporal reasoning, organizational structure, and performance on certain clinical tasks.<br /><br />A review of summary types showed that 62% of feedback was judged helpful without errors overall, though some clinically impactful errors remained. The authors note that the inductive coding process was imperfect and sometimes produced duplicate codes, but the structured output still provided useful insight into tool strengths, weaknesses, and interim guidance for clinicians.<br /><br />Overall, the study suggests that LLM-assisted thematic analysis can turn large volumes of clinician free text into actionable, quantitative feedback signals to improve health IT evaluation and deployment.

Asset Subtitle

Stephen P. Ma

Meta Tag

Author List April S. Liang, Jonathan Colston, Matthew A. Eisenberg, Olivia Aparicio-Kratz, Stephen P. Ma

Category Innovations

Concept Evaluation

Concept Inpatient AI Chart-Summarization Tool

Concept Clinician Feedback

Concept Feedback Analysis

Concept Clinician Review

Distinguished Finalist

Presenter Organization Stanford University School of Medicine

Presenting Author Stephen P. Ma

Track Technology in Hospital Medicine

Keywords

Stanford Medicine

LLM-assisted thematic analysis

clinician feedback

inpatient AI chart summarization

open-source LLM pipeline

EHR tools

sentiment analysis

care team identification

temporal reasoning

health IT evaluation

Evaluation

Inpatient AI Chart-Summarization Tool

Clinician Feedback

Feedback Analysis

Clinician Review