SHM Converge Scientific Abstract ePoster Gallery-Comparison of Internal Medicine Residency Application Personal Statements Generated by GPT-4 Versus Authentic Applicants

Pdf Summary

This study examined whether experienced Internal Medicine faculty could distinguish authentic residency personal statements from those generated by GPT-4, and whether perceived authenticity affected how statements were rated.<br /><br />Researchers collected 100 authentic personal statements from de-identified residency applications and generated 100 GPT-4 personal statements using applicants’ CVs and multiple rounds of prompt engineering. Four faculty reviewers with experience evaluating personal statements each blindly reviewed 100 statements total (50 authentic and 50 AI-generated), with each statement reviewed twice. Reviewers scored statements on a 1–10 favorability scale and judged whether they seemed authentic or AI-generated.<br /><br />Key findings:<br />- Reviewers correctly identified authentic versus GPT-4 statements 75% of the time.<br />- Accuracy improved with experience, increasing by about 3.2% for every additional 10 statements reviewed.<br />- Authentic statements were rated higher than GPT-4 statements by 0.73 points on average.<br />- Statements believed to be AI-generated were rated 1.4 points lower than those perceived to be authentic.<br />- The 20 highest-rated statements were all authentic.<br /><br />The authors conclude that experienced faculty can reliably distinguish GPT-4-generated personal statements from human-written ones, and that authenticity strongly influences favorability. However, they note that real-world use may involve a hybrid process where applicants collaborate with AI, which could make detection harder. Despite AI advances, the personal statement remains an important opportunity for applicants to express individuality and agency.

Asset Subtitle

Ashwin Nayak

Meta Tag

Author List Ashwin Nayak, Jason Hom, Kevin Keet, Neera Ahuja, Poonam Hosamani, Vishnu P. Nair, Yingjie Weng

Category Research

Concept Personal Statement

Concept Residency Application

Concept Authenticity

Concept Faculty Reviewer

Concept Favorability Rating

Distinguished Non-Finalist

Presenter Organization Stanford University

Presenting Author Ashwin Nayak

Track Technology in Hospital Medicine

Keywords

GPT-4

personal statements

residency applications

Internal Medicine

authenticity

faculty reviewers

AI-generated text

human-written statements

favorability ratings

medical education

Personal Statement

Residency Application

Authenticity

Faculty Reviewer

Favorability Rating