LLMs for Medical Consultation Are Evaluated Too Late: The Preformulation Gap

Research on LLMs for medical consultation identifies a preformulation gap, noting that models are typically evaluated after clinical problems are clear. Testing three API models using physician-authored vignettes and standardized-patient simulations revealed that baseline conditions often led to premature self-care advice. However, implementing entry-to-care instructions eliminated this issue and significantly increased the production of structured handoff summaries. The study emphasizes the importance of evaluating models during the initial, often vague, stages of patient interaction.