Preprint: Healthcare LLM Agents Miss 32-97% of Required Escalations Despite High Accuracy
New benchmark shows that while clean accuracy is 0.97, uncertainty-prompted models miss most necessary escalations; rule-based controllers improve utility but fail to generalize to unseen paraphrases.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
A new preprint from Washington University in St. Louis reveals critical reliability gaps in how large language models handle escalation decisions within healthcare workflows. In a benchmark of 400 synthetic medical episodes, models were scored on their ability to choose between ANSWER, ASK CLARIFICATION, ABSTAIN, or ESCALATE when facing missing, stale, or conflicting evidence. While the models achieved a clean decision accuracy of 0.97, those prompted to express uncertainty missed 32% to 97% of cases where escalation was required.
The study found that larger models (9.7B parameters) narrowed but did not close this gap, with accuracy dropping to 0.78 under conflict conditions. Verbalized confidence proved inconsistent with actual error rates, and tuning confidence thresholds alone yielded no utility gain.
The researchers introduced a transparent workflow-aware controller that improved utility across all tested models and reduced missed escalations to 0.12–0.24. However, this solution lacks robustness: in a lexical stress test involving unseen paraphrases, the controller's rule detector recall dropped from 1.00 to 0.00, indicating that such routing mechanisms are effective only within the specific benchmark constraints and do not generalize.