AI Surpasses Doctors in High-Stakes ER Reasoning: Harvard Study Signals New Era in Clinical AI

A groundbreaking study published on April 30, 2026, in the journal Science has demonstrated that OpenAI’s o1-preview reasoning model significantly outperforms experienced physicians in multiple clinical reasoning tasks, including real-world emergency department cases. Led by researchers from Harvard Medical School and Beth Israel Deaconess Medical Center (BIDMC), with collaborators at Stanford, the research represents one of the most rigorous comparisons of advanced AI against human clinicians to date.

Study Design and Scope

The team evaluated the o1-preview model across six distinct clinical reasoning experiments. These included standardized benchmarks (such as New England Journal of Medicine clinicopathological conferences) and, crucially, unstructured data from 76 real emergency department cases from BIDMC in Boston. Cases were assessed at three critical touchpoints reflecting real clinical workflow:

• Initial ER triage (minimal information available, highest urgency)
• ER physician evaluation
• Admission to the medical floor or ICU (more data accumulated)

Two attending physicians provided baselines for comparison. Differential diagnoses generated by both AI and humans were evaluated by two additional blinded attending physicians, who could not reliably distinguish AI outputs from human ones. Additional experiments tested diagnostic accuracy on complex vignettes, management reasoning (choosing next steps and treatments), probabilistic reasoning, and performance against earlier models like GPT-4o.

Key ResultsEmergency Room Triage Performance

At the earliest and most critical stage with the least information:

• o1-preview identified the exact or very close diagnosis in 67.1% of cases.
• Two attending physicians scored 55.3% and 50.0%, respectively.

Accuracy improved for all as more data became available. At admission:

• AI reached 81.6%.
• Physicians scored 78.9% and 69.7%.

The AI’s edge was most pronounced in data-scarce, high-pressure early stages.

Structured Treatment Planning and Management

In management reasoning tasks, the AI achieved a median score of 89 out of 100 — more than doubling the performance of physicians using conventional resources or even AI assistance in some comparisons. It excelled at suggesting appropriate diagnostic tests and next steps.

Broader Clinical Reasoning

Across other experiments (including NEJM cases), o1-preview consistently matched or exceeded physician baselines and prior AI models. In one evaluation, it provided perfect clinical reasoning explanations in 98% of cases, compared to 35% for attending physicians. It also demonstrated superior probabilistic reasoning (e.g., estimating likelihoods of conditions like cardiac ischemia).

Limitations and Researcher Cautions

The study was strictly text-based, relying on electronic health record data without incorporating physical exams, imaging interpretation, real-time vital signs, patient appearance, or bedside nuances. No actual patient care was involved. Researchers emphasize that strong performance in controlled settings does not automatically translate to improved real-world outcomes. They explicitly call for prospective clinical trials to evaluate safety, integration into workflows, impact on patient outcomes, and potential risks such as over-reliance or errors in edge cases.

Implications

This research suggests that advanced reasoning LLMs have now “eclipsed most benchmarks of clinical reasoning.” It positions AI as a powerful potential “second opinion” tool that could reduce diagnostic errors, particularly in resource-limited or high-volume settings. However, the authors and experts stress a collaborative future: AI augmenting, rather than replacing, human clinicians.

Citations

• Brodeur et al. (2026). “Performance of a large language model on the reasoning…”. Science. DOI: 10.1126/science.adz4433.
• Harvard Medical School / BIDMC Press (April 30, 2026).
• Coverage in Harvard Magazine, The Guardian, Science, NPR, and others.

This study marks a pivotal moment in medical AI, highlighting both remarkable progress and the urgent need for careful, evidence-based implementation.