Research

Health AI Assurance

Federally funded. Active.

The division tests clinical AI tools on Northwell's own patients, before and after they reach the bedside. Its 2026 review of 22 external validations, covering more than 2.3 million patients, found three Epic models performing below the ranges Epic reports.

Pooled external validations of the Epic Deterioration Index, the Epic Sepsis Model, and the Epic Unplanned Readmission Model. Each forest plot gives the AUROC reported by every study, the random-effects estimate, and the AUROC Epic reports.
Pooled external validations of the Epic Deterioration Index, the Epic Sepsis Model, and the Epic Unplanned Readmission Model. Each forest plot gives the AUROC reported by every study, the random-effects estimate, and the AUROC Epic reports. Journal of General Internal Medicine, 2026 (opens in new tab). CC BY 4.0.

Clinical AI is reaching the bedside faster than the evidence needed to trust it. Health systems are offered early-warning models, diagnostic aids, and documentation tools whose performance is reported by their developers and seldom tested independently in the populations where they will be used. The program evaluates these tools on Northwell's own data across the life of a model: retrospective validation before deployment, with calibration, subgroup, and fairness analysis against the current standard of care; prospective evaluation of the effect on patient outcomes and clinician workload; and monitoring for performance drift once a tool is in use.

The case for local validation runs through the division's own record. In 2020 the division tested an interpretable COVID-19 mortality model on Northwell patients and reported in Nature Machine Intelligence that it did not transfer: more than half of the patients it predicted would die survived. In 2026 a systematic review and meta-analysis in the Journal of General Internal Medicine pooled 22 external validations of Epic's clinical decision support tools, covering more than 2.3 million patients at 34 sites. No model exceeded a pooled AUROC of 0.79, and three performed below the ranges Epic reports.

The work now extends to regulated devices. Under ECG-STARS, a contract funded by the U.S. Food and Drug Administration, the division is validating electrocardiogram devices.

How the work was done

  1. The question

    Clinical AI tools reach health systems with performance reported by their developers and seldom tested independently in the populations where they will be used. The program asks whether such tools perform as reported, before and after they reach the bedside.

  2. The data

    The 2026 review searched the PubMed, Scopus and Embase databases, from January 2018 to August 2025, for external validations of Epic's clinical decision support tools. It included 22 studies covering more than 2.3 million patients at 34 sites.

    Journal of General Internal Medicine, 2026

  3. The method

    The review was registered in advance. For each tool, the team pooled the performance reported across studies using a random-effects model, a method that combines results while allowing for differences between sites, and compared the pooled figure with the range Epic reports.

    Journal of General Internal Medicine, 2026

What the work has shown

  • Pooling 22 external validations that cover more than 2.3 million patients at 34 sites, no Epic clinical decision support model exceeded an AUROC of 0.79. The sepsis, readmission, and end-of-life models performed below the ranges Epic reports.

    Journal of General Internal Medicine, 2026

  • Tested on Northwell patients, a published interpretable COVID-19 mortality model predicted death with a precision of 0.48: more than half of the patients it flagged survived.

    Nature Machine Intelligence, 2020

AUROC, also written AUC, is the area under the receiver operating characteristic curve: the probability that a model ranks a patient who had the outcome above one who did not. A value of 0.5 is no better than chance and 1.0 is perfect.

Publications

  1. 2026

    Patel H, Crusco S, Hansen D, et al. A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools. Journal of General Internal Medicine. 2026.

  2. 2020

    Barish M, Bolourani S, Lau LF, Shah S, Zanos TP. External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19. Nature Machine Intelligence. 2020;3(1):25-27.