Health AI Assurance
Federally funded. Active.
The division tests clinical AI tools on Northwell's own patients, before and after they reach the bedside. Its 2026 review of 22 external validations, covering more than 2.3 million patients, found three Epic models performing below the ranges Epic reports.

Clinical AI is reaching the bedside faster than the evidence needed to trust it. Health systems are offered early-warning models, diagnostic aids, and documentation tools whose performance is reported by their developers and seldom tested independently in the populations where they will be used. The program evaluates these tools on Northwell's own data across the life of a model: retrospective validation before deployment, with calibration, subgroup, and fairness analysis against the current standard of care; prospective evaluation of the effect on patient outcomes and clinician workload; and monitoring for performance drift once a tool is in use.
The case for local validation runs through the division's own record. In 2020 the division tested an interpretable COVID-19 mortality model on Northwell patients and reported in Nature Machine Intelligence that it did not transfer: more than half of the patients it predicted would die survived. In 2026 a systematic review and meta-analysis in the Journal of General Internal Medicine pooled 22 external validations of Epic's clinical decision support tools, covering more than 2.3 million patients at 34 sites. No model exceeded a pooled AUROC of 0.79, and three performed below the ranges Epic reports.
The work now extends to regulated devices. Under ECG-STARS, a contract funded by the U.S. Food and Drug Administration, the division is validating electrocardiogram devices.
How the work was done
The question
Clinical AI tools reach health systems with performance reported by their developers and seldom tested independently in the populations where they will be used. The program asks whether such tools perform as reported, before and after they reach the bedside.
The data
The 2026 review searched the PubMed, Scopus and Embase databases, from January 2018 to August 2025, for external validations of Epic's clinical decision support tools. It included 22 studies covering more than 2.3 million patients at 34 sites.
The method
The review was registered in advance. For each tool, the team pooled the performance reported across studies using a random-effects model, a method that combines results while allowing for differences between sites, and compared the pooled figure with the range Epic reports.
What the work has shown
Pooling 22 external validations that cover more than 2.3 million patients at 34 sites, no Epic clinical decision support model exceeded an AUROC of 0.79. The sepsis, readmission, and end-of-life models performed below the ranges Epic reports.
Tested on Northwell patients, a published interpretable COVID-19 mortality model predicted death with a precision of 0.48: more than half of the patients it flagged survived.
AUROC, also written AUC, is the area under the receiver operating characteristic curve: the probability that a model ranks a patient who had the outcome above one who did not. A value of 0.5 is no better than chance and 1.0 is perfect.
Publications
- 2026
Patel H, Crusco S, Hansen D, et al. A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools. Journal of General Internal Medicine. 2026.
- 2020
Barish M, Bolourani S, Lau LF, Shah S, Zanos TP. External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19. Nature Machine Intelligence. 2020;3(1):25-27.
Figures from the papers
Reproduced from the papers behind this project. Each keeps its source and license.
