Publications

Background: Clinical Decision Support (CDS) tools integrated with Electronic Health Records increasingly guide clinical practice. Epic Systems, storing more than 325 million patient records, offers various proprietary predictive models to healthcare systems. Despite widespread adoption, no systematic review has examined these tools' real-world performance compared to vendor-reported metrics. Methods: This study was prospectively registered on PROSPERO (CRD420251148571). We systematically searched PubMed, Scopus, and Embase (January 2018 to August 2025) for external validations of Epic's CDS tools. We pooled Area Under the Receiver Operating Characteristic Curve (AUROC) values using random-effects models and assessed heterogeneity using Higgins' I2. Results: We included 22 studies in our systematic review, validating Epic CDS tools on a total of over 2.3 million patients and 34 sites. The Epic Deterioration Index (EDI, 7 studies, 542,983 patients) achieved pooled AUROC 0.79 [95% CI 0.76-0.80]; Epic Sepsis Model (ESM, 3 studies, 922,754 patients) 0.65 [0.61-0.70]; Epic Unplanned Readmission Model (EURM, 5 studies, 145,595 patients) 0.70 [0.68-0.73]; Epic End-of-Life Care Index (EEOL-CI, 3 studies, 217,885 patients) 0.76 [0.67-0.83]; and Epic Risk of Patient No-Show (ERPNS, 2 studies, 93,863 patients) 0.62 [0.54-0.68]. All models showed high heterogeneity (I2 ≥ 93%, p < 0.001). For ESM, EURM, and EEOL-CI, Epic's reported confidence intervals did not overlap with our pooled estimates, with Epic reporting consistently higher performance. Two additional models (ED-to-Inpatients and ICU Mortality) had one study each. Conclusions: Epic's CDS tools demonstrated modest real-world performance, with none exceeding AUROC 0.79. Three models (ESM, EURM, EEOL-CI) underperformed Epic's reported ranges. High heterogeneity across sites emphasizes the need for local validation before clinical deployment.

External validation of a previously published interpretable mortality prediction model for COVID-19 patients was conducted. The study demonstrated that the model does not perform as a triage tool based on the internal validation dataset provided by the original authors. The decision algorithm was not portable to the external validation dataset, both with unmodified and optimized parameters. Specifically, the precision was 0.48 for predicting mortality, meaning that over half of the patients that the model predicted would die actually survived. The accuracy was 0.88 and the F1 score was 0.41. These results emphasized the importance of externally validating models before their widespread adoption in actual clinical practice.