MIRACLE: Evaluation of Clinical Implementation of Machine Learning Based Decision Support for ICU Discharge
Study Details
Study Description
Brief Summary
Unexpected intensive care unit (ICU) readmission is associated with longer length of stay and increased mortality. Bedside decision support may prevent readmission and mortality and may allow optimizing ICU capacity. Using a recently developed and prospectively validated machine learning model that predicts ICU readmission and mortality rate after ICU discharge and shows trends in these predictions over time, we will evaluate the implementation of the European conformity (CE)-marked software based on this model (Pacmed Critical, Pacmed, Amsterdam) by investigating whether the software improves diagnostic accuracy compared to routine clinical evaluation by the treatment team and whether availability of the information from this software leads to changes in discharge management (either postponing or advancing discharge) for patients considered eligible for discharge.
| Condition or Disease | Intervention/Treatment | Phase |
|---|---|---|
|
Detailed Description
Rationale: Unexpected intensive care unit (ICU) readmission is associated with longer length of stay and increased mortality. Bedside decision support may prevent readmission and mortality and may allow optimizing ICU capacity. Several attempts to develop prediction models to prevent ICU readmission and/or death after discharge from the ICU for general adult critical care patients have been made previously. Although the performance of Machine Learning models versus physicians has been studied for diagnosing in medical imaging, there is scarce literature prospectively comparing physician's predictive performance when it comes to patient outcomes. In addition, currently, no readmission model is widely implemented nor tested to support ICU discharge
Aim: Using a recently developed and prospectively validated machine learning model that predicts ICU readmission and mortality rate after ICU discharge and shows trends in these predictions over time, we will evaluate the implementation of the European conformity (CE)-marked software based on this model (Pacmed Critical, Pacmed, Amsterdam) by investigating whether the software improves diagnostic accuracy compared to routine clinical evaluation by the treatment team and whether availability of the information from this software leads to changes in discharge management (either postponing or advancing discharge) for patients considered eligible for discharge. In addition, since this is a novel approach in supporting discharge decision support, information will be collected from end-users with respect to interpretability and usability. Furthermore, model and software improvement will take place during this pilot phase, e.g. with respect to out-of-distribution detection for recognizing patients that are insufficiently similar to the data the model was developed on. Results from this study will be used to develop a clinical trial to evaluate effect on readmission rate and/or mortality after ICU discharge, if considered feasible, based on the effect the software has on potentially changing intensivist decisions, and the estimated effect on readmission and mortality during the On-period.
Design: Before-and-after pilot implementation study.
For this evaluation, data will be collected both in the periods in which the Pacmed Critical software will not be available to end-users (Off-period, 3-6 months) and during the actual implementation phase where end-users are able to use the software at potential ICU discharge (On-period, 3-6 months). After the implementation phase an additional Off-period (3-6 months) will follow.
After the morning hand-off procedure the treatment team consisting of intensivists, fellows in intensive care medicine, medical residents, ICU nurses, and consulting medical specialists ('treatment team'), will determine which patients appear to be eligible for discharge to the nursing (non-ICU) ward. For those patients, the attending intensivist will digitally document the following:
For both On- and Off-periods:
-
'ready-for-discharge' status, based on the collective evaluation by the treatment team, taking into account the care that can be provided by the receiving ward based on local ICU discharge protocols. Patients that were initially considered 'eligible for ICU discharge' may thus ultimately be considered and documented as 'not ready-for-discharge'.
-
destination nursing ward
-
prediction for risk of readmission and/or mortality within 7 days (scale 0-100%), assuming the patient would be discharged
-
main factors contributing to that decision
-
Self-reporting of confidence of estimation (low-medium-high).
-
For patients with a 'ready-for-discharge' decision that were not transferred, at the end of day, to the regular ward the reason for that:
-
'Clinical deterioration'
-
'Insufficient bed capacity nursing ward'
-
'Insufficient isolation capacity nursing ward'
Additionally, during On-periods after reviewing the additional information from Pacmed Critical by the treatment team, the previous questions will be asked again to evaluate if re-evaluation with decision support had effect on that decision, i.e. the 'ready-for-discharge' status was changed.
During every period the final decision to discharge patients from the ICU is at the discretion of the lead unit intensivist responsible for the medical care of those patients and could change based on alterations in clinical condition of the patient (e.g. deterioration) and/or reasons that require re-evaluation of patients eligible for discharge, including the need to admit other patients.
Pseudonymized near real-time data will be extracted in a combined production/research database to perform predictions. The predictions accessed by end-users will be filed together with the additional data collected as specified above. In addition the predicted endpoint (ICU readmission and mortality within 7 days after discharge) will be collected for all patients actually discharged from the ICU.
Depending on whether the participating hospital has already passed the technical implementation (i.e. passed device interface and end-user acceptance) after start of the first Off-period Pacmed Critical will be either used prospectively to make the predictions and store the results at the moment of study documentation of the attending intensivist, or retrospectively. The On-period can only commence after the hospital has fully passed technical implementation in accordance with the CE-documentation.
Study Design
Arms and Interventions
| Arm | Intervention/Treatment |
|---|---|
| Discharged patients with decision support (On-period) For patients that have been evaluated as eligible for discharge: the current ICU discharge process will be followed based on routine clinical evaluation by the treatment team in combination with ICU discharge protocols. In addition, Pacmed Critical will be used as an additional source of information. Final discharge decision will be made by lead unit intensivist responsible for medical care. |
Device: Pacmed Critical
For patients in the On-period, Pacmed Critical will be available as decision support after initial eligibility screening for ICU discharge by treatment team
|
| Discharged patients without decision support (Off-period) For patients that have been evaluated as eligible for discharge: the current ICU discharge process will be followed based on routine clinical evaluation by the treatment team in combination with ICU discharge protocols. Final discharge decision will be made by lead unit intensivist responsible for medical care. |
Outcome Measures
Primary Outcome Measures
- area under the receiver operating characteristic curve (AUROC) [7 days after ICU discharge]
comparison of AUROC between Pacmed Critical model and intensivists estimation in predicting ICU readmission and/or mortality within 7 days following ICU discharge
- calibration curve (goodness-of-fit) [7 days after ICU discharge]
comparison of calibration curves (binned estimations) of Pacmed Critical model and intensivists estimation in predicting ICU readmission and/or mortality within 7 days following ICU discharge
Secondary Outcome Measures
- Number of changes in ready-for-discharge decision after reviewing decision support [through study completion (estimated 1 year)]
Change of ready-for-discharge decision after review of decision support software Pacmed Critical
- Readmission rate within 7 days after ICU discharge [7 days after ICU discharge]
Comparison of outcome between On an Off-periods
- Mortality rate within 7 days after ICU discharge [7 days after ICU discharge]
Comparison of outcome between On an Off-periods
- Length of ICU stay [up to 90 days after ICU admission]
Comparison of outcome between On an Off-periods
- Length of hospital stay [up to 90 days after hospital admission]
Comparison of outcome between On an Off-periods
- Estimation of intra-cluster correlation [through study completion (estimated 1 year)]
Estimation of intra-cluster correlation
- Average score on the 3-point Likert-scale 'confidence of risk estimation' with and without decision support [through study completion (estimated 1 year)]
Evaluate whether decision support has effect on 'confidence of risk estimation'
- Number of risk determinants, categorized by organ system as determined by physicians vs model [through study completion (estimated 1 year)]
Differences between physician derived risk and by model derived determinants using Shapley additive explanations (SHAP)
- Software usage metrics [through study completion (estimated 1 year)]
Time spent on user interface (UI) elements
Eligibility Criteria
Criteria
Inclusion Criteria:
-
Admission to intensive care or medium care unit
-
Age >= 18 years
-
ICU admission > 4 hours
-
Eligible for discharge at the discretion of the treatment team by not requiring treatment that can only be provided on the ICU (including but not limited to mechanical ventilation, high flow oxygen, vasopressor/inotropes, continuous renal replacement therapy).
Exclusion Criteria:
-
No-return (to ICU/MCU) policy and/or palliative/end-of-life care
-
Coronavirus disease (COVID)-19
-
Patients directly transferred to other hospitals after discharge
Contacts and Locations
Locations
| Site | City | State | Country | Postal Code | |
|---|---|---|---|---|---|
| 1 | Amsterdam UMC, location VUmc | Amsterdam | NH | Netherlands | 1081HV |
| 2 | Leiden University Medical Center (LUMC) | Leiden | ZH | Netherlands | 2333 ZA |
Sponsors and Collaborators
- Patrick J. Thoral
- Leiden University Medical Center
Investigators
- Study Director: Patrick J Thoral, MD, Amsterdam UMC, location VUmc
Study Documents (Full-Text)
None provided.More Information
Additional Information:
Publications
- Thoral PJ, Fornasa M, de Bruin DP, Tonutti M, Hovenkamp H, Driessen RH, Girbes ARJ, Hoogendoorn M, Elbers PWG. Explainable Machine Learning on AmsterdamUMCdb for ICU Discharge Decision Support: Uniting Intensivists and Data Scientists. Crit Care Explor. 2021 Sep 10;3(9):e0529. doi: 10.1097/CCE.0000000000000529. eCollection 2021 Sep.
- Thoral PJ, Peppink JM, Driessen RH, Sijbrands EJG, Kompanje EJO, Kaplan L, Bailey H, Kesecioglu J, Cecconi M, Churpek M, Clermont G, van der Schaar M, Ercole A, Girbes ARJ, Elbers PWG; Amsterdam University Medical Centers Database (AmsterdamUMCdb) Collaborators and the SCCM/ESICM Joint Data Science Task Force. Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example. Crit Care Med. 2021 Jun 1;49(6):e563-e577. doi: 10.1097/CCM.0000000000004916.
- 2021.0528