AI-REACT: AI Assisted Reader Evaluation in Acute Computed Tomography (CT) Head Interpretation

Sponsor

Oxford University Hospitals NHS Trust (Other)

Overall Status

Recruiting

CT.gov ID

NCT06018545

Collaborator

(none)

Enrollment

Locations

Anticipated Duration (Months)

7.5

Patients Per Site

0.5

Patients Per Site Per Month

Study Details

Study Description

Brief Summary

This study has been added as a sub study to the Simulation Training for Emergency Department Imaging 2 study (ClinicalTrials.gov ID NCT05427838).

The purpose of the study is to assess the impact of an Artificial Intelligence (AI) tool called qER 2.0 EU on the performance of readers, including general radiologists, emergency medicine clinicians, and radiographers, in interpreting non-contrast CT head scans. The study aims to evaluate the changes in accuracy, review time, and diagnostic confidence when using the AI tool. It also seeks to provide evidence on the diagnostic performance of the AI tool and its potential to improve efficiency and patient care in the context of the National Health Service (NHS). The study will use a dataset of 150 CT head scans, including both control cases and abnormal cases with specific abnormalities. The results of this study will inform larger follow-up studies in real-life Emergency Department (ED) settings.

Condition or Disease	Intervention/Treatment	Phase
Intracranial Hemorrhages Acute Ischemic Stroke Hydrocephalus Cerebral Infarction Cerebral Edema Cerebral Injury	Other: Ground truthing Other: Reading

Study Design

Study Type:

Observational

Anticipated Enrollment :

30 participants

Observational Model:

Cohort

Time Perspective:

Retrospective

Official Title:

AI Assisted Reader Evaluation in Acute CT Head Interpretation

Actual Study Start Date :

Jun 1, 2023

Anticipated Primary Completion Date :

Sep 29, 2023

Anticipated Study Completion Date :

Sep 1, 2024

Arms and Interventions

Arm	Intervention/Treatment
Readers 30 readers will be recruited across four NHS trusts including ten general radiologists, fifteen emergency medicine clinicians, and five CT radiographers of varying seniority. Readers will interpret each scan first without, then with, the assistance of the AI tool, with an intervening 4-week washout period. Using a panel of neuroradiologists as ground truth, the stand-alone performance of qER will be assessed, and its impact on the readers' performance will be analysed as change in accuracy, mean review time per scan, and self-reported diagnostic confidence. Subgroup analyses will be performed by reader professional group, reader seniority, pathological finding, and neuroradiologist-rated difficulty.	Other: Reading All 30 readers will review all 150 cases, in each of two study phases. The readers will provide their opinion on the presence or absence of some acute abnormalities, including intracranial haemorrhage, infarct, midline shift and fracture. They will provide a confidence in their diagnosis (10-point visual analogue scale), and a single click point to mark the location of each abnormality that they consider as being present. The time taken for each scan will be automatically recorded.
Ground truthers Two Consultant neuroradiologists will independently review the images to establish the 'ground truth' findings on the CT scans which will be used as the reference standard. In the case of disagreement, a third senior neuroradiologist's opinion will be sought for arbitration. A difficulty score will be assigned to each scan by the ground truthers using a 5-point Likert scale.	Other: Ground truthing Two Consultant neuroradiologists will independently review the images to establish the 'ground truth' findings on the CT scans which will be used as the reference standard. In the case of disagreement, a third senior neuroradiologist's opinion will be sought for arbitration.

Arm

Intervention/Treatment

Readers

30 readers will be recruited across four NHS trusts including ten general radiologists, fifteen emergency medicine clinicians, and five CT radiographers of varying seniority. Readers will interpret each scan first without, then with, the assistance of the AI tool, with an intervening 4-week washout period. Using a panel of neuroradiologists as ground truth, the stand-alone performance of qER will be assessed, and its impact on the readers' performance will be analysed as change in accuracy, mean review time per scan, and self-reported diagnostic confidence. Subgroup analyses will be performed by reader professional group, reader seniority, pathological finding, and neuroradiologist-rated difficulty.

Other: Reading

All 30 readers will review all 150 cases, in each of two study phases. The readers will provide their opinion on the presence or absence of some acute abnormalities, including intracranial haemorrhage, infarct, midline shift and fracture. They will provide a confidence in their diagnosis (10-point visual analogue scale), and a single click point to mark the location of each abnormality that they consider as being present. The time taken for each scan will be automatically recorded.

Ground truthers

Two Consultant neuroradiologists will independently review the images to establish the 'ground truth' findings on the CT scans which will be used as the reference standard. In the case of disagreement, a third senior neuroradiologist's opinion will be sought for arbitration. A difficulty score will be assigned to each scan by the ground truthers using a 5-point Likert scale.

Other: Ground truthing

Outcome Measures

Primary Outcome Measures

Reader performance: Sensitivity, specificity, comparative between with and without AI assistance. [During 6 weeks, which is the period for reading or reviewing the cases/scans.]
Reader performance will be evaluated as sensitivity, specificity, with and without AI assistance.
Reader performance: Positive and negative predictive value, comparative between with and without AI assistance. [During 6 weeks, which is the period for reading or reviewing the cases/scans.]
Reader performance will be evaluated as Positive Predictive Value (PPV) and negative predictive value (NPV), with and without AI assistance.
Reader performance: Area Under Receiver Operating Characteristic Curve (AUROC), comparative between with and without AI assistance. [During 6 weeks, which is the period for reading or reviewing the cases/scans.]
Reader performance will be evaluated as Area Under Receiver Operating Characteristic Curve (AUROC), with and without AI assistance.
Reader speed: Mean time taken to review a scan, with versus without AI assistance. [During 6 weeks, which is the period for reading or reviewing the cases/scans.]
Reader speed will be evaluated as the man time taken to review a scan, using time unite of seconds.
Reader confidence: Self-reported diagnostic confidence on a 10 point visual analogue scale, with vs without AI assistance. [During 6 weeks, which is the period for reading or reviewing the cases/scans.]
On the reading platform (RAIQC), one of the questions asks the level of confidence that the participant has in their diagnostic opinion. The question offers a scale of 1 to 10, where 1 is not confident, and 10 is highly confident.
qER (AI algorithm) performance: Sensitivity and specificity [During 6 weeks, which is the period for reading or reviewing the cases/scans.]
qER performance will be evaluated as sensitivity, specificity.
qER (AI algorithm) performance: Positive and negative predictive value. [During 6 weeks, which is the period for reading or reviewing the cases/scans.]
qER performance will be evaluated as Positive Predictive Value (PPV) and negative predictive value (NPV).
qER (AI algorithm) performance: Area Under Receiver Operating Characteristic Curve (AUROC). [During 6 weeks, which is the period for reading or reviewing the cases/scans.]
qER performance will be evaluated as Area Under Receiver Operating Characteristic Curve (AUROC)

Eligibility Criteria

Criteria

Ages Eligible for Study:

N/A and Older

Sexes Eligible for Study:

All

Accepts Healthy Volunteers:

Yes

Inclusion Criteria:

Radiologists/Radiographers/ED clinicians who review CT head scans as part of their clinical practice

Exclusion Criteria:

Neuroradiologists.
Non-radiologist groups: Clinicians with previous formal postgraduate CT reporting training
Emergency Medicine group: Clinicians with previous career in radiology/neurosurgery to registrar level

Contacts and Locations

Locations

	Site	City	State	Country	Postal Code
1	Oxford University Hospitals NHS Foundation Trust	Oxford	Oxfordshire	United Kingdom	OX3 9DU
2	NHS Greater Glasgow and Clyde	Glasgow		United Kingdom	G12 0XH
3	Guy's & St Thomas NHS Foundation Trust	London		United Kingdom	SE1 7EH
4	Northumbria Healthcare NHS Foundation Trust	Newcastle Upon Tyne		United Kingdom	NE27 0QJ

Sponsors and Collaborators

Oxford University Hospitals NHS Trust

Investigators

Principal Investigator: Alex Novak, MSc, National Health Services in the United Kingdom (NHS UK)
Principal Investigator: Sarim Ather, PhD, National Health Services in the United Kingdom (NHS UK)