AI Assisted Detection of Fractures on X-Rays (FRACT-AI)

Sponsor

Oxford University Hospitals NHS Trust (Other)

Overall Status

Not yet recruiting

CT.gov ID

NCT06130397

Collaborator

Gleamer (Industry)

Enrollment

Location

Anticipated Duration (Months)

5.3

Patients Per Site Per Month

Study Details

Study Description

Brief Summary

This study has been added as a sub study to the Simulation Training for Emergency Department Imaging 2 study (ClinicalTrials.gov ID NCT05427838). This work aims to evaluate the impact of an Artificial Intelligence (AI)-enhanced algorithm called Boneview on the diagnostic accuracy of clinicians in the detection of fractures on plain XR (X-Ray). The study will create a dataset of 500 plain X-Rays involving standard images of all bones other than the skull and cervical spine, with 50% normal cases and 50% containing fractures. A reference 'ground truth' for each image to confirm the presence or absence of a fracture will be established by a senior radiologist panel. This dataset will then be inferenced by the Gleamer Boneview algorithm to identify fractures. Performance of the algorithm will be compared against the reference standard. The study will then undertake a Multiple-Reader Multiple-Case study in which clinicians interpret all images without AI and then subsequently with access to the output of the AI algorithm. 18 clinicians will be recruited as readers with 3 from each of six distinct clinical groups: Emergency Medicine, Trauma and Orthopedic Surgery, Emergency Nurse Practitioners, Physiotherapy, Radiology and Radiographers, with three levels of seniority in each group. Changes in reporting accuracy (sensitivity, specificity), confidence, and speed of readers in two sessions will be compared. The results will be analyzed in a pooled analysis for all readers as well as for the following subgroups: Clinical role, Level of seniority, Pathological finding, Difficulty of image. The study will demonstrate the impact of an AI interpretation as compared with interpretation by clinicians, and as compared with clinicians using the AI as an adjunct to their interpretation. The study will represent a range of professional backgrounds and levels of experience among the clinical element. The study will use plain film x-rays that will represent a range of anatomical views and pathological presentations, however x-rays will present equal numbers of pathological and non-pathological x-rays, giving equal weight to assessment of specificity and sensitivity. Ethics approval has already been granted, and the study will be disseminated through publication in peer-reviewed journals and presentation at relevant conferences.

Condition or Disease	Intervention/Treatment	Phase
Fracture Bone Fracture Dislocation Fracture Multiple Fractures, Closed Fractures, Open	Other: Cases reading Other: Ground truthing

Study Design

Study Type:

Observational

Anticipated Enrollment :

21 participants

Observational Model:

Cohort

Time Perspective:

Retrospective

Official Title:

FRACT-AI: Evaluating the Impact of Artificial Intelligence-Enhanced Image Analysis on the Diagnostic Accuracy of Frontline Clinicians in the Detection of Fractures on Plain X-Ray

Anticipated Study Start Date :

Nov 30, 2023

Anticipated Primary Completion Date :

Feb 28, 2024

Anticipated Study Completion Date :

Mar 30, 2024

Arms and Interventions

Arm	Intervention/Treatment
Readers/participants Reader Selection: 18 readers will be selected from the following five clinical specialty groups (3 readers each): Emergency Medicine Trauma and Orthopaedic Surgery Emergency Nurse Practitioners Physiotherapy General Radiology Radiographers And from the following level of seniority/experience: Consultant/Senior/Equivalent - >10yrs experience Middle Grade/Registrar/Equivalent - 5-10yrs experience Junior Grade/Senior House Officer/Equivalent - <5yrs experience Each specialty reader group will include 1 reader at each level of experience. Readers will be recruited from across 5 NHS organisations which comprise the Thames Valley Emergency Medicine Research Network (www.TaVERNresearch.org): Oxford University Hospitals NHS Foundation Trust Royal Berkshire NHS Foundation Trust Buckinghamshire Healthcare NHS Trust Frimley Health NHS Foundation Trust Milton Keynes University Hospital NHS Foundation Trust	Other: Cases reading The reading will be done remotely via the Report and Image Quality Control site (www.RAIQC.com), an online platform allowing medical imaging viewing and reporting. Participants can work from any location, but the work must be done from a computer with internet access. For avoidance of doubt, the work cannot be performed from a phone or tablet. The project is divided into two phases and participants are required to complete both phases. The estimated total involvement in the project is up to 20-24 hours. Phase 1: Time allowed: 2 weeks - Participants must review 500 X-rays and express a clinical opinion through a structured reporting template (multiple choice, no open text required). Rest/washout period - Time allowed: 4 weeks, to mitigate the effects of recall bias. Phase 2 - Time allowed: 2 weeks - Review 500 X-rays together with an AI report for each case and express their clinical opinion through the same structured reporting template used in Phase 1.
Ground truthers Two consultant musculoskeletal radiologists. A third senior musculoskeletal radiologist's opinion (>20 years experience) will undertake arbitration.	Other: Ground truthing Two consultant musculoskeletal radiologists will independently review the images to establish the 'ground truth' findings on the XRs, where a consensus is reached this will then be used as the reference standard. In the case of disagreement, a third senior musculoskeletal radiologist's opinion (>20 years experience) will undertake arbitration. A difficulty score will be assigned to each abnormality by the ground truthers using a 4-point Likert scale (1 being easy/obvious to 4 being hard/poorly visualised).

Arm

Intervention/Treatment

Readers/participants

Reader Selection: 18 readers will be selected from the following five clinical specialty groups (3 readers each): Emergency Medicine Trauma and Orthopaedic Surgery Emergency Nurse Practitioners Physiotherapy General Radiology Radiographers And from the following level of seniority/experience: Consultant/Senior/Equivalent - >10yrs experience Middle Grade/Registrar/Equivalent - 5-10yrs experience Junior Grade/Senior House Officer/Equivalent - <5yrs experience Each specialty reader group will include 1 reader at each level of experience. Readers will be recruited from across 5 NHS organisations which comprise the Thames Valley Emergency Medicine Research Network (www.TaVERNresearch.org): Oxford University Hospitals NHS Foundation Trust Royal Berkshire NHS Foundation Trust Buckinghamshire Healthcare NHS Trust Frimley Health NHS Foundation Trust Milton Keynes University Hospital NHS Foundation Trust

Other: Cases reading

The reading will be done remotely via the Report and Image Quality Control site (www.RAIQC.com), an online platform allowing medical imaging viewing and reporting. Participants can work from any location, but the work must be done from a computer with internet access. For avoidance of doubt, the work cannot be performed from a phone or tablet. The project is divided into two phases and participants are required to complete both phases. The estimated total involvement in the project is up to 20-24 hours. Phase 1: Time allowed: 2 weeks - Participants must review 500 X-rays and express a clinical opinion through a structured reporting template (multiple choice, no open text required). Rest/washout period - Time allowed: 4 weeks, to mitigate the effects of recall bias. Phase 2 - Time allowed: 2 weeks - Review 500 X-rays together with an AI report for each case and express their clinical opinion through the same structured reporting template used in Phase 1.

Ground truthers

Two consultant musculoskeletal radiologists. A third senior musculoskeletal radiologist's opinion (>20 years experience) will undertake arbitration.

Other: Ground truthing

Two consultant musculoskeletal radiologists will independently review the images to establish the 'ground truth' findings on the XRs, where a consensus is reached this will then be used as the reference standard. In the case of disagreement, a third senior musculoskeletal radiologist's opinion (>20 years experience) will undertake arbitration. A difficulty score will be assigned to each abnormality by the ground truthers using a 4-point Likert scale (1 being easy/obvious to 4 being hard/poorly visualised).

Outcome Measures

Primary Outcome Measures

Performance of AI algorithm: sensitivity [During 4 weeks of reading time]
Evaluation of the Gleamer Boneview algorithm will be performed comparing it to the reference standard in order to determine sensitivity.
Performance of AI algorithm: specificity [During 4 weeks of reading time]
Evaluation of the Gleamer Boneview will be performed comparing it to the reference standard in order to determine specificity.
Performance of AI algorithm: Area under the ROC Curve (AU ROC) [During 4 weeks of reading time]
Evaluation of the Gleamer Boneview algorithm will be performed comparing it to the reference standard. Continuous probability score from the algorithm will be utilised for the ROC analyses, while binary classification results with a predefined operating cut-off will be used for evaluation of sensitivity, specificity, positive predictive value, and negative predictive value.
Performance of readers with and without AI assistance: Sensitivity [During 4 weeks of reading time]
The study will include two sessions (with and without AI overlay), with all 18 readers reviewing all 500 XR cases each time separated by a washout period to mitigate recall bias. The cases will be randomised between the two reads and for every reader.
Performance of readers with and without AI assistance: Specificity [During 4 weeks of reading time]
The study will include two sessions (with and without AI overlay), with all 18 readers reviewing all 500 XR cases each time separated by a washout period to mitigate recall bias. The cases will be randomised between the two reads and for every reader.
Performance of readers with and without AI assistance: Area under the ROC Curve (AU ROC) [During 4 weeks of reading time]
The study will include two sessions (with and without AI overlay), with all 18 readers reviewing all 500 XR cases each time separated by a washout period to mitigate recall bias. The cases will be randomised between the two reads and for every reader.
Reader speed with vs without AI assistance. [During 4 weeks of reading time]
Mean time taken to review a XR, with vs without AI assistance.

Eligibility Criteria

Criteria

Ages Eligible for Study:

N/A and Older

Sexes Eligible for Study:

All

Accepts Healthy Volunteers:

Yes

Inclusion Criteria:

Emergency medicine doctors, trauma and orthopaedic surgeons, emergency nurse practitioners, physiotherapists, general radiologists and radiographers reviewing X-rays as part of their routine clinical practice.
Currently working in the National Health Service (NHS).