E-CLAIR: Efficiency and Cost-effectiveness of Artificial Intelligence Based Diabetic Retinopathy Screening in Flanders
Study Details
Study Description
Brief Summary
To evaluate the efficiency and cost-effectiveness of an artificial intelligence based diabetic retinopathy screening program in Flanders
| Condition or Disease | Intervention/Treatment | Phase |
|---|---|---|
|
N/A |
Detailed Description
The increase of diabetes patients is a 21st century global health challenge with a predicted 642 million people suffering from the disease by 2040. Diabetes mellitus is characterized by high blood sugar levels over a prolonged period of time. These uncontrolled blood sugar levels can damage the inner lining of blood vessels which on the long term causes microvascular complications that affect small blood vessels. Retinopathy is the most prevalent microvascular complication of diabetes and is caused by small blood vessel damage, and neural damage at the back layer of the eye, the retina.
Diabetic retinopathy (DR) is the leading cause of blindness and visual disability in the working population. According to a study of the Eye Diseases Prevalence Research group, 40% of adult diabetes patients in the United States have some degree of DR and 8% have vision-threatening forms of DR. In addition, the DR Barometer study indicated that many patients with diabetes do not have a regular appointment with ophthalmology for an eye examination. Risk of vision loss can be significantly decreased with annual retinal screening and detection of cases that need to be referred for follow-up and treatment. The best example showing the value of eye screening is from the United Kingdom (UK). As a result of an implementation of a nationwide screening program, DR is no longer the leading cause of irreversible blindness in the UK.
In Flanders, and in Belgium as a whole, no such well-organized, nationwide DR screening program is in place and the approach is more fragmented. Flemish guidelines for diabetes care recommend an annual visit to the ophthalmologist for all the diabetic patients who receive insulin therapy in order to check if they have DR. About 30% of the diabetics will be diagnosed for DR and 70% are disease free or in a very early stage that doesn't need further treatment. However, manual detection of DR performed by an occupied, scarce ophthalmologist is labor-intensive and expensive, causing long waiting times for the patient and possibly resulting in a lack of care when needed.
Given the extent of the diabetes population in Flanders it is self-evident that there are difficulties to screen all patients in a timely manner by ophthalmologists. Indeed, a large amount of diabetes type 2 patients do not follow the annual referral by their general practitioner (GP) and are therefore screened at a too late stage, resulting in high, avoidable costs for the patient and society. Even more, the screening of the diabetic patients by an ophthalmologist put a resource burden on our healthcare system. Task differentiation, where trained graders or GP's instead of ophthalmologists grade for referable DR, can offer a solution for the too long waiting times and the high cost. Nevertheless, manual grading of DR still is labor-intensive and costly. Even more, despite the implementation of nationwide screening programs for DR and their accompanying grading protocols, there is still substantial room for improvement in the accuracy of manual DR grading.
Recently, deep learning (DL), a form of artificial intelligence (AI), has been introduced for automated analysis of images. In a landmark paper, Gulshan and co-workers published on a deep learning algorithm with high sensitivity and specificity for detecting referable DR. This study paved the way for further developments in the field of deep learning for automated DR detection, resulting in DL models that achieve specialist-level accuracy in diagnosing DR severity. IDx, for example, obtained the first-ever FDA authorization for an AI diagnostic system in any field of medicine for DR detection.
Implementation of software for automated analysis is seen as a cost-effective solution to support decision-making in an eye screening program. In the study by Tufail et al. three different AI grading tools were retrospectively compared for their performance and cost-effectiveness in the DR screening program in the UK. In a follow-up study by Heydon et al. the most promising AI grading tool was prospectively evaluated for use in the UK screening program, demonstrating high sensitivity with a specificity that could halve the workload of the manual graders.
Despite recent research there is still an existing gap for AI to be implemented effectively and efficiently in DR screening programs. For example, the high false-positive rate of AI based results hamper the clinical workflow. Also important to note is that DL models cannot replace the breadth and contextual knowledge of human specialists. It is the case that even the most accurate models will still need to be implemented into an existing clinical workflow before they can improve patient care at all. Besides, the real-world uptake of AI applications is slow and this is partly due to a lack of convincing evidence of the economical impact.
Taken all together, renewal within diabetes care in Flanders, and more in particular further development of a more efficient DR screening pathway, is necessary to ensure that the accessibility and quality of diabetic eye care can be guaranteed at manageable costs. Flanders can undoubtedly benefit from a more efficient and cost-effective AI-assisted DR screening workflow that is at least as accurate as a human specialist. Note that the translation of study results abroad to the Flanders situation is limited. After all, one cannot simply assume that cost-effectiveness ratios from foreign economic evaluations also apply in the Flanders context. Meaning that policymakers cannot base their decisions on the possible introduction of preventive screening interventions in Flanders directly on foreign studies. These findings demonstrate the clear need to set up a specific research project in Flanders to evaluate the efficiency and cost-effectiveness of a tailor-made DR screening program in Flanders.
Study Design
Arms and Interventions
| Arm | Intervention/Treatment |
|---|---|
| Active Comparator: current workflow in Flanders patient visits ophthalmologist |
Diagnostic Test: gold standard
examination by ophthalmologist
|
| Active Comparator: AI-only workflow patient is imaged, images are interpreted by DR AI tool, only referrable cases identified by DR AI tool will visit ophthalmologist |
Device: deep learning
a form of artificial intelligence (AI), has been introduced for automated analysis of images
|
| Active Comparator: AI-human workflow patient is imaged, images are interpreted by DR AI tool, referrable cases identified by DR AI tool will be remotely graded by a human, only the high risk patients will visit ophthalmologist |
Diagnostic Test: remote grading of fundus images
referrable cases identified by DR AI tool will be remotely graded by a human
|
Outcome Measures
Primary Outcome Measures
- sensitivity [6 months]
To evaluate the efficiency of the use of AI in screening for DRP: sensitivity
- specificity [6 months]
To evaluate the efficiency of the use of AI in screening for DRP: specificity
- AUC [6 months]
To evaluate the efficiency of the use of AI in screening for DRP: AUC
Secondary Outcome Measures
- precision [6 months]
performance of three DR screening workflows: precision
- decision tree model [6 months]
cost-effectiveness of three DR screening workflows: decision tree model
- recall [6 months]
performance of three DR screening workflows : recall
- F1 score [6 months]
performance of three DR screening workflows: F1 score
- false positives and false negatives [6 months]
performance of three DR screening workflows: false positives and false negatives
Eligibility Criteria
Criteria
Inclusion Criteria:
-
Diagnosis of diabetes mellitus
-
Age > 18 years old
-
Patient is capable of giving informed consent
-
Fluent in written and oral Dutch, or interpreter present
Exclusion Criteria:
-
- History of treatment for diabetic retinopathy or diabetic macular edema (laser or intravitreal injections)
-
Participant is contraindicated for imaging by fundus imaging systems used in the study
Contacts and Locations
Locations
| Site | City | State | Country | Postal Code | |
|---|---|---|---|---|---|
| 1 | UZA | Antwerpen | Belgium | ||
| 2 | ZNA | Antwerp | Belgium | ||
| 3 | AZ sint Jan | Brugge | Belgium | ||
| 4 | AZ Turnhout | Turnhout | Belgium |
Sponsors and Collaborators
- Universitaire Ziekenhuizen Leuven
Investigators
- Principal Investigator: Julie Jacob, MD PhD, Universitaire Ziekenhuizen Leuven
Study Documents (Full-Text)
None provided.More Information
Publications
- Abrà moff MD, Folk JC, Han DP, Walker JD, Williams DF, Russell SR, Massin P, Cochener B, Gain P, Tang L, Lamard M, Moga DC, Quellec G, Niemeijer M. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol. 2013 Mar;131(3):351-7. doi: 10.1001/jamaophthalmol.2013.1743.
- Farley TF, Mandava N, Prall FR, Carsky C. Accuracy of primary care clinicians in screening for diabetic retinopathy using single-image retinal photography. Ann Fam Med. 2008 Sep-Oct;6(5):428-34. doi: 10.1370/afm.857.
- Harding SP, Broadbent DM, Neoh C, White MC, Vora J. Sensitivity and specificity of photography and direct ophthalmoscopy in screening for sight threatening eye disease: the Liverpool Diabetic Eye Study. BMJ. 1995 Oct 28;311(7013):1131-5.
- Heydon P, Egan C, Bolter L, Chambers R, Anderson J, Aldington S, Stratton IM, Scanlon PH, Webster L, Mann S, du Chemin A, Owen CG, Tufail A, Rudnicka AR. Prospective evaluation of an artificial intelligence-enabled algorithm for automated diabetic retinopathy screening of 30 000 patients. Br J Ophthalmol. 2021 May;105(5):723-728. doi: 10.1136/bjophthalmol-2020-316594. Epub 2020 Jun 30.
- Kapetanakis VV, Rudnicka AR, Liew G, Owen CG, Lee A, Louw V, Bolter L, Anderson J, Egan C, Salas-Vega S, Rudisill C, Taylor P, Tufail A. A study of whether automated Diabetic Retinopathy Image Assessment could replace manual grading steps in the English National Screening Programme. J Med Screen. 2015 Sep;22(3):112-8. doi: 10.1177/0969141315571953. Epub 2015 Mar 5.
- Kempen JH, O'Colmain BJ, Leske MC, Haffner SM, Klein R, Moss SE, Taylor HR, Hamman RF; Eye Diseases Prevalence Research Group. The prevalence of diabetic retinopathy among adults in the United States. Arch Ophthalmol. 2004 Apr;122(4):552-63.
- Lee R, Wong TY, Sabanayagam C. Epidemiology of diabetic retinopathy, diabetic macular edema and related vision loss. Eye Vis (Lond). 2015 Sep 30;2:17. doi: 10.1186/s40662-015-0026-2. eCollection 2015. Review.
- Liew G, Michaelides M, Bunce C. A comparison of the causes of blindness certifications in England and Wales in working age adults (16-64 years), 1999-2000 with 2009-2010. BMJ Open. 2014 Feb 12;4(2):e004015. doi: 10.1136/bmjopen-2013-004015.
- Lin DY, Blumenkranz MS, Brothers RJ, Grosvenor DM. The sensitivity and specificity of single-field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: a comparison with ophthalmoscopy and standardized mydriatic color photography. Am J Ophthalmol. 2002 Aug;134(2):204-13.
- Ogurtsova K, da Rocha Fernandes JD, Huang Y, Linnenkamp U, Guariguata L, Cho NH, Cavan D, Shaw JE, Makaroff LE. IDF Diabetes Atlas: Global estimates for the prevalence of diabetes for 2015 and 2040. Diabetes Res Clin Pract. 2017 Jun;128:40-50. doi: 10.1016/j.diabres.2017.03.024. Epub 2017 Mar 31.
- Sayres R, Taly A, Rahimy E, Blumer K, Coz D, Hammel N, Krause J, Narayanaswamy A, Rastegar Z, Wu D, Xu S, Barb S, Joseph A, Shumski M, Smith J, Sood AB, Corrado GS, Peng L, Webster DR. Using a Deep Learning Algorithm and Integrated Gradients Explanation to Assist Grading for Diabetic Retinopathy. Ophthalmology. 2019 Apr;126(4):552-564. doi: 10.1016/j.ophtha.2018.11.016. Epub 2018 Dec 13.
- Sussman EJ, Tsiaras WG, Soper KA. Diagnosis of diabetic eye disease. JAMA. 1982 Jun 18;247(23):3231-4.
- Tufail A, Kapetanakis VV, Salas-Vega S, Egan C, Rudisill C, Owen CG, Lee A, Louw V, Anderson J, Liew G, Bolter L, Bailey C, Sadda S, Taylor P, Rudnicka AR. An observational study to assess if automated diabetic retinopathy image assessment software can replace one or more steps of manual imaging grading and to determine their cost-effectiveness. Health Technol Assess. 2016 Dec;20(92):1-72.
- Tufail A, Rudisill C, Egan C, Kapetanakis VV, Salas-Vega S, Owen CG, Lee A, Louw V, Anderson J, Liew G, Bolter L, Srinivas S, Nittala M, Sadda S, Taylor P, Rudnicka AR. Automated Diabetic Retinopathy Image Assessment Software: Diagnostic Accuracy and Cost-Effectiveness Compared with Human Graders. Ophthalmology. 2017 Mar;124(3):343-351. doi: 10.1016/j.ophtha.2016.11.014. Epub 2016 Dec 23.
- Wolff J, Pauling J, Keck A, Baumbach J. The Economic Impact of Artificial Intelligence in Health Care: Systematic Review. J Med Internet Res. 2020 Feb 20;22(2):e16866. doi: 10.2196/16866.
- Xie Y, Nguyen QD, Hamzah H, Lim G, Bellemo V, Gunasekeran DV, Yip MYT, Qi Lee X, Hsu W, Li Lee M, Tan CS, Tym Wong H, Lamoureux EL, Tan GSW, Wong TY, Finkelstein EA, Ting DSW. Artificial intelligence for teleophthalmology-based diabetic retinopathy screening in a national programme: an economic analysis modelling study. Lancet Digit Health. 2020 May;2(5):e240-e249. doi: 10.1016/S2589-7500(20)30060-1. Epub 2020 Apr 23.
- S64955