A Multi-center Study to Evaluate Performance of an Automated Device for the Detection of Diabetic Retinopathy

• Sponsor: IDx LLC (Industry)
• Overall Status: Completed
• CT.gov ID: NCT02963441
• Collaborator: The Emmes Company, LLC (Industry)
• Enrollment: 900
• Actual Duration (Months): 11

Study Details

Study Description

Brief Summary

Diabetes affects 30.3 million people or 9.3% of the population of the United States. Results of a study by the Eye Diseases Prevalence Research Group reveal that 40% of diabetes patients have some degree of diabetic retinopathy (DR) and that as many as 8% have severe, vision-threatening forms of DR. Early laser photocoagulation in high-risk proliferative diabetic retinopathy (PDR) has been shown to decrease the relative risk of vision loss by as much as 52%. Injections of anti-VEGF agents preserve and improve vision in people with PDR and/or diabetic macular edema. Despite effective treatment however, tens of thousands of people with diabetes are going blind each year largely because they don't undergo annual screening for retinopathy. Currently, less than 50%-60% of people with diabetes have a yearly eye exam and there may not be enough eye specialists to see the balance.

The IDx-DR System is intended for use by health care providers to automatically detect more than mild diabetic retinopathy (mtmDR) in adults (22 years of age or older) diagnosed with diabetes who have not been previously diagnosed with diabetic retinopathy. IDx-DR is indicated for use with the Topcon NW400.

Condition or Disease	Intervention/Treatment	Phase
Diabetic Retinopathy

Detailed Description

From January 2017 to July 2017, 900 participants were prospectively enrolled in this observational study at 10 primary care practice sites throughout the United States. The study was approved by the institutional review board for each site, and all participants provided written informed consent. The study, which was funded by IDx LLC, was designed by the authors with input from the U.S. Food and Drug Administration (FDA) on the endpoints, statistical testing, and study design. Emmes Corp, a contract research organization (CRO), provided overall project management, including data management and independent monitoring and auditing services for all sites. CCR, Inc., an Algorithm Integrity Provider (AIP), was contracted to lock the AI system, hold any intermediate and final results and images in escrow, and interdict access to these by the Sponsor, from prior to the start of the study until final data lock. Because the Sponsor was thus interdicted from access to the AI system, the AIP performed all necessary maintenance and servicing activities during the study as well as throughout closeout.

The target population was asymptomatic persons, ages of 22 and older, who had been diagnosed with diabetes and had not been previously diagnosed with DR. A diagnosis of diabetes was defined as meeting the criteria established by either the World Health Organization (WHO) or the American Diabetes Association (ADA); Hemoglobin A1c (HbA1c) ≥ 6.5% based on repeated assessments; Fasting Plasma Glucose (FPG) ≥ 126 mg/dL (7.0 mmol/L) based on repeated assessments; Oral Glucose Tolerance Test (OGTT) with two-hour plasma glucose (2-hr PG) ≥ 200 mg/dL (11.1 mmol/L) using the equivalent of an oral 75 g anhydrous glucose dose dissolved in water; or symptoms of hyperglycemia or hyperglycemic crisis with a random plasma glucose (RPG) ≥ 200 mg/dL (11.1 mmol/L).

To help enroll a sufficient number of mtmDR participants for the evaluation of sensitivity, a stepwise enrichment strategy, as indicated in the prespecified protocol, was utilized mid-study to recruit sufficient numbers of mtmDR participants. The enrichment strategy sought higher risk participants with elevated HbA1c (>9.0%) levels or elevated Fasting Plasma Glucose; this enrichment was independently activated by the statistician while always remaining masked to the AI system outputs and the ETDRS disease levels. To account for any unintentional spectrum bias in the no/mild population, the study pre-defined a specificity outcome parameter to correct for any potential spectrum bias resulting from this enrichment strategy as co-primary.

All primary care sites in the study identified one or more inhouse operator trainees to perform the AI system protocol. After installation of the equipment by the Sponsor at the site, but before any participant was recruited, AI system operator trainees had to attest that they had not previously performed ocular imaging. Also, before start of study recruitment at each site, AI system operator trainees underwent a one-time standardized 4 h training program. They were trained how to acquire images, how to improve image quality if the AI system gave an insufficient quality output, and how to put images for analysis into the AI system. No additional training was provided to any of the AI system operators for the duration of the study. Independently, FPRC certified expert photographers were identified in geographic locations close to each site by the CRO, and documented 4W-D FPRC certification was required before any participant was imaged. The CRO independently completed site initiation visits at each site to ensure each site met all the good clinical practice requirements prior to start of enrollment.

Study protocol. All participants gave written informed consent to participate in both the AI system protocol, as well as the FPRC imaging protocol, using two different cameras:

The AI system protocol consisted of the following steps:

1. Operator takes images with a nonmydriatic retinal camera (NW400, Topcon Medical Systems, Oakland, NJ) according to a standardized imaging protocol with one disc and one fovea centered 45° image per eye;

2. Operator submits images to the AI system for automated image quality and protocol adherence evaluation;

3. If the AI system outputs insufficient quality, steps 1-2 are repeated until sufficient quality is output or 3 attempts were made. If the AI system still indicates that images are of insufficient quality, the participant's pupils are dilated with tropicamide 1.0% eyedrops, (provided by the Sponsor at each site), until the pupil diameter is at least 5mm in each eye or 30 minutes have passed, and steps 1-2 are repeated until sufficient quality is output or 3 attempts were made. If the AI system still outputs that images are of insufficient quality, the AI system output of insufficient quality is automatically provided to the CRO via secure data transfer;

4. Whenever the AI system indicates sufficient quality, the AI system disease level output (either mtmDR detected or mtmDR not detected) is automatically provided to the CRO via secure data transfer; the final AI system output provided to the CRO after this protocol was mtmDR detected, mtmDR not detected or insufficient quality

The FPRC imaging protocol was then conducted, and consisted of the following steps, all performed by an FPRC certified photographer:

1. If participant is not already dilated, dilating eye drops of tropicamide 1.0% are administered;

2. Digital widefield stereoscopic fundus photography is performed, using a camera capable of widefield photography, (Maestro, Topcon Medical Systems, Oakland, NJ) according to the FPRC 4W-D stereo protocol, by an FPRC certified photographer;

3. Anterior segment photography for media opacity assessment is performed according to the Age Related Eye Disease Study, by an FPRC certified photographer;

4. OCT of the macula is performed using a standard OCT system capable of producing a cube scan containing at least 121 B scans, (Maestro, Topcon Medical Systems, Oakland, NJ) according to the FPRC OCT protocol, by an FPRC certified photographer.

The FPRC certified photographers were masked to the AI system outputs at all times.

Reference standards. The FPRC grading protocol consisted of determination of ETDRS Severity Scale (SS) levels for fundus photographs and standardized OCT grading, as follows: the 4W-D images were read by three experienced and validated readers at the FPRC according to the well established ETDRS SS, using a majority voting paradigm. The macular OCT images were evaluated for the presence of center-involved DME by experienced readers at the FPRC according to the DRCR grading paradigm. For each participant, the ETDRS levels were mapped to mtmDR+(ETDRS level 35 or higher and /or DME present), or mtmDR- (ETDRS level 10-20 and DME absent), taking the worst of two eyes to correspond to the outputs of the AI system at the participant level.16 To measure sensitivity for the cases requiring immediate followup, called vision threatening DR, we defined vtDR + as ETDRS level 53 or higher, and/ or DME present, See Supplemental Table 2 for the mapping from ETDRS and DME levels to dichotomous mtmDR- and mtmDR +and vtDR+. Because DME can be identified both on the basis of retinal thickening on stereoscopic fundus photographs, as well as on the basis of retinal thickening on OCT, we separately analyzed both. Stereoscopic fundus based Clinically Significant DME (CSDME) was identified if there was either retinal thickening or adjacent hard exudates < 600 μm from the foveal center, or a zone of retinal thickening > 1 disc area, part of which is less than 1 disc diameter from the foveal center, according to the FPRC, in any eye. OCT based center-involved DME was identified if a participant had central subfield (a 1.0mm circle centered on the fovea) thickness that was >300 μm according to the FPRC, in any eye.20

Accordingly, we further specify the definition of mtmDR where relevant:

fundus mtmDR+ is defined as

• ETDRS level ≥ 35 (determined from fundus photographs)and/or

• CSDME (determined from fundus photographs)

and multimodal mtmDR+is defined as:

• ETDRS level ≥ 35 (determined from fundus photographs), and / or

• CSDME (determined from fundus photographs) and / or

• center-involved DME (determined from OCT).

and similarly for vtDR + . FPRC readers were masked to the AI system outputs at all times, masked to the fundus photograph reading when evaluating the OCT images, and masked to OCT readings when evaluating fundus photographs.

Study Design

• Study Type: Observational
• Actual Enrollment: 900 participants
• Observational Model: Case-Only
• Time Perspective: Cross-Sectional
• Official Title: A Multi-center Study to Evaluate Performance of an Automated Device for the Detection of Diabetic Retinopathy
• Actual Study Start Date: Jan 1, 2017
• Actual Primary Completion Date: Nov 1, 2017
• Actual Study Completion Date: Dec 1, 2017

Outcome Measures

Primary Outcome Measures

1. Sensitivity Corrected for Enrichment [Day 1]

   Sensitivity corrected for enrichment using logistic regression

2. Specificity Corrected for Enrichment [Day 1]

   Specificity corrected for enrichment using logistic regression

3. Observed Sensitivity [Day 1]

   The sensitivity observed without adjustment by logistic regression

4. Observed Specificity [Day 1]

   The specificity observed without adjustment by logistic regression

5. Sufficient vs. Insufficient Image Quality Output From IDx-DR [Day 1]

   Image-ability, defined as the percentage of participants with a completed reading center grading and a disease level output from the investigational device (IDx-DR)

Eligibility Criteria

Criteria

• Ages Eligible for Study: 22 Years and Older
• Sexes Eligible for Study: All
• Accepts Healthy Volunteers: No

Inclusion Criteria:

1. Documented diagnosis of diabetes mellitus, e.g.:

2. Having met the criteria established by either the World Health Organization (WHO) or the American Diabetes Association (ADA)

3. Hemoglobin A1c (HbA1c) ≥ 6.5%

4. Fasting Plasma Glucose (FPG) ≥ 126 mg/dL (7.0 mmol/L)

5. Oral Glucose Tolerance Test (OGTT) with two-hour plasma glucose (2-hr PG) ≥ 200 mg/dL (11.1 mmol/L), using the equivalent of an oral 75 g anhydrous glucose dose dissolved in water

6. Symptoms of hyperglycemia or hyperglycemic crisis with a random plasma glucose (RPG) ≥ 200 mg/dL (11.1 mmol/L)

7. Age 22 or older

8. Understand the study and volunteer to sign the informed consent

Exclusion Criteria:

1. Persistent vision loss, blurred vision, or floaters.

2. Diagnosed with macular edema, severe non-proliferative retinopathy, proliferative retinopathy, radiation retinopathy, or retinal vein occlusion.

3. History of laser treatment of the retina or injections into either eye, or any history of retinal surgery.

4. Currently participating in another investigational eye study and actively receiving investigational product for DR or DME.

5. Participant has a condition that, in the opinion of the investigator, would preclude participation in the study (e.g., unstable medical status including blood pressure or glycemic control, microphthalmia or previous enucleation).

6. Participant is contraindicated for imaging by fundus imaging systems used in the study:

• Participant is hypersensitive to light

• Participant recently underwent photodynamic therapy (PDT)

• Participant is taking medication that causes photosensitivity

Contacts and Locations

Locations

No locations specified.

Sponsors and Collaborators

• IDx LLC
• The Emmes Company, LLC

Investigators

None specified.

Study Documents (Full-Text)

• Study Protocol - Aug 17, 2017
• Statistical Analysis Plan - Aug 20, 2017

More Information

Publications

None provided.

• Responsible Party: IDx LLC
• ClinicalTrials.gov Identifier: NCT02963441
• Other Study ID Numbers: IDx-DR1
• First Posted: Nov 15, 2016
• Last Update Posted: May 28, 2021
• Last Verified: May 1, 2021

• Individual Participant Data (IPD) Sharing Statement: No
• Plan to Share IPD: No
• Keywords provided by IDx LLC: Diabetic Retinopathy; Macular Edema
• Additional relevant MeSH terms: Retinal Diseases; Diabetic Retinopathy

Study Results

Participant Flow

Recruitment Details	From January 2017 to July 2017, 900 participants were prospectively enrolled in this observational study at 10 primary care practice sites throughout the United States.

Arm/Group Title	Intent to Screen Population
Arm/Group Description	Prospective study participants
Period Title: Overall Study
STARTED	900
COMPLETED	819
NOT COMPLETED	81

Baseline Characteristics

Arm/Group Title	Effective Analysis Population
Arm/Group Description	All subjects with a reading center reference standard and AI output (population that is available for statistical analysis).
Overall Participants	819
Age (Count of Participants)
<=18 years	0 0%
Between 18 and 65 years	566 69.1%
>=65 years	253 30.9%
Sex/Gender, Customized (Count of Participants)
Male	389 47.5%
Female	430 52.5%
Ethnicity (NIH/OMB) (Count of Participants)
Hispanic or Latino	134 16.4%
Not Hispanic or Latino	680 83%
Unknown or Not Reported	5 0.6%
Race (NIH/OMB) (Count of Participants)
American Indian or Alaska Native	3 0.4%
Asian	12 1.5%
Native Hawaiian or Other Pacific Islander	0 0%
Black or African American	231 28.2%
White	523 63.9%
More than one race	10 1.2%
Unknown or Not Reported	40 4.9%
Region of Enrollment (participants) [Number]
United States	819 100%

Outcome Measures

1. Primary Outcome

Title	Sensitivity Corrected for Enrichment
Description	Sensitivity corrected for enrichment using logistic regression
Time Frame	Day 1

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title	Analyzable Study Population
Arm/Group Description	Participants that had reading center grading and AI disease outputs that were available for statistical analysis.
Measure Participants	819
Number (95% Confidence Interval) [percentage of true positives]	87.2

2. Primary Outcome

Title	Specificity Corrected for Enrichment
Description	Specificity corrected for enrichment using logistic regression
Time Frame	Day 1

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title	Analyzable Study Population
Arm/Group Description	Participants that had reading center grading and AI disease outputs that were available for statistical analysis.
Measure Participants	819
Number (95% Confidence Interval) [percentage of true negatives]	90.7

3. Primary Outcome

Title	Observed Sensitivity
Description	The sensitivity observed without adjustment by logistic regression
Time Frame	Day 1

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title	Analyzable Study Population
Arm/Group Description	Participants that had reading center grading and AI disease outputs that were available for statistical analysis.
Measure Participants	819
Number (95% Confidence Interval) [percentage of true positives]	87.4

4. Primary Outcome

Title	Observed Specificity
Description	The specificity observed without adjustment by logistic regression
Time Frame	Day 1

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title	Analyzable Study Population
Arm/Group Description	Participants that had reading center grading and AI disease outputs that were available for statistical analysis.
Measure Participants	819
Number (95% Confidence Interval) [percentage of true negatives]	89.5

5. Primary Outcome

Title	Sufficient vs. Insufficient Image Quality Output From IDx-DR
Description	Image-ability, defined as the percentage of participants with a completed reading center grading and a disease level output from the investigational device (IDx-DR)
Time Frame	Day 1

Outcome Measure Data

Analysis Population Description

Image-ability, defined as the percentage of participants with a completed FPRC grading and a disease level IDx-DR output

Arm/Group Title	Analyzable Study Population
Arm/Group Description	Participants with the reading center results needed for study analysis.
Measure Participants	852
Number (95% Confidence Interval) [percentage of participants]	96.1 11.7%

Adverse Events

Time Frame	Adverse events were collected from Day 1 to 30 days after last study contact.
Arm/Group Title	Enrolled Population
Arm/Group Description	All study subjects enrolled into the clinical study.
All Cause Mortality
	Enrolled Population
	Affected / at Risk (%)	# Events
Total	0/900 (0%)
Serious Adverse Events
	Enrolled Population
	Affected / at Risk (%)	# Events
Total	0/900 (0%)
Other (Not Including Serious) Adverse Events
	Enrolled Population
	Affected / at Risk (%)	# Events
Total	5/900 (0.6%)
Eye disorders
Eye edema	1/900 (0.1%)	1
Vision blurred	1/900 (0.1%)	1
Eye irritation	1/900 (0.1%)	1
Metabolism and nutrition disorders
Hypoglycemic Episode	1/900 (0.1%)	1
Nervous system disorders
Headache	1/900 (0.1%)	1
Psychiatric disorders
Panic attack	1/900 (0.1%)	1
Vascular disorders
Hypotension	1/900 (0.1%)	1

Limitations/Caveats

[Not Specified]

More Information

Certain Agreements

Principal Investigators are NOT employed by the organization sponsoring the study.

If a multi-center publication is not forthcoming within twenty-four (24) months following completion of the Study, Institution will be free to publish the methods, data and results of the Study