PROJECT 2 EXAMPLE: Feedback X Prevalence Using Dermatology Stimuli

Sponsor

Brigham and Women's Hospital (Other)

Overall Status

Completed

CT.gov ID

NCT05244122

Collaborator

(none)

803

Enrollment

Location

Arm

Actual Duration (Days)

4888.3

Patients Per Site Per Month

Study Details

Study Description

Brief Summary

Imagine that a dermatologist spends the morning seeing patients who have been referred for suspicion of skin cancer. Many of them do, in fact, have skin lesions that require treatment. For this set of patients, disease 'prevalence' would be high. Suppose that the next task is to spend the afternoon giving annual screening exams to members of the general population. Here disease prevalence will be low. Would the morning's work influence decisions about patients in the afternoon? It is known from other contexts that recent history can influence current decisions and that target prevalence has an impact on decisions. In this study, decisions were decisions about skin lesions from individuals with varying degrees of expertise, using an online, medical imaging labelling app (DiagnosUs). This allowed examination of the effects of feedback history and prevalence in a single study. Blocks of trials could be of low or high prevalence, with or without feedback. Over 300,000 individual judgements were collecte. It was found that participants became more conservative after a low prevalence block with feedback and more liberal after at high prevalence block with feedback. Blocks without feedback did not significantly alter behavior on the next block. This study happens to use dermatology stimuli but the basic effects may apply in any situation where prevalence and feedback change over time. Interestingly, the effects of one block on the next did not appear to fade with time, even over intervals of more than a day. (taken from Wolfe,

1. (2022). How one block of trials influences the next: Persistent effects of disease prevalence and feedback on decisions about images of skin lesions in a large online study. .

Cognitive Research: Principles and Implications (CRPI), 7, 10. doi:

https://doi.org/10.1186/s41235-022-00362-0

Condition or Disease	Intervention/Treatment	Phase
Decision Making	Behavioral: Feedback Behavioral: Prevalence	N/A

Detailed Description

This description is based on a preregistration on the Open Science Framework site. Note that this is a "BESH" study. This type of research is not designed as a traditional clinical trial, but it is being reported here because of changes in NIH clinical trial reporting rules.

Levari et al (2018) found that people responded to a decrease in the prevalence of a stimulus by expanding their concept of it. Specifically, they asked observers to judge on each trial whether a dot, drawn from a blue-purple continuum, was blue or not. The results showed that observers were more likely to call ambiguous stimuli "blue" when blue items were less prevalent. In signal detection theory (SDT) terms, this is a liberal shift of response criterion. This is "prevalence induced concept change" (PICC). However, previous results obtained the opposite results in a long series of experiments on prevalence effects. The standard finding is that Os miss more targets at low prevalence. When blue is rare, they are less likely to call something blue. In SDT terms, this is a conservative criterion shift. This is the classic Low Prevalence Effect (LPE). In a round of earlier experiments, Lyu et al (2021) found that feedback is a critical variable. With trial-by-trial feedback, we get an LPE. With no feedback, the data usually show PICC results.

Do LPE and PICC effects show up when experts view stimuli in their expert domain? There is evidence for the LPE from search tasks (e.g. Evans, K. K., Birdwell, R. L., & Wolfe, J. M. (2013). If You Don't Find It Often, You Often Don't Find It: Why Some Cancers Are Missed in

Breast Cancer Screening. . PLoS ONE 8(5): e64366. , 8(5), e64366. doi:

doi:10.1371/journal.pone.0064366). However, PICC evidence has not been collected and there is no data from single item decision tasks like the "Is this dot blue?" task. This is important because criterion shifts of the sort described above can have obvious health care implications.

This study will repeat the basic "Is this dot blue" experiment using dermatology stimuli (Is this melanoma or just a nevus (a mole)?)

Hypotheses:

(H1) without feedback, Os are more likely to label a spot as cancer when cancer prevalence is low (prevalence-induced-concept-change).

(H2) that with feedback, Os are less likely to label a spot as cancer when cancer prevalence is low (classic low prevalence effect)

Dependent variable

The main dependent variable is the proportion of cancer responses as a function of the cancer prevalence in the image set, but we will also record reaction times.

Conditions

How many and which conditions will participants be assigned to?

Four conditions will be run, between observers.

50% cancer images with feedback
50% cancer images without feedback
20% cancer images with feedback
20% cancer images without feedback

Observers will make a simple 2AFC cancer/no cancer decision. Observers will be awarded points based on the correctness of the answer (more correct, more points)

There will be 200 trials in each block. That will produce 40 target present trials in the low prevalence conditions which should produce a hit rate that is not too coarse.

Stimuli will be images of moles from the ISIC archive. Each image comes with a known answer of either melanoma (cancer) or nevus (negative).

Analyses

The data will peoduce a continuum from not-cancer to cancer based on the observers responses in the 50% with feedback condition. This will give yield a psychometric function rising (it may be assumed) from near 0% cancer responses to near 100%.

Using that ordering, psychometric functions will be generated for the other three conditions.

To examine the effect of prevalence and the presence and absence of feedback on observers' response behavior, \run a logistic regression with prevalence and feedback as factors in a generalized mixed model will be run using jamovi software.

The data will also be used to compute the signal detection measures of sensitivity (d') and criterion (c) based on the actual truth about the images. That is, "cancer" responses will be coded as True positives if the images show cancer and as "false positives" if they do not. T-tests will be performed to examine whether d' and/or c (criterion) change significantly as a function of prevalence and feedback.

Outliers and Exclusions

N/A

Sample Size

Separate blocks of trials will be run and conditions will be compared with unpaired t-tests.

G* Power says suggests 36 observers PER GROUP or a total of 144 observers for alpha = 0.05, power = 0.80. The plan will be to attempt to run 45 Os per group, anticipating about 20% loss of Os due to the vagaries of online testing.

Study Design

Study Type:

Interventional

Actual Enrollment :

803 participants

Allocation:

N/A

Intervention Model:

Single Group Assignment

Intervention Model Description:

We tested ~800 observers on line in a study where they could participate in multiple conditions. We were looking for the effect of condition 1 on condition N+1We tested ~800 observers on line in a study where they could participate in multiple conditions. We were looking for the effect of condition 1 on condition N+1

Masking:

None (Open Label)

Masking Description:

Observers were naive about the hypothesis but could have figured out if a specific condition did or did not have feedback, for example. Once the data were collected, investigators could determine what conditions were tested on which observers

Primary Purpose:

Basic Science

Official Title:

Title of Protocol: Prevalence Effects in Visual Search: Theoretical and Practical Implications NOTE: This IRB Protocol Covers Many Small Experiments, Not Just This Study

Actual Study Start Date :

Jun 22, 2021

Actual Primary Completion Date :

Jun 27, 2021

Actual Study Completion Date :

Jun 27, 2021

Arms and Interventions

Arm	Intervention/Treatment
Experimental: Experimental Observers could choose to participate in each of four experiments on each of six days	Behavioral: Feedback presence or absence of trial by trial feedback Behavioral: Prevalence In some blocks, skin cancer "target" images were present on 50% of trials (high prevalence). In other blocks, disease prevalence was 20%.

Arm

Intervention/Treatment

Experimental: Experimental

Observers could choose to participate in each of four experiments on each of six days

Behavioral: Feedback

presence or absence of trial by trial feedback

Behavioral: Prevalence

In some blocks, skin cancer "target" images were present on 50% of trials (high prevalence). In other blocks, disease prevalence was 20%.

Outcome Measures

Primary Outcome Measures

D' [During experimental session. D' is based on the accuracy of responses during the experiment]
D' (d-prime) is the signal detection theory measure of the level of performance on a task.
criterion [During experimental session. Criterion is also based on the accuracy of responses during the experiment]
The signal detection theory measure of the bias ("liberal" or "conservative") of observers' decisions

Eligibility Criteria

Criteria

Ages Eligible for Study:

18 Years and Older

Sexes Eligible for Study:

All

Accepts Healthy Volunteers:

Yes

Inclusion Criteria:

All welcome to enroll on line

Exclusion Criteria:

Those participants running only a single block were removed from most analyses because our primary interest is the influence of one block upon the next