Message Testing Experiments: Psychological Determinants of Behavior Change
Study Details
Study Description
Brief Summary
We are conducting online experiments to systematically manipulate exposure to health-related messages in a series of controlled, randomized studies.
In one set of experiments, the goal is to test effects of various vaccine policies on vaccination decisions and strength of intention. Embedded within the survey experiment, we will compare different methods of measuring the outcome. Another experiment is designed to compare the effects of financial and social incentives (when deciding to enroll in health-related research, such as a RCT to increase exercise).
| Condition or Disease | Intervention/Treatment | Phase |
|---|---|---|
|
N/A |
Detailed Description
We are conducting randomized, controlled experiments that are embedded within a survey. The studies will enroll national samples recruited through MTurk and/or Prolific platforms.
Subjects are randomized to control and experimental conditions for the following experiments:
Comparing vaccine incentives, penalties and mandates to each other and a control condition:
This experiment first tests a relatively large and small financial incentives. We will separately test the effects of 10 experimental conditions, with a counter-balanced experimental manipulation using an FDA approval message.
Testing mandate message effects: This experiment tests the effects of mandatory vaccination versus freedom language.
Factors influencing the decision to participate in research: In this experiment, the study outcome will be interest in RCT enrollment for behavior change. We will test whether those who are most motivated to change behavior are also most likely to enroll in a (hypothetical) RCT and compare the effects of offering financial versus social incentives for behavior change.
In our experiments, socio-demographic variables, including age, gender, race, ethnicity, political affiliation, education, income, and financial stress will be measured for descriptive purposes.
Recommended data cleaning procedures: Attention checks can identify those who should be excluded from the main analyses. (Regardless of performance on the attention check, all participants will be compensated for their time.) It is also recommended that the final analyses exclude duplicate IDs, those with a high fraud score, and those who finished the fastest (fastest 5%).
Replication studies will include the same study design and procedures.
Analytic plans for vaccine message experiments (testing financial incentives, penalties, and mandates):
As noted above, this survey experiment involves randomizing individuals to a control condition or one of 10 experimental conditions that expose subjects to messages about different COVID-19 vaccine policies. The control group, with no vaccine policy presented will be compared to: cash incentives for $1000, $200, or $100; a $1,000 tax credit; lotteries for $100,000, $200,000, or $1 million; $1,000 tax on the unvaccinated; and mandates by employers or airlines, bars, and restaurants. The main outcome is whether they would want to get vaccinated soon given the hypothetical vaccine policy.
(Those assigned to the employer mandate condition will be excluded from the final analyses if they report being unlikely to have an employer.)
OLS specification will be our main result and the other measures are provided as robustness checks.
The OLS model can exclude all the demographic controls and run the binary dependent variable on the treatment variables. (Note that this approach is legitimate because the treatments are being randomized across respondents.) The treatments include the financial policies (incentives and penalties of different amounts and types) and mandates (of different types) being noted in a message.
• Type of model: We will perform pairwise t tests of percent of respondents answering "Yes" comparing those treated with an incentive to the control group. We will perform these pairwise tests on subsets by race, gender, income, education, and other socio-demographics.
Additionally, we will conduct these pairwise tests on by type of treatment. Comparing lottery to cash incentive, comparing positive incentive vs. penalty, comparing size of incentive, and comparing employer mandates against the control.
We will also conduct regression analyses on the pooled dataset where the left-hand side observations are individual responses where those answering "Yes" will be coded as 1 and those answering, "No" or "Unsure" will be coded zero. We will include a set of controls (race, gender, income, education, etc) as well as an indicator variable reflecting whether the respondent received a treatment. Regression models will include ordinary least squares, probit, nearest neighbor matching, and propensity score matching. We will also run these regressions where the treatment variable is split up into several indicator variables reflecting the type of treatment provided as well as an indicator for FDA approval.
We will estimate a model-alternatively using ordinary least squares and logistic regression-with a binary-outcome dependent variable (equal to one if the respondent wanted to be vaccinated, and otherwise equal to zero). For explanatory variables, we include dummy variables for each of the ten treatment arms.
• Criteria for statistical significance: We will use .05 as our threshold for statistical significance.
Embedded within, we will test if methodological differences in the response option for the primary outcome effect the percent reporting "yes". To do this, we will test 2 (Yes and No) vs 3 (Yes, No and Unsure) level response options and randomly order both sets.
Sample size calculation for the survey experiment comparing 10 different vaccine policies: We estimate that if the final sample sizes for each condition include at least 300 subjects, we can detect about 5% or larger difference. We plan to double the allocation for the control and $1000 conditions to allow for planned comparisons.
Methodological experiment: This experiment will examine whether the proportion responding "yes" to the same question (about whether they want to vaccinate soon) varies depending on the order of response options and whether they include a maybe/unsure option. We will run cross tabs and chi-square tests for the 2 vs 3 response levels and the order.
The experiments testing the effects of relatively large and small financial incentives on vaccination decisions will report point estimates and 95% CIs for the overall sample and demographic sub-groups. We will also report summary statistics for all the overall sample and sub-populations. We will test whether, compared to a control condition, either of the financial incentives increase, decrease, or have no effect on the percentage who want to vaccinate. In a fourth study arm, subjects will receive an educational message that will also be compared to the control condition.
Study Design
Arms and Interventions
| Arm | Intervention/Treatment |
|---|---|
| No Intervention: general health topic Those in this arm will receive a message about how people have different health preferences. |
|
| Experimental: specific health incentive information 1 Those in this arm will learn about a financial incentive (with a relatively large sum) or a social incentive |
Behavioral: specific health incentive information 1 with facts
a brief message that describes a debate over whether to pay people a large financial incentive to vaccinate
|
| Experimental: specific health incentive information 2 Those in this arm will learn about a financial incentive (with a relatively small sum) or a social incentive |
Behavioral: specific health incentive information 2 with facts
a brief message that describes a debate over whether to pay people a smaller financial incentive to vaccinate
|
| Experimental: facts only Those in this arm will receive a statement about vaccine safety. |
Behavioral: just facts
a brief message that describes vaccine safety
|
| Experimental: mandatory language Those in this arm will receive a statement about how vaccination may be mandatory. |
Behavioral: requirement to vaccinate message
a brief message that describes vaccination is likely to be required for work, plus safety and efficacy vac info
|
| Experimental: freedom language Those in this arm will receive a statement about how vaccination is currently not required and they have the freedom to choose what they want to do. |
Behavioral: freedom to decide
a brief message that notes you have the freedom to choose if you want to be vaccinated, and also include safety and efficacy vac info
|
Outcome Measures
Primary Outcome Measures
- measure of decision to vaccinate [through the duration of the experiment: less than 1 day]
subjects select a response option from a 2 ("yes" or "no") or 3 (yes, no, unsure) level response set
- measure of whether they are interested in RCT enrollment [through the duration of the experiment: less than 1 day]
subjects will select a response option reflecting whether they want RCT info
Secondary Outcome Measures
- likelihood of behavior [through the duration of the experiment: less than 1 day]
measures their perceived likelihood of performing the behavior with response options
Eligibility Criteria
Criteria
Inclusion Criteria:
-
Adults (18 years or older)
-
residing in the US
Exclusion Criteria:
-
children
-
those living outside the US
Contacts and Locations
Locations
| Site | City | State | Country | Postal Code | |
|---|---|---|---|---|---|
| 1 | Center for Mental Health. Perelman School of Medicine | Philadelphia | Pennsylvania | United States | 19104 |
Sponsors and Collaborators
- University of Pennsylvania
Investigators
- Principal Investigator: Jessica Fishman, PhD, University of Pennsylvania
Study Documents (Full-Text)
None provided.More Information
Publications
None provided.- 834772