Emails Promoting COVID-19 Vaccination Among Healthcare Workers

• Sponsor: Geisinger Clinic (Other)
• Overall Status: Completed
• CT.gov ID: NCT04728594
• Collaborator: (none)
• Enrollment: 9,566
• Location: 1
• Arms: 3
• Actual Duration (Days): 5
• Patients Per Site Per Month: 58233

Study Details

Study Description

Brief Summary

In this evaluation, two versions of emails will be sent to Geisinger employees who have not been vaccinated against COVID-19. A delayed-contact control group will be emailed after a delay of at least two days. The researchers hypothesize that either of the emails - which use content informed by behavioral nudge theory - will lead to more COVID-19 vaccinations than the delayed-contact control group.

Condition or Disease	Intervention/Treatment	Phase
Vaccination Refusal; Communication	Behavioral: Social Proof; Behavioral: Reframing; Behavioral: Scarcity Message	N/A

Detailed Description

The researchers aim to evaluate the effectiveness of email reminders in promoting COVID-19 vaccinations among Geisinger employees. The project will compare two emails, which will vary in the subject line and email content. Both emails will test different approaches that draw from principles in behavioral science: social proof, reframing, and scarcity. Employees receiving these emails will be compared with employees in a delayed-contact group, who will not receive emails. Employees in the control group will later receive one of the emails.The responses to the emails will help determine the communications later sent to the control group.

All emails will ask recipients if they want to receive a COVID-19 vaccination. If they click on a hyperlink indicating yes, they will be forwarded to an online registration and scheduling homepage. If they click on a hyperlink indicating no, they will be forwarded to an online questionnaire surveying people about their main reason for declining the vaccine at this time. The questionnaire will automatically present information intended to assuage the specific concern that people endorse, followed by a hyperlink to the scheduling site.

The data will be analyzed with logistic regression models with the control group as the reference group, to compare the two email conditions versus the control group. This set of analyses will only be conducted for scheduling a vaccination appointment as opposed to email engagement outcomes (e.g., number of emails opened), which will not be applicable for the control group. A second set of logistic regression models predicting scheduling a vaccination appointment and email engagement will also be run, comparing the two email conditions against each other. The data will be examined two days later (to help decide which email to send to the delayed-control group) and four days later (when the policy for employee vaccination will change, making the content of the email irrelevant).

Update: The researchers looked at 3 days of data - as the emails to the delayed contact group were only sent at that time - and 4 days of data when the policy for employee vaccinations changed.

Study Design

• Study Type: Interventional
• Actual Enrollment: 9566 participants
• Allocation: Randomized
• Intervention Model: Parallel Assignment
• Intervention Model Description: 1 x 3 design1 x 3 design
• Masking: Double (Participant, Care Provider)
• Masking Description: Participants will not know what group they are in. The relevant care provider (either their primary care provider or provider administering the vaccine) will not know about this study or assignment into groups.
• Primary Purpose: Prevention
• Official Title: Evaluation of an Email Intervention Promoting COVID-19 Vaccination Among Healthcare Workers
• Actual Study Start Date: Jan 15, 2021
• Actual Primary Completion Date: Jan 17, 2021
• Actual Study Completion Date: Jan 20, 2021

Arms and Interventions

Arm	Intervention/Treatment
No Intervention: Delayed Contact This group will not receive an email for at least two days.
Experimental: Social Proof This group will receive an email that points out that vaccine availability for employees will soon be limited. It also emphasizes how many fellow health care workers have been vaccinated. Seeing the behaviors of other people might encourage recipients to copy that behavior.	Behavioral: Social Proof; Email; Behavioral: Scarcity Message; Email
Experimental: Reframing Side Effects and Adverse Reactions This group will receive an email that points out that vaccine availability for employees will soon be limited. It also addresses concerns about the side effects and adverse reactions of the vaccine. Due to potentially overblown concerns about the vaccine caused by the salience of side effects in the (social) media, the email attempts to reframe the risks by explicitly noting the small possibility of being affected by serious side effects and by contrasting that to the more severe effects of COVID-19. As a result of this reframing, recipients might recalibrate their perception of risks and benefits and opt for vaccination.	Behavioral: Reframing; Email; Behavioral: Scarcity Message; Email

Outcome Measures

Primary Outcome Measures

1. Number of Employees Who Scheduled Vaccination Appointments After 3 Days [3 days]

   Binary variable of a scheduled vaccination appointment 3 days after the emails were sent

Secondary Outcome Measures

1. Number of Employees Who Opened Emails After 3 Days [3 days]

   Binary variable indicating whether the employee opened the email 3 days after the emails were sent

2. Number of Employees Who Clicked Scheduling Links After 3 Days [3 days]

   Binary variable indicating whether the employee clicked the link to schedule a vaccination appointment 3 days after the emails were sent

Other Outcome Measures

1. Number of Employees Who Scheduled Vaccination Appointments Until Policy Change (4 Days) [4 days]

   Binary variable of a scheduled vaccination appointment up to the end of January 19, when new distribution policies were enacted.

2. Number of Employees Who Opened Emails Until Policy Change (4 Days) [4 days]

   Binary variable indicating whether the employee opened the email up to the end of January 19, when new distribution policies were enacted.

3. Number of Employees Who Clicked the Scheduling Link Until Policy Change (4 Days) [4 days]

   Binary variable indicating whether the employee clicked the link to schedule a vaccination appointment up to the end of January 19, when new distribution policies were enacted.

Eligibility Criteria

Criteria

• Ages Eligible for Study: N/A and Older
• Sexes Eligible for Study: All
• Accepts Healthy Volunteers: Yes

Inclusion Criteria:

• Employees at Geisinger Health System who have not gotten a COVID-19 vaccination

Exclusion Criteria:

• None

Contacts and Locations

Locations

	Site	City	State	Country	Postal Code
1	Geisinger	Danville	Pennsylvania	United States	17822

Sponsors and Collaborators

• Geisinger Clinic

Investigators

• Principal Investigator: Michelle M Meyer, PhD, Geisinger Clinic

Study Documents (Full-Text)

• Study Protocol and Statistical Analysis Plan - Jul 13, 2021

More Information

Publications

None provided.

• Responsible Party: Michelle N. Meyer, Faculty Co-Director, Behavioral Insights Team, Geisinger Clinic
• ClinicalTrials.gov Identifier: NCT04728594
• Other Study ID Numbers: 2021-0128
• First Posted: Jan 28, 2021
• Last Update Posted: Jul 26, 2021
• Last Verified: Jul 1, 2021

• Individual Participant Data (IPD) Sharing Statement: Yes
• Plan to Share IPD: Yes
• Studies a U.S. FDA-regulated Drug Product: No
• Studies a U.S. FDA-regulated Device Product: No
• Keywords provided by Michelle N. Meyer, Faculty Co-Director, Behavioral Insights Team, Geisinger Clinic: Immunization; Persuasive Communication; Electronic Mail; Economics, Behavioral; Psychology

Study Results

Participant Flow

Arm/Group Title	Delayed Contact	Social Proof	Reframing Side Effects and Adverse Reactions
Arm/Group Description	This group will not receive an email for at least two days.	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also emphasizes how many fellow health care workers have been vaccinated. Seeing the behaviors of other people might encourage recipients to copy that behavior. Social Proof: Email Scarcity Message: Email	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also addresses concerns about the side effects and adverse reactions of the vaccine. Due to potentially overblown concerns about the vaccine caused by the salience of side effects in the (social) media, the email attempts to reframe the risks by explicitly noting the small possibility of being affected by serious side effects and by contrasting that to the more severe effects of COVID-19. As a result of this reframing, recipients might recalibrate their perception of risks and benefits and opt for vaccination. Reframing: Email Scarcity Message: Email
Period Title: Overall Study
STARTED	3178	3198	3190
COMPLETED	3178	3198	3190
NOT COMPLETED	0	0	0

Baseline Characteristics

Arm/Group Title	Delayed Contact	Social Proof	Reframing Side Effects and Adverse Reactions	Total
Arm/Group Description	This group will not receive an email for at least two days.	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also emphasizes how many fellow health care workers have been vaccinated. Seeing the behaviors of other people might encourage recipients to copy that behavior. Social Proof: Email Scarcity Message: Email	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also addresses concerns about the side effects and adverse reactions of the vaccine. Due to potentially overblown concerns about the vaccine caused by the salience of side effects in the (social) media, the email attempts to reframe the risks by explicitly noting the small possibility of being affected by serious side effects and by contrasting that to the more severe effects of COVID-19. As a result of this reframing, recipients might recalibrate their perception of risks and benefits and opt for vaccination. Reframing: Email Scarcity Message: Email	Total of all reporting groups
Overall Participants	3178	3198	3190	9566
Age (Count of Participants)
<=18 years	0 0%
Between 18 and 65 years	0 0%
>=65 years	0 0%
Sex: Female, Male (Count of Participants)
Female	0 0%
Male	0 0%
Race and Ethnicity Not Collected (Count of Participants)
Count of Participants [Participants]	0 0%
Region of Enrollment (participants) [Number]
United States	3178 100%	3198 100%	3190 100%	9566 100%

Outcome Measures

1. Primary Outcome

Title	Number of Employees Who Scheduled Vaccination Appointments After 3 Days
Description	Binary variable of a scheduled vaccination appointment 3 days after the emails were sent
Time Frame	3 days

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title	Delayed Contact	Social Proof	Reframing Side Effects and Adverse Reactions
Arm/Group Description	This group will not receive an email for at least two days.	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also emphasizes how many fellow health care workers have been vaccinated. Seeing the behaviors of other people might encourage recipients to copy that behavior. Social Proof: Email Scarcity Message: Email	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also addresses concerns about the side effects and adverse reactions of the vaccine. Due to potentially overblown concerns about the vaccine caused by the salience of side effects in the (social) media, the email attempts to reframe the risks by explicitly noting the small possibility of being affected by serious side effects and by contrasting that to the more severe effects of COVID-19. As a result of this reframing, recipients might recalibrate their perception of risks and benefits and opt for vaccination. Reframing: Email Scarcity Message: Email
Measure Participants	3178	3198	3190
Count of Participants [Participants]	101 3.2%	207 6.5%	220 6.9%

Statistical Analysis 1

Statistical Analysis Overview	Comparison Group Selection	Delayed Contact, Social Proof
	Comments	We conducted a logistic regression with the two experimental conditions as dummy variables with delayed contact as the reference group.
	Type of Statistical Test	Superiority
	Comments
Statistical Test of Hypothesis	p-Value	<.001
	Comments	We used an a priori threshold of p < .05.
	Method	Regression, Logistic
	Comments
Method of Estimation	Estimation Parameter	Odds Ratio (OR)
	Estimated Value	2.11
	Confidence Interval	(2-Sided) 95%; 1.65 to 2.69
	Parameter Dispersion	Type: Value:
	Estimation Comments

Statistical Analysis 2

Statistical Analysis Overview	Comparison Group Selection	Delayed Contact, Reframing Side Effects and Adverse Reactions
	Comments	We conducted a logistic regression with the two experimental conditions as dummy variables with delayed contact as the reference group.
	Type of Statistical Test	Superiority
	Comments
Statistical Test of Hypothesis	p-Value	<.001
	Comments	We used an a priori threshold of p < .05.
	Method	Regression, Logistic
	Comments
Method of Estimation	Estimation Parameter	Odds Ratio (OR)
	Estimated Value	2.26
	Confidence Interval	(2-Sided) 95%; 1.77 to 2.87
	Parameter Dispersion	Type: Value:
	Estimation Comments

Statistical Analysis 3

Statistical Analysis Overview	Comparison Group Selection	Social Proof, Reframing Side Effects and Adverse Reactions
	Comments	We conducted a logistic regression with the two experimental conditions as dummy variables with delayed contact as the reference group.
	Type of Statistical Test	Superiority
	Comments
Statistical Test of Hypothesis	p-Value	<.001
	Comments	We used an a priori threshold of p < .05.
	Method	Regression, Logistic
	Comments
Method of Estimation	Estimation Parameter	Odds Ratio (OR)
	Estimated Value	1.07
	Confidence Interval	(2-Sided) 95%; 0.88 to 1.30
	Parameter Dispersion	Type: Value:
	Estimation Comments

2. Secondary Outcome

Title	Number of Employees Who Opened Emails After 3 Days
Description	Binary variable indicating whether the employee opened the email 3 days after the emails were sent
Time Frame	3 days

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title	Social Proof	Reframing Side Effects and Adverse Reactions
Arm/Group Description	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also emphasizes how many fellow health care workers have been vaccinated. Seeing the behaviors of other people might encourage recipients to copy that behavior. Social Proof: Email Scarcity Message: Email	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also addresses concerns about the side effects and adverse reactions of the vaccine. Due to potentially overblown concerns about the vaccine caused by the salience of side effects in the (social) media, the email attempts to reframe the risks by explicitly noting the small possibility of being affected by serious side effects and by contrasting that to the more severe effects of COVID-19. As a result of this reframing, recipients might recalibrate their perception of risks and benefits and opt for vaccination. Reframing: Email Scarcity Message: Email
Measure Participants	3198	3190
Count of Participants [Participants]	1481 46.6%	1398 43.7%

3. Secondary Outcome

Title	Number of Employees Who Clicked Scheduling Links After 3 Days
Description	Binary variable indicating whether the employee clicked the link to schedule a vaccination appointment 3 days after the emails were sent
Time Frame	3 days

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title	Social Proof	Reframing Side Effects and Adverse Reactions
Arm/Group Description	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also emphasizes how many fellow health care workers have been vaccinated. Seeing the behaviors of other people might encourage recipients to copy that behavior. Social Proof: Email Scarcity Message: Email	This group will receive an email that points out that vaccine availability for employees will soon be limited. It also addresses concerns about the side effects and adverse reactions of the vaccine. Due to potentially overblown concerns about the vaccine caused by the salience of side effects in the (social) media, the email attempts to reframe the risks by explicitly noting the small possibility of being affected by serious side effects and by contrasting that to the more severe effects of COVID-19. As a result of this reframing, recipients might recalibrate their perception of risks and benefits and opt for vaccination. Reframing: Email Scarcity Message: Email
Measure Participants	3198	3190
Count of Participants [Participants]	562 17.7%	382 11.9%

4. Other Pre-specified Outcome

Title	Number of Employees Who Scheduled Vaccination Appointments Until Policy Change (4 Days)
Description	Binary variable of a scheduled vaccination appointment up to the end of January 19, when new distribution policies were enacted.
Time Frame	4 days

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title

Arm/Group Description

5. Other Pre-specified Outcome

Title	Number of Employees Who Opened Emails Until Policy Change (4 Days)
Description	Binary variable indicating whether the employee opened the email up to the end of January 19, when new distribution policies were enacted.
Time Frame	4 days

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title

Arm/Group Description

6. Other Pre-specified Outcome

Title	Number of Employees Who Clicked the Scheduling Link Until Policy Change (4 Days)
Description	Binary variable indicating whether the employee clicked the link to schedule a vaccination appointment up to the end of January 19, when new distribution policies were enacted.
Time Frame	4 days

Outcome Measure Data

Analysis Population Description

[Not Specified]

Arm/Group Title

Arm/Group Description

Adverse Events

Time Frame	No adverse events were evaluated.
Adverse Event Reporting Description	We only had access to email engagement data and registration data, which was kept separate from clinical data.
Arm/Group Title	Delayed Contact		Social Proof		Reframing Side Effects and Adverse Reactions
Arm/Group Description	This group will not receive an email for at least two days.		This group will receive an email that points out that vaccine availability for employees will soon be limited. It also emphasizes how many fellow health care workers have been vaccinated. Seeing the behaviors of other people might encourage recipients to copy that behavior. Social Proof: Email Scarcity Message: Email		This group will receive an email that points out that vaccine availability for employees will soon be limited. It also addresses concerns about the side effects and adverse reactions of the vaccine. Due to potentially overblown concerns about the vaccine caused by the salience of side effects in the (social) media, the email attempts to reframe the risks by explicitly noting the small possibility of being affected by serious side effects and by contrasting that to the more severe effects of COVID-19. As a result of this reframing, recipients might recalibrate their perception of risks and benefits and opt for vaccination. Reframing: Email Scarcity Message: Email
All Cause Mortality
	Delayed Contact		Social Proof		Reframing Side Effects and Adverse Reactions
	Affected / at Risk (%)	# Events	Affected / at Risk (%)	# Events	Affected / at Risk (%)	# Events
Total	0/0 (NaN)		0/0 (NaN)		0/0 (NaN)
Serious Adverse Events
	Delayed Contact		Social Proof		Reframing Side Effects and Adverse Reactions
	Affected / at Risk (%)	# Events	Affected / at Risk (%)	# Events	Affected / at Risk (%)	# Events
Total	0/0 (NaN)		0/0 (NaN)		0/0 (NaN)
Other (Not Including Serious) Adverse Events
	Delayed Contact		Social Proof		Reframing Side Effects and Adverse Reactions
	Affected / at Risk (%)	# Events	Affected / at Risk (%)	# Events	Affected / at Risk (%)	# Events
Total	0/0 (NaN)		0/0 (NaN)		0/0 (NaN)

Limitations/Caveats

In the original ClinicalTrials record, the researchers preregistered that the study would examine primary and secondary outcomes after two days. This was extended to about three days, which was the time when the delayed control group was emailed. This altered timing was dictated by when the organization's marketing department was ready to send emails to the delayed control group.

More Information

Certain Agreements

All Principal Investigators ARE employed by the organization sponsoring the study.

There is NOT an agreement between Principal Investigators and the Sponsor (or its agents) that restricts the PI's rights to discuss or publish trial results after the trial is completed.