RPC: Evaluation of the Research to Policy Collaboration Model

Sponsor

Penn State University (Other)

Overall Status

Completed

CT.gov ID

NCT03671434

Collaborator

William T. Grant Foundation (Other), Penn State Social Science Research Insititute (Other), London School of Hygiene and Tropical Medicine (Other), Child Trends (Other)

322

Enrollment

Locations

Arms

25.3

Actual Duration (Months)

107.3

Patients Per Site

4.2

Patients Per Site Per Month

Study Details

Study Description

Brief Summary

This work aims to evaluate an approach for improving federal legislators' use of evidence-known as the Research-to-Policy Collaboration (RPC) - which seeks to address known barriers to policymakers' use of research, including a lack of personal contact between researchers and policymakers and limited relevance of research translation efforts to current policy priorities. The RPC involves structured processes for identifying policymakers' priorities, building researchers' capacity for nonpartisan responses to current policy priorities, and facilitating ongoing and productive researcher-policymaker interactions. This implementation of the RPC will focus on child and family policies relevant to child maltreatment.

This study assesses both processes for collaboration and policymakers' use of research within a randomized controlled trial (RCT) employing a mixed methods approach-including quantitative and qualitative evaluation of impact. The proposed project will be guided by three overarching questions:

How does the RPC impact researchers and legislative staff?
How does the RPC impact legislative activity?
How might perceptions and experiences of collaboration through the RPC relate to different forms of evidence use among researchers and policymakers?

The RPC's effectiveness will be tested through experimental design (randomization) using qualitative and quantitative assessments of researcher-policymaker interactions and impact. This includes surveying congressional staff and researchers, reviewing records of policymaker's public statements and introduced legislation, and conducting qualitative interviews around researchers' and legislative staffs' experiences with researcher-policymaker collaboration prior to and during the RPC.

Condition or Disease	Intervention/Treatment	Phase
Legislation Policy Child Abuse Family and Household	Behavioral: Research-to-Policy Collaboration Behavioral: Light Touch Policy Training	N/A

Detailed Description

Objective: The aim of this study is to examine the effectiveness of a model for supporting policymakers' use of research evidence within a randomized controlled trial (RCT) employing both qualitative and quantitative analyses of survey, interview, and observational data to understand the use of research in policy and collaboration. Legislators' research use will be assessed with multiple methods, including (1) self-report via a survey protocol, (2) legislative activity (i.e., introduced bills and official statements), and (3) qualitative interviews. Additionally, interviews and surveys will assess researchers' reported efficacy, skill and engagement in public policy, and the extent to which their work is informed by the policy process.

Intervention: The Research-to-Policy Collaboration (RPC) builds upon theoretical and empirical literature that suggests the need to (a) translate research relevant to current policy priorities, and (b) facilitate productive interactions between policymakers and researchers. The RPC is a nonpartisan, manualized model implemented by an intermediary organization that cultivates relationships between researchers and legislative offices. Implementation occurs in two phases involving a series of seven interrelated activities. During the capacity-building phase, the RPC simultaneously aims to support researcher's development of honest knowledge broker skills with training and coaching, while also conducting iterative needs assessments with congressional offices regarding their current priorities and desire for research evidence. In the collaboration phase, researchers with relevant expertise are coalesced into rapid response teams that are matched with legislative offices. Through a scaffolded series of interactions, the RPC model is used to cultivate productive researcher-policymaker relationships and support responses to legislative requests. Evaluating the effectiveness of the RPC is the focus of the current study.

Study Design:

This study employs both qualitative and quantitative analyses of survey, interview, and observational data within the context of an RCT. Three overarching research questions guide this work and align with the study's three different components - self-report via a quantitative survey, observation of research use via coded legislative activity, and ethnography via in-depth interviews and participant observations.

How does the RPC impact researchers and legislative staff? (Self-Report)
Do researchers report improved policy competencies and motivation for conducting policy-relevant research?
Will legislative staff report increases in positive attitudes toward, knowledge of, intended application, and actual use of research evidence?
Might the impact on legislative and research participants will vary as a function of aspects of their collaborative relationship (e.g., enhanced by mutual trust)?
How does the RPC impact legislative activity? (Observation)

Will congressional offices increase their use of research evidence in legislative activities (i.e., bills, official statements)?

How might perceptions and experiences of collaboration through the RPC relate to different forms of evidence use among researchers and policymakers? (Ethnography)
What are the barriers and facilitators of participation in the RPC?
What types of interactions between researchers and policymakers does the RPC influence?
How does the RPC affect participants' assumptions about each other, interactions with each other, and their intentions?
What types of evidence use are supported by RPC participation?

Sampling Framework All study components draw from an initial sampling framework through which researchers and congressional staff who opt into participating in the RPC will be asked to participate in the study; those who agree to participate in the study will be randomized for either the full RPC intervention or a light-touch control group that receives little direct engagement. This study will oversample to combat anticipated attrition. The final sample includes 96 congressional offices (48 in RPC, 48 in control group) and 226 researchers (151 in RPC, 75 in control group). These samples were derived from an identified population of eligible intervention participants.

A population of relevant legislative offices (Senate and House) will be identified based on the number of child/family bills legislator have (co)sponsored. The random selection process will be stratified to ensure proportional partisan representation and representation of legislators who range in activity related to child welfare. Prior to randomization, the population of relevant legislators will be assigned an Activity Ranking based on the number of times a legislator has sponsored a child/family policy bill. Activity Rankings will range from 1-10 such that 10% of legislators who (co)sponsored the fewest bills will be ranked 1, and 10% of legislators who co-sponsored the most bills will be ranked 10. Recruitment and stratified sampling across a continuum of activity on child/family policy will improve bipartisan representation of the sample. Also aiding with bipartisan representation are targeted outreach efforts that intensify along party lines if participation becomes skewed.
Researchers who work on relevant child/family research will be identified via research institutions, professional networks and listservs, literature searches, and referrals from other researchers. Researchers include research-oriented practitioners and program evaluators from both academic and non-academic settings who are interested in translating their expertise with policymakers. Subsequent to opting-in to participate in a policy engagement intervention, researchers are asked to participate in the study. Study participants will be randomized to receive the RPC or the control condition.

Study Component 1: Quantitative Evaluation of Self-Reported RPC Impact A structured survey will be used to assess researchers' capacity for engaging in public policy processes. A corresponding survey protocol will assess legislative staff's attitudes, knowledge, intent, and reported behavior regarding the use of research evidence. Both researchers and legislative staff will be asked about the nature of their collaborative interactions with one another. Surveys will be conducted approximately every three months. Survey constructs are described in outcome measures. Missing data will be handled with multiple imputation because this is best practice when data are not missing at random. Multilevel modelling will be used to examine individual trajectories in outcome variables. Multiple regression will examine changes between select time points; for instance, whether researchers' reported policy skills improve between baseline and subsequent to trainings. These analyses will allow us to examine whether RPC participation is associated with increases in (1) researchers' reported efficacy to engage with policymakers, (2) legislative staff's interest and willingness to use research evidence, and (3) both researchers' and policymakers' positive sentiment toward working together. Longitudinal analyses will further allow us to explore the trajectory of outcome variables over time (e.g., improved researcher competencies, legislators' use of research)-specifically modelling change in evidence use and other outcomes of interest. Analyses of researcher and legislative staff outcomes are all powered at least at a 90% level to detect effect sizes of 0.2 or greater (power analysis conducted with Optimal Design plus Empirical Evidence).

Study Component 2: Quantitative Evaluation of Observed RPC Impact on Legislative Activity To complement the self-reported quantitative assessment of change, use of research evidence in legislative activities will be observed before and following RPC participation. Data will be selected, extracted, and coded prior to quantitative analyses with indicators derived from deductive codes. Legislative data will be selected based upon the use of keywords indicating relevance to child/family. Since each bill will comprise more text than is feasible to code with traditional qualitative methods, relevant provisions will be extracted prior to coding (those containing keywords). Further, all subsection titles for each sampled bill will be reviewed for potential relevance and extracted if research evidence is used but not captured otherwise by keywords. Data on legislative activity will be coded based on a detailed codebook informed by prior findings of how research has been used by policymakers and in legislation. This coding scheme will be applied by two coders and the codebook will be revised and further specified until the intercoder reliability reaches at least .70 and 30% of the data for the first time period has been coded. Subsequently, trained coders will code individually and a lead investigator will randomly sample 10% of these to ensure consistency in coding quality. These codes will be converted into quantitative indicators of research use, which are described in outcome measures. Legislative activity will be assessed annually. Multivariate Poisson regression analyses will assess change in research use in thematic codes of (1) bills and (2) official statements between the RPC intervention and control conditions. Moderation analyses will further explore the extent to which changes in legislative activity depend on the nature of collaboration experiences, other survey data (e.g., policymakers' perceptions of the value of research), and legislator characteristics (e.g., party affiliation, years of age).

Study Component 3: Ethnographic Evaluation of the RPC model The qualitative component of this study is closely linked with the quantitative work and seeks to better understand the experiences of RPC participants and processes underlying the use of research evidence in the RPC context. Using participatory and ethnographic methods, data will be collected from two sources: (1) semi-structured interviews of trial participants and RPC staff, and (2) observations of researcher trainings and researcher-staff meetings. Interview participants will be sampled from the pool of study participants randomly assigned to receive the RPC, including 11 RPC researchers and 11 RPC congressional office staff that will be interviewed prior to their involvement in the RPC, and 11 RPC researchers and 11 congressional office staff that will be interviewed following collaboration activities via the RPC. Additionally, 3 RPC staff will be interviewed prior to and following the RPC. To gain additional insight into the RPC operations and impact, observations will be conducted of trainings for researchers, formal meetings and informal interactions between researchers and congressional staff, and meetings that occur as part of any RPC collaboration. Following ethnographic best practices, "open-observations" of key meetings and training events will be conducted to enrich the interviews with more information about how attendees interact and how they respond to the RPC. Field notes and transcripts will be analysed with thematic analysis and an interpretive policy analysis that explores different rationalities and rhetoric when discussing evidence used in the collaboration.

Study Design

Study Type:

Interventional

Actual Enrollment :

322 participants

Allocation:

Randomized

Intervention Model:

Parallel Assignment

Intervention Model Description:

Participants will be randomly assigned to either a comprehensive intervention or a control group that receives minimal intervention.Participants will be randomly assigned to either a comprehensive intervention or a control group that receives minimal intervention.

Masking:

Double (Participant, Outcomes Assessor)

Masking Description:

Participants: Participants will not be informed of their status as part of an intervention or control group. Provider: No masking Investigator: No masking Outcome Assessor: Survey data will indicate intervention assignment; observational data will be masked such that document coders will not be aware of the condition associated with coded documents.

Primary Purpose:

Other

Official Title:

Testing an Approach to Improve the Use of Evidence

Actual Study Start Date :

Jan 8, 2019

Actual Primary Completion Date :

Feb 17, 2021

Actual Study Completion Date :

Feb 17, 2021

Arms and Interventions

Arm	Intervention/Treatment
Experimental: RPC Researchers Researchers who are assigned to the experimental group that is eligible to receive the full RPC intervention	Behavioral: Research-to-Policy Collaboration The RPC is a behavioral intervention through which RPC Researchers and RPC Congressional Offices are prepared and matched for collaboration. Specifically, congressional offices are asked to identify opportunities for researcher engagement in policy efforts, researchers with expertise related to policy opportunities are identified and prepared to collaborate with congressional offices, researchers and congressional staff are matched for ongoing collaborative partnerships, and both researchers and congressional staff receive ongoing support to facilitate research translation.
Experimental: RPC Congressional Offices Congressional offices that are assigned to the experimental group that is eligible to receive the full RPC intervention	Behavioral: Research-to-Policy Collaboration The RPC is a behavioral intervention through which RPC Researchers and RPC Congressional Offices are prepared and matched for collaboration. Specifically, congressional offices are asked to identify opportunities for researcher engagement in policy efforts, researchers with expertise related to policy opportunities are identified and prepared to collaborate with congressional offices, researchers and congressional staff are matched for ongoing collaborative partnerships, and both researchers and congressional staff receive ongoing support to facilitate research translation.
Active Comparator: Control Researchers Researchers who are assigned to an Active Comparator control group that is enrolled in a light-touch intervention	Behavioral: Light Touch Policy Training Control Researchers are provided information on policy engagement via email.
No Intervention: Control Congressional Offices Congressional offices that are assigned to the control group that receives no intervention

Outcome Measures

Primary Outcome Measures

Change in Reported Use of Evidence [Assessed multiple times through study completion, an average of one year.]
Congressional study participants will be asked how often they have accessed research and used research in decision-making processes in the past 3 months.
Change in Evidence Sources [Assessed multiple times through study completion, an average of one year.]
Congressional study participants will be asked what sources (e.g., personal contacts, academic journals) use to obtain evidence on policy issues.
Change in Interactions with Researchers [Assessed multiple times through study completion, an average of one year.]
Congressional study participants will be asked how often they have interacted with researchers in different settings in the past 3 months.
Change in Attitudes and Readiness Regarding Evidence Use [Assessed multiple times through study completion, an average of one year.]
Congressional study participants will be asked how valuable research is perceived by the staff and within the congressional office as a whole, as well as perceived benefit of social science specifically.
Change in Policy Engagement [Assessed multiple times through study completion, an average of one year.]
Researcher study participants will be asked how frequently they have engaged with policymakers in different activities and at different stages of the policy process.
Change in Policy-Informed Research [Assessed multiple times through study completion, an average of one year.]
Researcher study participants will be asked their perceived value in engaging policymakers in the research process, the extent to which their research activities are informed or guided by policymakers' needs, and the extent to which researchers have actively engaged policymakers in the research process in the last 3 months.
Change in Policy-Related Self-Efficacy [Assessed multiple times through study completion, an average of one year.]
Researcher study participants will be asked how confident and prepared they feel about engaging with policymakers.
Change in Reported Policy Knowledge [Assessed multiple times through study completion, an average of one year.]
Researcher study participants will be asked the extent to which they understand policy processes and norms, and the degree to which they perceive a need for additional policy training.
Change in Official Statement Research Use [Bills that were introduced one year prior to the RPC implementation, during the implementation period and one year following the RPC completion.]
Legislators' public statements will be dichotomously coded to indicate any type of research use. These statement-level indicators will be aggregated by legislator.
Change in Research Use in Legislation [Bills that were introduced one year prior to the RPC implementation, during the implementation period and one year during or following the RPC completion.]
Child/Family bills will be dichotomously coded to indicate any type of research use. These statement-level indicators are then aggregated by legislator.
Change in Intensive Research Use in Legislation [Bills that were introduced one year prior to the RPC implementation, during the implementation period and one year during or following the RPC completion.]
Each section in child/family bills will be dichotomously coded to indicate any type of research use. Each bill will be scored to indicate a proportion of sections in which evidence is used out of the sum of sections in the bill. These statement-level indicators are then aggregated by legislator.
Change in Type of Use in Legislation [Bills that were introduced one year prior to the RPC implementation, during the implementation period and one year during or following the RPC completion.]
Child/family bills will be dichotomously coded to indicate the observation of different types of research use (e.g., conceptual, instrumental). These statement-level indicators are then aggregated by legislator.

Secondary Outcome Measures

Satisfaction with Collaboration [Assessed during the collaboration component of the intervention or up to one year after study initiation.]
Both researcher and congressional study participants will be asked about the quality of the collaborative interactions, such as the adequacy of communication and working styles of fellow collaborators.
Perceived Value of the Partnership [Assessed during the collaboration component of the intervention or up to one year after study initiation.]
Both researcher and congressional study participants will be asked the extent to which the partnership was valuable in relation to time investment and would be worth recommending or doing again.
Perceived Impact of the Collaboration [Assessed during the collaboration component of the intervention or up to one year after study initiation.]
Both researcher and congressional study participants will be asked the extent to which the collaboration improved aspects of participants' work, mutual understanding, or addressed needs and common goals.
Trust and Respect [Assessed during the collaboration component of the intervention or up to one year after study initiation.]
Both researcher and congressional study participants will be asked about their perceptions of the quality of their interactions with collaborative partners, including comfort, trust, shared commitment, and conflict management.

Eligibility Criteria

Criteria

Ages Eligible for Study:

18 Years and Older

Sexes Eligible for Study:

All

Accepts Healthy Volunteers:

Yes

Inclusion Criteria:

Participants who voluntarily enlist in the RPC will be asked to participate in the trial.

Exclusion Criteria:

Participants who choose to stop participating in the study or the RPC itself. All study participants can choose to opt-out of the study at any time.

Contacts and Locations

Locations

	Site	City	State	Country	Postal Code
1	Child Trends	Bethesda	Maryland	United States	20814
2	Penn State University	University Park	Pennsylvania	United States	16802
3	London School of Hygiene & Tropical Medicine	London		United Kingdom	WC1E 7HT