Designing a Mobile App to Support Academic Success for Student Veterans
Study Details
Study Description
Brief Summary
Compared to civilian students in higher education, student Veterans have high rates of mental health disorders (PTSD: 40% vs. 9% and Depression: 24% vs 12.1%). As a result, Veterans with mental health disorders can be more likely to experience academic issues, such as lower enrollment rates and slower degree attainment on average. In addition, student Veterans with mental health disorders can experience substantial challenges with the already-difficult transition to the student role, with difficulties related to education planning, academic skills, and mental health management. Though many student Veterans could benefit from programming embedded in supported education interventions, current in-person VA supported education treatments are often difficult to access - or not available locally - for these Veterans. In addition, there is no widely available, VA-specific online or mobile-app based resource for students, which is a substantial gap in resources for student Veterans.
The goal of this project is to develop and evaluate a comprehensive mobile app for student Veterans with mental health disorders. This intervention will use the principles of Veteran supported education research and manualized treatments to develop a personalized academic success app, VetEd, to address a variety of academic and psychiatric symptom-related educational barriers for student Veterans. Specifically, VetEd will provide a resource to (1) orient student Veterans with mental health disorders to successfully transition to the role of student as defined by their self-created educational roadmap, which will include helping students acquire (2) academic skills, (3) mental health management skills, and (4) up-to-date information on psychiatric, academic, and financial resources to help them successfully meet higher-education expectations.
This overall study will involve three aims: 1) Developing a Veteran-centered educational support app to help student Veterans with mental health disorders to identify their perceived academic needs, app preferences, and evaluate Veteran-centered content; 2) Testing and iteratively revising the VetEd app (n =15) by assessing app software, content, human-computer interface, usability, satisfaction data, and preliminary exploration of changes in educational functioning (course activity completion, academic self-efficacy, and retention; and 3) Completing final revisions of the VetEd app for a future grant application of a larger RCT. This pilot project is significant and innovative in three key respects: (1) it extends services based in previous, effective supported education research to address both psychiatric and academic concerns for Veterans with mental health disorders; (2) is potentially cost-effective and easy to disseminate nationally; and (3) focuses on improving Veterans' academic functioning and quality of life is substantially different than current VA mobile app offerings. Results from this pilot project will inform the development of a Merit Review application for a larger, randomized clinical trial of VetEd with student Veterans.
| Condition or Disease | Intervention/Treatment | Phase |
|---|---|---|
|
Early Phase 1 |
Detailed Description
The goals of this phase are to: 1) gain feedback on the VetEd mobile app via Veteran usability testing; 2) assess Veteran interest, satisfaction, and usability of VetEd; 3) iteratively address any technological concerns, and 4) collect preliminary information on the impact of the VetEd app on academic functioning (retention, perceived course task ability, and educational self-efficacy) and community reintegration. The investigators will conduct pilot usability testing with 15 Veterans who meet eligibility criteria, conducted in three waves. Each usability testing wave will include 5 participants, based on the finding that approximately 80% of usability problems can be detected using a sample of 4 to 5 participant per wave.
Recruitment. Participants will be 15 Veterans recruited from the Bedford VAMC VITAL, CWT, and local colleges. As noted in Phase 1, based on the rate of student Veterans seeking educational support services at the Bedford facility and the number of Veteran students in the community, the investigators should have no difficulty recruiting the proposed number of participants. Additionally, this research group has had no problem meeting prior, higher recruitment goals for the recent VITAL-SEd RCT (n = 43) using similar recruitment strategies.
Screening. Potential participants will be pre-screened by phone to determine if basic study eligibility criteria are met. Individuals who remain eligible after the pre-screen will be invited to attend a comprehensive in-person baseline session. At the beginning of the baseline session, the investigators will provide participants with detailed information about the study and obtain written informed consent. Following informed consent, participants will complete study assessments and, with staff assistance, and download the app to their personal device.
Usability testing procedures. Participants will be provided with the app and log-in information, to utilize at home over a four-week period. During that time, they will complete at least two in-app tasks within per week, including setting up an educational roadmap, tracking course assignments, learning about successful study strategies, completing a coping exercise, and finding information related to GI Bill benefits.
Participants will be evaluated after app use in the following areas: (a) app usefulness:
whether users can successfully complete designated tasks on the app, (b) effectiveness: the ability to accomplish assigned educational app tasks quickly and easily, (c) learnability: the ability to meet pre-determined app navigation goals (e.g. accessing and inputting information into the education roadmap), (d) satisfaction: users' feelings and opinions about the app related to their educational functioning and alignment with mental-health needs, and (e) engagement: initial and sustained interaction between a user and a mobile application. Usability measures will include two quantitative system usability scales (SUS; AES), task completion (quantitative), app log-in and use information (engagement). Perceived usefulness, engagement, and overall satisfaction will also be assessed through a post-app-testing qualitative interview (1.5 hours).
After each round of testing, the investigative team will work with the programming team to improve the app. The following measures will be utilized: Mental Health Diagnostic Measure: The Structured Clinical Interview for the DSM (SCID-5) is a reliable semi-structured instrument that is the gold standard for diagnosing mental health disorders.
System Usability Scale (SUS): The SUS a 10-item measure, scored on a 5-point Likert scale, that assesses human-computer interaction. A SUS score above 58 is regarded as above average, and a SUS score above 80 a score above which participants are likely to recommend the product to friends.
Acceptability E-Scale (AES): The AES is a 5-item validated measure of the acceptability of mobile interventions, which has a cut-off of 80% for acceptability covering overall and specific component satisfaction.
Academic Functioning Outcomes: Increased academic function is the primary non-usability related goal of the VetEd app. Consequently, preliminary educational functioning information related to academic self-efficacy, preparedness for course tasks, and perceived confidence in course completion/retention will also be collected and described. In line with educational research suggesting non-GPA factors can be stronger predictors of educational success and graduation, the investigators will specifically assess academic self-efficacy (Academic Self-Efficacy Scale: ASES), ability to complete course tasks (The Coping With Education Barriers Subscale: CWBS), and confidence in completing coursework and academic retention (qualitative interview).
Community Reintegration: The Military to Civilian questionnaire (M2C-Q) is a 16-item intended to measure dimensions of community reintegration related to productivity (in education, work, and domestic life); social relations; community engagement; and perceived meaning in life, self-care, and leisure.
Data management. VetEd will be developed and hosted on a secure server by Geisel Software, Inc. The app will collect usability and content data, but will not collect personally identifiable information. Coded usernames and IDs will be used for app log-in. When data collection is complete, Geisel Software, Inc. will transfer the coded usability data to the Bedford VAMC research staff. Data will not be shared outside the VA.
Paper research files will be kept at the Bedford VAMC in a locked cabinet and use deidentified ID numbers. Data will be checked for inconsistencies, omissions, and errors. The research assistant will be trained in data management and confidentiality procedures, and be responsible for data entry and management.
Data analysis plan. The main Phase 2 quantitative measures of acceptability and usability will be the Acceptability E-Scale and the System Usability Scale (SUS), respectively. The investigators expect that Acceptability E-Scale and SUS ratings will increase across the 3 waves of app revision, and that by the third wave the investigators will reach the benchmark of 80% on the Acceptability E-Scale and a 68 on the SUS (good acceptability). Additional data will be provided by the qualitative interviews, using RAP processes outlined in Phase 1. The investigators will also assess feasibility and engagement by measuring the proportion of individuals who successfully complete the 4 week app testing, with at least 5 completed activities (of requested 8, 2 per week) or 63% of the program completed, in line with the mean number of completed sessions found in previous mobile interventions for chronic conditions. The investigators will also assess academic functioning (academic self-efficacy, course task completion, and perceived retention/course completion) and community reintegration (M2C-Q) pre- and post-app use for each student Veteran specifically to 1) characterize the sample in terms of baseline academic functioning and community reintegration, 2) describe changes in academic functioning and community reintegration, and 3) test the feasibility of these measures in anticipation of a larger, future RCT.
Study Design
Arms and Interventions
| Arm | Intervention/Treatment |
|---|---|
| Experimental: VetEd Mobile Application Participants be provided with the VetEd mobile app, with instructions on how to utilize the app at home over a four-week period. |
Behavioral: VetEd Mobile Application
Participants be provided with the VetEd mobile app, with instructions on how to utilize the app at home over a four-week period
|
Outcome Measures
Primary Outcome Measures
- System Usability Scale (SUS) [4-week time point]
The SUS a 10-item measure, scored on a 5-point Likert scale, that assesses human-computer interaction. A SUS score above 58 is regarded as above average, and a SUS score above 80 a score above which participants are likely to recommend the product to friends.
Secondary Outcome Measures
- Acceptability E-Scale (AES) [4-week time point]
The AES is a 5-item validated measure of the acceptability of mobile interventions, which has a cut-off of 80% for acceptability covering overall and specific component satisfaction. It will take approximately 2 minutes.
- The Academic Self-Efficacy Scale (ASES) [Baseline and 4-week time point]
An eight-item rapid assessment instrument to assess respondent self-efficacy regarding several academic skills related to academic achievement, such as time management, taking notes, taking tests, and general academic ability.
- The Coping With Education Barriers Subscale (CWBS) [Baseline and 4-week time point]
A subscale of the full 28 item scale, specifically the Coping with Educational Barriers scale (20 items), which assesses participants' degree of confidence in overcoming potential educational barriers (both internal and external).
- Military to Civilian questionnaire (M2C-Q) [Baseline to 4-week time point]
A 16-item intended to measure dimensions of community reintegration related to productivity (in education, work, and domestic life); social relations; community engagement; and perceived meaning in life, self-care, and leisure.
Eligibility Criteria
Criteria
Inclusion Criteria:
-
Currently enrolled in a minimum of at least 2 courses in a higher education setting
-
Current DSM-5 mental health disorder
-
Competent to provide informed consent
-
Owns a smart phone capable of supporting mobile apps
-
Are willing and able to read in English
-
Self-report struggling with current course activities
Exclusion Criteria:
-
Not currently enrolled as a student
-
Current or recent (within 1 month of study entry) moderate or severe DSM-5 alcohol or drug use disorder
-
Cognitive impairment that would interfere with participation
-
Severe suicidal ideation or a psychiatric hospitalization within the past month
Contacts and Locations
Locations
| Site | City | State | Country | Postal Code | |
|---|---|---|---|---|---|
| 1 | VA Bedford HealthCare System, Bedford, MA | Bedford | Massachusetts | United States | 01730 |
Sponsors and Collaborators
- VA Office of Research and Development
Investigators
- Principal Investigator: Erin D. Reilly, PhD, VA Bedford HealthCare System, Bedford, MA
Study Documents (Full-Text)
None provided.More Information
Publications
None provided.- D3897-P
- RX-21-003