Unbiased estimates of treatment effectiveness in longitudinal randomized clinical trials require meeting many design criteria, especially ensuring full exposure to intervention services. However, engaging participants into interventions, and retaining them at high rates, can be thwarted by everyday challenges faced by disadvantaged populations. We are unaware of studies evaluating effective strategies for engaging and retaining transition-age youth with disabilities in clinical trials of community-based transition programs.
The purpose of this paper is to describe and qualitatively assess the effectiveness of strategies used by Maryland PROMISE staff for reengaging youth, who have disengaged from services, and are at high risk of study dropout.
Data collected from the project’s management information system, and from interviews with staff assigned solely to reengaging participants, was analyzed to describe effective strategies for reengaging youth in program services.
Staff successful at reengaging hard-to-serve youth into program services are persistent, flexible, and trustworthy. They increased the overall engagement rate from about 50 to 80 percent by study endpoint.
An intensive and proactive focus on engagement improves retention rates of youth participating in field-based randomized controlled trials of intervention programs. We suggest investigators conducting similar trials for hard-to-serve populations develop plans and allocating resources for engaging youth in program services.
Youth with disabilities, especially those receiving Supplemental Security Income (SSI)1 from the Social Security Administration (SSA), encounter significant challenges in successfully transitioning from secondary school to post-graduate education and employment. Over their lifespan, SSI youth recipients face elevated risks of long-term poverty, unemployment, isolation from natural community and social supports, and likely lifelong dependence on public assistance programs (e.g., Enayati & Karpur, 2019). To mitigate these dismal futures confronting SSI youth, multiple federal agencies have partnered over recent years to fund efforts to expand capacity of career-focused programs for improving their education, employment, and financial outcomes (e.g., Federal Partners in Transition Workgroup, 2015).
Promoting the Readiness of Minors in Supplemental Security Income (PROMISE) is a joint federal research demonstration of the U.S. Departments of Education (USDOE), Health and Human Services, and Labor, with evaluation support for the demonstration from the Social Security Administration. As the lead federal partner, the USDOE funded six model demonstration projects to address barriers and obstacles to economic independence and promote successful education and employment post-school outcomes for youth who receive SSI. Youth SSI recipients between the ages of 14 to 16 were eligible to enroll in the PROMISE demonstration. The USDOE provided approximately $230 million to the following demonstration projects over a five to six year period which commenced in October of 2013: Arkansas; California; Maryland; New York; Wisconsin; and a six-state consortium which included Arizona, Colorado, Montana, North Dakota, South Dakota and Utah.
The State of Maryland’s PROMISE program enrolled 2,006 SSI youth with disabilities, aged 14 and 16, who were randomly assigned on a 1 : 1 ratio to either the PROMISE services condition (n = 997) or to the control (i.e. usual) services condition (n = 1009). The purpose of this study is to describe the extent to which the Maryland PROMISE program intervention condition engaged youth after random assignment, retained them in services over time, and how those who disengaged from services and at high risk of study dropout were reengaged.
Unbiased estimates of treatment effectiveness require meeting several design criteria, one of which is ensuring exposure to pre-planned treatment services that are delivered with high fidelity to a program’s model standards. Receiving a full “dose” of intervention services requires immediate engagement into a treatment intervention, and high retention rates over the duration of study participation. However, engagement and retention can be thwarted by everyday challenges that disadvantaged populations, such as PROMISE program participants encounter, especially legal status as a minor, disability, and poverty, all of which make it extremely difficult to fulfill basic survival needs, let alone summoning necessary time and energy to fully participate in research intervention studies (e.g., Becker, Boustani, Gellatly, & Chorpita, 2018; Lindsay et al., 2014).
Few formal studies have evaluated or reviewed effective strategies for engaging and retaining participants in field-based clinical trials (e.g., Richard, et al., 2017), and we are not aware of studies reporting on strategies for engaging and retaining transition-age youth with disabilities in community-based transition programs.
2Background: Maryland PROMISE
The key services provided by the Maryland PROMISE’s collaborative, integrated community-based intervention program included (a) assertive case management; (b) work-based learning experiences; (c) benefits counseling; (d) financial literacy services; and (e) information for families about existing resources and services for supporting youth’s attainment of education and employment goals. The 997 youth randomly allocated to the PROMISE program services condition were assigned to one of 27 intervention teams operating across the five geographic regions of the state: (a) Baltimore City; (b) Eastern Maryland; (c) Northern Maryland; (d) Southern Maryland; and (e) Western Maryland. Each intervention team consisted of a case manager, family employment specialist, benefits counselor, and, if necessary, school personnel. They worked collaboratively to deliver all the PROMISE program services, while attempting to retain youth and families in services meeting program standards for frequency, intensity, and continuity. (See also, Luecking, Crane, & Gingerich article in this special issue, for a report on the Maryland PROMISE program’s fidelity of implementation to its model standards.)
Real-time tracking of participants’ service receipt was done with the electronic Maryland PROMISE management information system (eVolve), which produced a fine-grained profile of each participant’s service contacts, service delivery dates, specific types of service received, and progress notes for each contact from date of enrollment through project termination.
In April, 2014, Maryland PROMISE began recruitment and enrollment, reaching its target recruitment goal in February, 2016. Study enrollment started slowly to give staff, contracted by an external agency, adequate time to be oriented to and trained in Maryland PROMISE program procedures. As enrollment proceeded, intervention teams began delivering services before they were fully staffed up to handle planned caseloads, making it very challenging to devote necessary time both to engage youth fully into program services, and to provide services at the frequency, intensity, and continuity specified in program standards. By June, 2016, or about four months after meeting the project enrollment target, 39 percent of all enrolled youth were not consistently receiving PROMISE program services, and therefore at very high risk for study dropout.
Such a high proportion of participants at high risk for treatment dropout can severely threaten clinical trial internal validity and inferences about program effectiveness. Therefore, in June 2016, Maryland PROMISE added five new staff, “specialized case managers” (SCM), devoted exclusively to reengaging participants at risk for dropout, and assigned one each to the five geographic regions. SCMs used various strategies for reengaging participants, including communicating by letters, emails, text messages, and telephone calls; and by meeting in-person with youth. They focused first on reengaging those youth lacking a Family Plan, which articulated the individualized mix of program services for maximizing a youth’s education, employment, and financial outcomes. When an SCM successfully reengaged a youth, he/she would ensure reassignment of the youth back to the PROMISE program intervention team specified in the Family Plan. SCMs and the Maryland PROMISE Program Manager met weekly to review reengagement rates, and for youth not responding to SCM efforts, to discuss alternative reengagement approaches.
In this paper, we address two aims. First, for the entire PROMISE program sample of transition-age SSI youth, we report rates of service engagement, disengagement, and reengagement over the project participation period. Second, we describe and qualitatively assess the effectiveness of strategies used by specialized case managers for reengaging youth disengaged from services and at very high risk of dropping out altogether. We hope this report’s findings modestly address the knowledge gap on strategies for engaging, retaining, and reengaging youth with disabilities in field-based randomized clinical trials, and offer some suggestions for researchers in secondary transition to successfully engage and retain study participants in future studies.
Of the 997 youth in the enrolled Maryland PROMISE intervention program condition, 46% were female and 51% male. Geographically, across the five regions, the largest number of youth were enrolled in the Baltimore City (48%) and the Western (39%) regions, the latter of which covered the largest geographic area, including suburbs of Washington, DC, and other rural areas. The smallest number of youth were enrolled in the Eastern region (28%), which included the rural Maryland coastal areas. With SSA’s 23 disability classifications organized into six higher-order categories, the distribution of primary disability across youth in decreasing order was (a) mental or behavioral health disabilities (48%); (b) intellectual or developmental disabilities (26%); (c) autism-spectrum disorders (10%); (d) medical disorders (6%); (e) sensory disabilities (5%); and (f) other (4%).
3.2Operational definition of youth engagement status
PROMISE operationalized youth program service engagement status as: (a) engaged: documented service contact between a youth and his/her assigned PROMISE program intervention team during the prior 60 days; (b) disengaged: no documented contact between a youth and his/her assigned PROMISE program intervention team for greater than 60 days; and (c) reengaged: disengaged youth with a documented contact between a youth and his/her assigned PROMISE program intervention team subsequent to SCM interventions. Maryland PROMISE case managers (CM), family employment specialists (FES), and specialized case managers (SCM), all entered service contacts with youth as dated case notes into eVolve. Thus, in real-time, a youth’s engagement status (i.e. engaged, disengaged, reengaged,) could be retrieved from eVolve entries: (a) engaged = planned service contacts from assigned intervention teams beginning at the time of project enrollment; (b) disengaged = referral to SCM; (c) reengaged = program intervention team contact subsequent to SCM referral; and (d) study dropout = no further program intervention team contact subsequent to SCM referral.
3.3Specialized case managers’ reengagement strategies
From June 2016 onward, disengaged youth were assigned an SCM, while engaged youth continued with their assigned PROMISE program intervention team as specified in family plans. We examine strategies used by SCMs to reengage youth in program services, using two data sources: eVolve notes and semi-structured interviews conducted by the first author with the SCMs. eVolve notes reported on the type and frequency of SCM reengagement strategies. Semi-structured interviews elicited perspectives on reengagement strategies used by the SCMs to achieve positive outcomes.
eVolve notes. Beginning in June 2016 through project termination (September 30 2018), SCM logged each successful and unsuccessful contact with a disengaged youth; and each reengagement strategy(ies) used at each contact, allowing aggregation of the frequency of reengagement strategies used by SCMs across all disengaged youth.
Semi-structured interviews and thematic analysis. After termination of the Maryland PROMISE project period, the first author conducted one-hour, semi-structured interviews with the SCMs, consisting of a series of open-ended questions (e.g., What was your primary role as a SCM?; What personal attributes and reengagement strategies do you feel helped you reengage youth?), followed by prompts for elaboration. All interviews were audio-recorded and transcribed.
A consensual qualitative research (CQR) approach (Hill, 2012) was adapted to identify common themes emerging from the interviews in several steps. First, the first author summarized each interview’s content in detail, and then analyzed theme for common themes. Second, to identify new themes and critique the first author’s proposed themes, two additional individuals reviewed the interview summaries. Third, the PROMISE principal investigator and first author reviewed all interview summaries and proposed themes, and through consensus discussion, articulated the final set of core themes.
4.1Youth engagement status in program services
By June, 2016, or about four months after meeting the project enrollment target, 39 percent of all enrolled youth had disengaged from Maryland PROMISE program services (i.e. no program intervention team contacts over the past 60 days), and were therefore at very high risk for dropping out of the study. Disengagement rates varied modestly by geographic region, ranging from a low of 28 percent in the Eastern Shore to 48 percent in Baltimore City (Table 1).
|Region||N youth enrolled in PROMISE program||N (%) of youth disengaged from PROMISE program|
|Baltimore City||233||113 (48%)|
|Eastern Shore||150||42 (28%)|
Source: Maryland PROMISE Fidelity Report (June, 2016).
Youth engagement status (i.e., engaged, disengaged, reengaged) in PROMISE program services changed considerably between hiring of SCMs (June 2016) and project termination (September 2018) (Table 2). Of the 997 youth enrolled into PROMISE services, slightly more than half were disengaged from services (i.e. 60 days or more without documented contact) at least once during this time period (n = 501, 50.3%); the remaining youth (n = 497, 49.7%) were continuously engaged throughout the project period. Of the 501 disengaged youth, 297 (59.3%) were ultimately reengaged in program services, and 204 (41.7%) dropped out.
|June 2016||September 2018|
|Engagement status||n (%)||n (%)|
|Engaged||496 (49.7)||496 (49.7)|
|Disengaged||501 (50.3)||204 (20.5)|
|Total||997 (100.0)||997 (100.0)|
Source: Maryland PROMISE eVolve Note. Proportion of disengaged youth at 2016 who reengaged at project termination = 59.3% (297/501 * 100).
4.2Frequency of reengagement strategies
Table 3 summarizes the aggregated frequencies of each reengagement strategy used by the five SCMs (n = 7103). These strategies ranged from low-intensity (letters, emails, phone calls, text messages) to high-intensity (meetings in-person with youth and families) efforts. Telephone calls (36%) were the most commonly used reengagement method, followed by in-person meetings (27%), text messages (22%), regular mail (10%), and emails (5%). Meetings between SCMs and youth and their families typically occurred at their home or a public place.
|Reengagement strategy||n (%)|
|Meeting in person||1934||(27%)|
Source: Maryland PROMISE eVolve.
4.3Key characteristics of specialized case managers
Semi-structured interviews with the SCMs revealed several traits and behaviors that may be associated with effective reengagement into program services of disengaged youth, especially (a) persistence; (b) flexibility; and (c) trustworthiness.
Persistence. Hired directly from regions in which they served, and carefully screened for experience, communication skills, creative problem-solving, can-do attitude, and expertise working with transition-aged youth with disabilities, SCMs reported that it took several attempts to reengage most youth into program services, a finding consistent across profiles of reengagement strategies summarized in Table 3. One SCM reported making 15 home visits in a single day. Another responded that if families were not home, a tag with “sorry we missed you,” was posted on a door with the SCM’s contact information.
As a five-member team, SCMs met weekly to monitor reengagement rates retrieved from eVolve records of service contacts, set weekly goals, discussed strategies for reaching difficult-to-reach youth and families, and regularly updated youth contact information, using SSA-provided lists and Lexis-Nexis search tools to locate youth and families. Most importantly, SCMs reported that multiple contacts of different types (telephone calls, emails, text messages, and meeting in person) were essential to reengage youth, that is, they had to persist and exercise remarkable flexibility and patience for reengaging reluctant youth into program services.
For example, one SCM stated, I would reach out to families in a variety of ways—call, in-person visits—over an extended period. The understanding was that as some point the youth or family will need something, and more than likely they will reach out to the person who is always contacting them. Similarly, another SCM indicated that she had to vary the times she tried to contact the youth. It usually took several attempts before I was able to connect. A similar expression from another SCM captures persistence: I had to stick to it.
Flexibility. SCMs stated that reengaging in Maryland PROMISE services must be made as convenient as possible for youth and their families, and meeting in person at their homes appeared to be an essential and effective strategy. To be available for meeting families in the community, SCMs had to flex their daily work schedules to include evenings and weekends. For example, one SCM spoke of meeting with family in their home late in the evening (8 : 30 pm), after several failed contact attempts, realizing that scheduling future face-to-face meetings during the late evening would be the only available times for the youth and family. Further, to increase chances that youth and families would keep their appointments, SCMs indicated that reminding them in advance with a telephone call or text message was important. Another SCM recounted how she met with a youth and family member at a bus stop, because it was most convenient for them. As one SCM stated. “It was important for us to show families we were committed to helping them meet their needs and meet them wherever and whenever.”
Trustworthiness. To successfully reengage youth and families into PROMISE services, SCMs stressed that establishing trusting relationships took first and highest priority. By explicitly conveying to youth and families that SCMs appreciated the daily challenges these families faced with living in households of limited means, while raising a child with a disability, they communicated unmistakable empathy, the foundation of building trust. Although Maryland PROMISE did not draft a formal protocol for reengaging youth at risk of dropping out, SCMs reported that person-centered counseling techniques, such as motivational interviewing, were efficient and effective approaches for building trust and addressing the youth or family’s reluctance to reengage in program services. Once SCMs succeeded in helping youth and families identify desired educational, employment, and financial outcomes, SCMs showed them how Maryland PROMISE services would help them attain these career-related goals, while meeting their immediate financial, health, transportation, and other needs. For example, one SCM shared that after months of unsuccessful attempted contacts, a youth responded to this SCM for help with schoolwork, and she connected him to tutoring services. From that point, the youth remained engaged in Maryland PROMISE services, graduated high school, obtained paid employment, and enrolled in a postsecondary school.
To put youth and families at ease, SCMs dressed casually for in-home visits, which demonstrated they were genuine and “not above them.” They also sent hand-written birthday cards and notes to communicate their ongoing interest and concerns. For example, one SCM commented
When asked, one family shared that I was the reason they reengaged in Maryland PROMISE services. They said that I showed interest in working with them on their goals and not what I felt was important. In addition, they reported I did what I said I was going to do; checked in regularly; was not bothersome; returned phone calls, texts and emails; and was always pleasantly available to meet with them and their daughter.
Engaging and retaining participants in randomized controlled trials of community-based service programs invariably requires extensive commitment of staff and resources. About halfway through this randomized controlled trial testing the effectiveness of the Maryland PROMISE program to improve educational, employment, and financial outcomes for 997 transition-aged SSI youth allocated to the program services condition, nearly 50 percent (n = 501) had disengaged from services, and were at risk of dropping out of program services altogether. Such attrition rates will threaten study validity and all inferences of program effectiveness.
In response, Maryland PROMISE hired five “specialized case managers” (SCM), whose sole responsibility was to reengage youth and their families at risk of program services dropout. As a five-member team, SCMs met weekly, reviewed reengagement activities, monitored reengagement rates, set weekly goals, and discussed strategies for reaching youth and families not responding to their initial efforts. Of the 501 youth disengaged during and after the time of their hire, SCMs succeeded in reengaging nearly 60 percent (n = 297). Ultimately, by the end of the five-year project period, the final engagement rate of youth randomly assigned to PROMISE program services approached 80% (n = 793 of 997), a considerably higher rate of engagement compared to those reported for many field-based randomized controlled trials (e.g., Gupta et al., 2005; Ingoldsby, 2010).
5.1Implications for practice
Maryland PROMISE demonstrated that youth traditionally defined as hard-to-serve can be engaged and retained into community-based treatment interventions, although the efforts to do so require additional time and resources compared to populations facing fewer obstacles to participating. Alhough hiring full-time staff to focus solely on reengaging youth into program services might not be practical for similar and far less-resourced programs, we hope that the strategies we learned can be applied and modified to meet the requirements of modestly-resourced research intervention projects across diverse settings.
(1) Proactively plan for and track reengagement activities. Participant engagement status should be operationally defined in a way that is consistent with the study protocol’s standards for service frequency, intensity, and continuity, such that any disengagement from services will be rapidly identified and addressed.
(2) Ensure that intervention and/or specialized “engagement” staff are representative of and familiar with the geographic region and the demographic they will serve. Locally-hired staff are more likely to be knowledgeable about existing services and supports available to youth they serve, and be more likely to represent, or be sensitive to, the cultural dynamics of the target population.
(3) Focus on youths’ and families’ motivations and goals. Youth engage when they feel accepted and affirmed. By actively listening to the youths’ motivations, needs, interests, and goals, as well as helping to identify and address barriers to research project participation, treatment staff can emphasize the personal benefits and values to the family derived from ongoing participation.
(4) Prioritize flexibility and family convenience in engagement practices. Traditionally, service providers expect youth to meet with them during regular office or service hours, which are times that families from low SES backgrounds with multiple demands on their resources are not available. The SCMs had reengagement success when meeting youth in their homes during evening and weekend hours, or in other locations convenient and accessible to families.
5.2Implications for research
This study focused on the type and frequency of various treatment engagement strategies, as well as the underlying assumptions or principles associated with achieving reengagement success from the perspective of staff hired to implement them. Future research might explore the motivations to reengage in treatment interventions from the perspective of the youth or families in order to validate these findings, and expand our understanding of what strategies work for whom and under what circumstances. It would also be interesting to explore the application of the strategies used by MD PROMISE staff across other treatment intervention or research projects, particularly those that might not have the robust resources of MD PROMISE. Finally, future studies should further examine and define the traits of staff who successfully retain youth in services.
Youth who have disabilities and live in poverty face many unique challenges. They often live in households struggling to meet basic needs, lack transportation, and in some cases, may have had few positive experiences with community-based service providers. These factors frequently interfere with attempts to form relationships and engage youth and their families in services designed to improve their educational, employment and financial outcomes. This study extends the field’s knowledge on how to engage and reengage this population of youth in such services. While our analysis was limited to this one, we found that an intensive and proactive focus on reengagement can improve retention rates of youth participating in research intervention studies. We suggest that investigators conducting long-term, field-based randomized controlled trials for hard-to-serve populations in the future develop clear, pre-study protocols for engaging and retaining youth in program services, including hiring dedicated staff.
Conflict of interest
None to report.
The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Office of Special Education Programs, associated with PROMISE Award #H418P140002 and under a grant from the Department of Education (cooperative agreement #H418P130005). Selete Avoke served as the project officer. The views expressed herein do not necessarily represent the positions or policies of the Department of Education or its federal partners. No official endorsement by the U.S. Department of Education of any product, commodity, service or enterprise mentioned in this publication is intended or should be inferred.
Becker, K. D. , Boustani, M. , Gellatly, R. , & Chorpita, B. F. (2018). Forty years of engagement research in children’s mental health services: multidimensional measurement and practice elements. Journal of Clinical Child & Adolescent Psychology, 47(1), 1–23. https://doi.org/10.1080/15374416.2017.1326121
Enayati, H. , & Karpur, A. (2019). Impact of participation in school-to-work programs on postsecondary outcomes for youth with disabilities from low-income families. Journal of Disability Policy Studies, 29(4), 235–244. https://doi.org/10.1177/1044207318789419
Federal Partners in Transition Workgroup. (2015). The 2020 federal youth transition plan: A federal interagency strategy. Retrieved from https://www.dol.gov/odep/pdf/20150302-FPT.pdf
Gupta, A. , Calfas, K. J. , Marshall, S. J. , Robinson, T. N. , Rock, C. L. , Huang, J. S. ,...& Patrick, K. (2015). Clinical trial management of participant recruitment, enrollment, engagement, and retention in the SMART study using a Marketing and Information Technology (MARKIT) model. Contemporary Clinical Trials, 42, 185–195. https://doi.org/10.1016/j.cct.2015.04.002
Hill, C. E. (2012). Consensual qualitative research: A practical resource for investigating social science phenomena. Retrieved from https://www.apa.org/pubs/books/4313031
Ingoldsby, E. M. (2010). Review of interventions to improve family engagement and retention in parent and child mental health programs. Journal of Child and Family Studies, 19(5), 629–645. https://doi.org/10.1007/s10826-009-9350-2
Lindsey, M. A. , Brandt, N. E. , Becker, K. D. , Lee, B. R. , Barth, R. P. , Daleiden, E. L. , & Chorpita, B. F. (2014). Identifying the common elements of treatment engagement interventions in children’s mental health services. Clinical Child and Family Psychology Review, 17(3), 283–298. https://doi.org/10.1007/s10567-013-0163-x
Ribisl, K. M. , Walton, M. A. , Mowbray, C. T. , Luke, D. A. , Davidson, W. S. , & Bootsmiller, B. J. (1996). Minimizing participant attrition in panel studies through the use of effective retention and tracking strategies: Review and recommendations. Evaluation and Program Planning, 19(1), 1–25. https://doi.org/10.1016/0149-7189(95)00037-2
Richard, L. , Piper, D. , Weavell, W. , Callander, R. , Iedema, R. , Furler, J. , ... Palmer, V. J. (2017). Advancing engagement methods for trials: the CORE study relational model of engagement for a stepped wedge cluster randomized controlled trial of experience-based co-design for people living with severe mental illnesses. Trials, 18(1), 169. https://doi.org/10.1186/s13063-017-1878-7
Social Security Administration. (2018). 2018 Red book: A summary guide to employment supports for persons with disabilities under the social security disability insurance (SSDI) and supplemental security income (SSI) programs (No. SSA Publication No. 64-030). Retrieved from Author website: https://www.ssa.gov/redbook/documents/TheRedBook2018.pdf