You are viewing a javascript disabled version of the site. Please enable Javascript for this site to function properly.
Go to headerGo to navigationGo to searchGo to contentsGo to footer
In content section. Select this link to jump to navigation

Implementation and impacts of the substantial gainful activity project demonstration in Kentucky

Abstract

BACKGROUND:

The Substantial Gainful Activity (SGA) Project demonstration tested innovations to improve the employment of nonblind vocational rehabilitation (VR) clients receiving Social Security Disability Insurance (SSDI) benefits.

OBJECTIVE:

We describe the SGA Project model components, the implementation experience in Kentucky, and the impact of the innovations on VR service and employment outcomes.

METHODS:

The evaluation used information from site visits and VR administrative data. We estimated impacts by comparing the outcomes of SSDI-only clients who applied for services at randomly assigned offices that implemented the SGA Project innovations to those who applied at other offices.

RESULTS:

Participants did not consistently receive all components of the innovations. Nevertheless, the innovations led to a 17 percentage-point increase in clients with a signed individualized plan for employment within 30 days of application, an 8 percentage-point increase in closures with competitive employment, and nearly 6 percentage-point increase in the number of clients with earnings at or above the SGA level.

CONCLUSIONS:

The early, positive impacts on key outcomes suggest the SGA Project innovations could hold promise for other VR agencies and for a broader set of VR clients. The evaluation illustrates the potential for random assignment demonstrations to test innovations in VR service delivery.

1Introduction

Many Social Security Disability Insurance (SSDI) beneficiaries who return to work acknowledge the support of state vocational rehabilitation (VR) agencies in their efforts (U.S. Government Accountability Office, 2007a). However, there is substantial variation nationwide in the employment and program outcomes of SSDI beneficiaries served by state VR agencies (Stapleton & Martin, 2012). In 2010 the Rehabilitation Services Administration awarded the Institute for Community Inclusion (ICI) a grant to implement the Substantial Gainful Activity (SGA) Project demonstration, in an effort to identify and promote promising VR agency practices that improve the employment of Social Security Administration (SSA) disability beneficiaries. The goal was to help nonblind SSDI-only beneficiaries attain earnings at or above the SGA level. The project’s focus on SSDI-only beneficiaries was due to concerns about the rapid growth in the SSDI program in recent years, and the fact that SSDI beneficiaries have significant work histories that might be leveraged for return-to-work efforts. The demonstration did not target blind SSDI-only beneficiaries because blind individuals are subject to a different set of SSA work incentive provisions and ongoing eligibility criteria related to SGA.

During the demonstration period, SSA defined the monthly SGA amount for nonblind individuals as $1,090 (2015), $1,130 (2016), and $1,170 (2017). SGA-level earnings represent an important milestone for beneficiaries, SSA, and the VR system. For beneficiaries, SGA-level earnings indicate progress on the path to higher income and financial independence. From the government’s perspective, finding ways to encourage and support beneficiaries to work above the SGA level can lead to reduced government expenditures and increased tax revenue. For the many VR agencies that are eligible for reimbursement by SSA for services to beneficiaries, payments occur only if the beneficiary has become employed and achieved nine months of earnings above the SGA level.

ICI developed the SGA Project model, as described by Foley et al. (2020a) and Foley et al. (2020b) in this volume, and recruited two states, Kentucky and Minnesota, to implement the demonstration. With technical assistance from ICI, each state’s VR agency customized the model to fit their service delivery environment (Marrone et al., 2020). This article describes the Kentucky SGA Project model components, the extent to which the model was successfully implemented, findings on the impact of the model on participant outcomes, and lessons learned for policy and practice. The implementation and evaluation of the SGA Project in Minnesota is described in a separate article in this volume (Honeycutt & Kehn, 2018).

1.1Background and intervention

As described in Foley et al. (2019), the literature has shown that a very small percentage of SSA beneficiaries return to paid employment for an extended period of time. Roughly 40 percent of SSA beneficiaries who use VR services increase their earnings in the following year, although a large majority (88 percent) do not earn above the SGA level (U.S. Government Accountability Office, 2007a). There is evidence that some VR agency practices are correlated with better employment outcomes among SSA beneficiary clients (U.S. Government Accountability Office, 2007b). For example, VR agencies that provide clients with specialized benefits counseling, job placement services, and faster service delivery are associated with improved VR outcomes (Honeycutt & Stapleton, 2013; Tremblay et al., 2006). Honeycutt and Stapleton (2013) examined the impact of the amount of time beneficiaries must wait between application and receiving VR services; they found that, 48 months after application, those who waited longer had fewer months of SGA-level employment and benefits than those served more quickly.

The SGA Project model was designed with insights from these findings and data collected and analyzed by ICI and Mathematica (Foley et al., 2020a). The model consists of four innovation components: a faster pace of services and rapid client engagement, financial and benefits planning, job placement services, and a coordinated team approach.

1.2Kentucky office of vocational rehabilitation service delivery environment

The Kentucky Office of Vocational Rehabilitation (OVR) offered a unique service environment to customize and test the SGA Project model. During the period of the demonstration, Kentucky residents had similar employment rates to the national population but lower levels of educational attainment and higher rates of poverty (U.S. Census Bureau, 2018). Although the Kentucky average household income was less than the national average, the average monthly SSDI benefit amount was almost as high—$1,138 for Kentucky compared with the U.S. average of $1,165 (Social Security Administration, 2015). OVR had a slightly lower share of cases that closed with competitive employment compared to closures across all agencies—54 percent versus 59 percent in fiscal year 2014 (Rehabilitation Services Administration, 2016).

OVR agreed to participate in the SGA Project for several reasons. For many years, OVR had been interested in building its capacity with work incentive counseling, and the SGA Project demonstration provided an opportunity to incorporate benefits counseling into day-to-day operations. The SGA Project demonstration also aligned with asset development efforts in Kentucky’s State Plan for Independent Living. The SGA Project also offered an opportunity to build on and guide the state’s Employment First program goals, which seek to improve the employment rates and quality of life for individuals with disabilities.

1.3SGA project innovations versus usual VR services

As previously noted, ICI and OVR collaborated to customize the model for implementation in Kentucky. In this section and Table 1, we compare the SGA Project enhanced services with usual VR services received by Kentucky clients. More detailed information about the treatment sites that delivered enhanced services and training and technical assistance activities is available in Sevak et al. (2017).

Table 1

Substantial Gainful Activity Project Innovations in Kentucky

Enhanced serviceExpectationUsual practice
Rapid client engagement
  Rapid response to referralSchedule application appointment within 24 hours of referraln/a
  Application appointmentConduct application appointment within 10 business days of referraln/a
  Presumptive eligibility determinationDetermine eligibility within 2 business days of applicationEligibility determination within 60 days of applicationa
  IPE developmentDevelop IPE within 30 calendar days of applicationWithin 90 calendar days of eligibility
Financial and benefits planning
  Benefits planning query reviewQuery received within three weeks of applicationAs needed
  Benefits summary analysis coordinationCompleted analysis within eight weeks of applicationRequested through work incentive coordinator (if needed)
  Financial inventoryCompleted financial inventory and resource tool, as neededn/a
  Financial plan addendumOptionaln/a
  Follow-up with work incentives coordinatorOngoing follow-up via coordinated team approach or otherwiseRequested through work incentives coordinator (if needed)
Job placement services
  Pre-IPE meetingConduct pre-IPE meeting with client, as appropriate, to discuss job plans, strategies, and servicesn/a
  Follow-up contact to support job searchWeekly contact with client during job searchAs needed
  Follow-up during supported employmentMonthly contact with client during supported employment/IPSAs needed
  Follow-up meetings/contact during collegeQuarterly contact with client during long-term training/collegeAs needed
  Follow-up contact during employmentWeekly contact with client during first 8 weeks of employmentAs needed
Coordinated team approach
  Initial team meetingTeam meets with client within 5 business days of eligibility determinationn/a
  Follow-up team meetingTeam meets for second time around IPEn/a
  Quarterly team follow-up meetings with clientMeet at least quarterly after second meetingn/a
  Post-employment team follow-up meetingTeam determines mode for quarterly follow-up with client post-employmentn/a

Notes. IPE = individualized plan for employment, IPS = Individualized placement and support, n/a = not applicable. aOffice of Vocational Rehabilitation permitted extensions to the 60-day guideline for eligibility determination if exceptional or unforeseen circumstances occurred, and if the client agreed to the extension.

1.4Pacing and engagement

The aim of the SGA Project enhanced services was to accelerate the VR process for application, eligibility determination, and Individualized plan for employment (IPE) development in ways that differed significantly from usual service provision. Under the enhanced services, VR counselors were to use presumptive eligibility guidelines and complete the eligibility process within two days of application. Under usual practice, Kentucky VR counselors coordinated the eligibility determination process within 60 days. Pacing to an IPE was also faster under enhanced services, with counselors encouraged to complete an IPE within 30 days of application; usual VR practice required an IPE within 90 days of eligibility.

1.5Financial counseling

The SGA Project emphasized financial counseling. While under usual service OVR relied on fee-for-service vendors to provide work incentive coordination assistance, the SGA Project allowed for hiring Kentucky work incentive coordinators who were trained and certified to provide a much richer set of early and ongoing financial counseling. The work incentive coordinators were co-located with OVR staff, with each one assigned to two or three districts to serve exclusively SSDI-only beneficiaries enrolled in the SGA Project demonstration. Coordinators completed benefit summary analyses for clients, provided additional supports including the development of financial inventories, and offered asset development counseling and related coaching. These services occurred after IPE development.

1.6Job placement services

The SGA Project had expectations of earlier and more frequent client contact with job placement specialists under enhanced service than usual service. With enhanced services, the specialists also participated in team meetings with clients, VR counselors, and work incentive coordinators before IPE development; provided follow-up contact to support job search activities; and maintained periodic contact with clients about employment issues.

1.7Coordinated team approach

A key component of the SGA Project was a coordinated team approach for staff to discuss and determine the services needed by SSDI-only clients. The team consisted of a VR counselor, a work incentive coordinator, and a job placement specialist, with the VR counselor acting as the team leader for purposes of scheduling and leading the initial meeting. The first team meeting was to occur within five days of the eligibility determination and before IPE completion. By working as a team, these key OVR staff aligned their views of the client’s goals and needs, resulting in a mutual understanding on the part of the client and team about the VR process, work incentives, vocational abilities, and opportunities for a successful and timely competitive employment outcome. Usual service in OVR did not use a team approach.

2Methods

2.1Random assignment and enrollment

We randomly assigned seven of the 14 OVR service districts in the demonstration to provide enhanced SGA Project services (treatment) and the remaining seven districts to provide usual VR services (control). To divide the districts into groups with similar profiles, we conducted random assignment within pairs of districts matched on geographic region, urban versus rural location, and SSDI-only client employment outcomes before the demonstration began. We randomized districts, rather than individual counselors or clients, to minimize the burden of implementing the demonstration with fidelity.

OVR enrolled all eligible individuals who applied for OVR services between May 1, 2015, and July 29, 2016, into the demonstration. Eligibility criteria included being ages 18 to 64 at application, having a disability other than blindness, receiving SSDI on the basis of one’s own employment, and not receiving SSI at application.

One concern with a treatment and control group design is that contamination can occur if some members of the comparison group receive service changes as a result of the project’s implementation. Based on the qualitative interviews we conducted with staff, it appeared that staff at districts providing usual services were familiar with the SGA Project innovations. Nonetheless, they maintained a business-as-usual service delivery approach. Most staff we interviewed at the control sites expressed little interest in implementing the innovations, such as faster pacing. In addition, staff at the control sites did not have access to technical assistance or enhanced benefits planning and early job placement resources implemented at the treatment sites. Hence, we believe the potential for contamination at the control sites was minimal.

2.2Implementation evaluation

We evaluated the implementation of the demonstration based on: (a) VR administrative data on eligible applicants through April 19, 2017; (b) data collected in Spring 2016 and 2017 during two rounds of site visits featuring multiple interviews with OVR leaders and staff at districts across the state providing enhanced and usual VR services; and (c) interviews with staff at ICI who provided training and technical assistance to OVR throughout the demonstration.

2.3Impact evaluation

Using administrative data from OVRs case management system through April 19, 2017, we examined four pre-specified service and employment outcomes related to the goals of the demonstration. We estimated impacts of the demonstration as a whole, rather than impacts of each of the innovations separately, because all of the enhanced service elements of the SGA Project were available at treatment sites.

To assess whether the innovations had an impact on service pace, we compared the rates at which SSDI-only applicants at the treatment and control sites obtained a signed IPE within 30 days of application. We selected this outcome for pace of services because the IPE is an important service-delivery milestone—services generally do not begin until such a plan is in place and the 30-day window was a goal in the SGA Project innovations. To assess whether the innovations had an impact on client engagement, we examined whether clients disengaged from VR before attaining competitive employment. By this definition, clients whose cases had closed for reasons other than competitive employment were classified as not being successfully engaged in services. We selected this measure for client engagement because it is common for VR applicants determined eligible for services to drop out before services have begun or are completed.

To assess the impact of the innovations on employment, we examined two outcomes. We selected the first, the rate at which applicants closed from OVR with a competitive employment outcome because it is an important goal of VR services. We selected the second, closure with earnings at or above the nonblind SGA level because achieving SGA-level earnings was the demonstration’s ultimate goal. We measured the employment outcomes at the time clients’ cases closed (that is, when they stopped receiving or attempting to receive services); it is possible that employment outcomes could evolve over longer periods of time not covered by our evaluation.

Using a multivariate regression framework, we estimated impacts as the difference in each of these four outcomes between demonstration participants at treatment sites and control sites. We used an “intent-to-treat” design, meaning all applicants meeting the study inclusion criteria at the treatment and control sites are included in the analysis, regardless of whether they received services or disengaged with OVR. While we found little to no differences in client characteristics (Table 2) or pre-demonstration district outcomes, we controlled for pre-demonstration district outcomes and characteristics of the clients, including sex, race, Hispanic ethnicity, age at application, education at application, previous VR application, primary impairment, employment status at application and month of application. The models also include adjustments to standard errors to account for clustering due to office level, rather than individual-level random assignment. We used a wild cluster-bootstrap percentile-t procedure (Cameron et al., 2008). This approach uses bootstrapping to address issues present when estimating cluster-robust standard errors with a small number of clusters (5 to 30); in our analysis each of the 14 offices represented a cluster. The approach is conservative and as such it reduces the probability of falsely concluding the demonstration had significant impacts.

Table 2

Characteristics at Application of Clients at Treatment and Control Sites

CharacteristicTreatmentControlDifferencea
Number of applicants522447
Sex (%)
  Male53.353.7–0.5
  Female46.746.30.5
Race (%)
  White77.071.65.4
  Black21.827.3–5.4
  Other1.11.10.1
Hispanic ethnicity (%)0.80.40.4 **
Age (%)
  18–242.15.4–3.3 ***
  25–3413.611.32.3
  35–4423.425.6–2.2
  45–5432.631.61.0
  55–6428.426.12.2
Education (%)
  No high school diploma10.312.3–1.9
  High school diploma42.336.16.2
  Some postsecondary education23.426.1–2.7
  Associate’s degree10.28.71.5
  Bachelor’s degree or more13.816.9–3.1
Previous VR applicant (%)53.453.7–0.2
Primary impairment (%)
  Sensory/communicative6.13.62.5
  Physical43.542.31.2
  Cognitive/psychosocial47.144.72.5
  Missing impairment3.39.4–6.2 ***
Employment status at application (%)
  Employed15.115.6–0.4
  Not employed84.984.40.4

Note. From Office of Vocational Rehabilitation case file data. aMay not equal the simple difference between the treatment and control estimates due to rounding.  */ **/ ***indicates significantly different from zero at the.10/.05/.01 level.

3Results

3.1Participant characteristics and validity of comparison group

Treatment and control group clients had similar demographic and background characteristics (Table 2), affirming our confidence in the random assignment and the credibility of the control group. OVR served 522 nonblind SSDI-only clients at treatment sites and 447 nonblind SSDI-only clients at control sites. Just over half of control group members were male, 77 percent were white, about 22 percent were black, and less than 1 percent identified as Hispanic. Two percent of treatment clients were transition age (ages 18 to 24); most were ages 45 to 54 or ages 55 to 64 (32 and 28 percent, respectively). Almost half of treatment group clients had cognitive or psychosocial impairments as the primary impairment. At the time of VR application, 42 percent of treatment group clients had earned a high school diploma, 23 percent had some postsecondary education (but no degree), 10 percent had an associate’s degree, and about 14 percent had earned at least a bachelor’s degree. A majority (53 percent) had previous VR closures prior to the demonstration start. The two groups differed significantly on only three characteristics: treatment group members were roughly half a percentage point more likely to be Hispanic, three percentage points less likely to be transition age, and six percentage points less likely to have an unknown impairment.

3.2Implementation

Data collected during the demonstration revealed that while treatment sites successfully delivered many of the SGA Project innovations, participants did not fully or consistently receive all components of the innovation (Table 3). We summarize implementation findings here; additional findings including office to office differences are included in the interim and final evaluation reports (Martin et al., 2017; Sevak et al., 2017).

Table 3

Fidelity of Delivery of Enhanced Substantial Gainful Activity Project Services at Treatment Sites

Enhanced service
Eligibility within 2 days of application (% of all applicants)39.5
Number of business days between application and eligibility (mean among all clients with an eligibility determination)10.7
Signed IPE within 30 days of application (% of all applicants)27.0
Number of days between application and signed IPE (mean among clients with a signed IPE)61.5
Benefits counseling (% of all applicants)63.6
Job placement services (% of all applicants)30.7
Participation in a coordinated team meeting (% of all applicants)55.6
Participation in a team meeting within 5 business days of eligibility (% of clients that had a team meeting)26.9

Notes. From OVR case file data. IPE = individualized plan for employment.

While the goal was for eligibility determination to occur within two days of application, that goal was met for about 40 percent of all nonblind SSDI-only applicants and the average time between application and eligibility was 11 business days. Roughly a quarter had a signed IPE within the project goal of 30 days of VR application and the average number of days from application to IPE was 62 days. More than 60 percent received benefits counseling, and one-third received job placement services, each which were to be provided as appropriate. And although over half of clients participated in an initial team meeting, just 27 percent of those who had a meeting completed it within the project goal of 5 business days after application.

Information we learned from site visits and interviews is consistent with these findings. Staff at the treatment sites encountered challenges that affected their clients’ access to the innovations. First, because the pacing innovation was a major shift from their usual, more deliberate approach to IPE development, some counselors resisted adopting the faster approach at first. Second, delays in obtaining documentation from SSA needed for eligibility determinations and benefits analyses prevented some staff from meeting the pacing targets. Large caseloads and staff turnover also made it difficult for staff to meet the SGA Project’s targets for pacing and service delivery for some clients. Finally, some staff struggled to effectively adopt the coordinated team approach because of logistical barriers related to scheduling team meetings and confusion about the roles and responsibilities of the team members. But over time and with technical assistance from ICI, counselors grew more adept and were better able to meet the pacing and service goals.

3.3Impact estimates

We found that the SGA Project had a large and statistically significant impact on the pace of services (Table 4). At treatment sites, 27 percent of applicants obtained a signed IPE within 30 days of application while 8 percent of control group clients met that target. After accounting for client characteristics and site differences in pacing before the demonstration period, this translates to a 17 percentage-point increase, which indicates that the SGA Project innovations more than doubled the rate of IPE development within 30 days of application.

Table 4

Regression Adjusted Estimates of Impact of Substantial Gainful Activity Project Innovations on Primary Outcomes

MeasureTreatmentControlRegression-adjusted difference
Number of applicants522477
Applicants with a signed IPE within 30 days of application (%)27.08.216.9 **
Applicants who did not drop out before obtaining competitive employment (%)58.853.83.2
Applicants who closed with competitive employment (%)25.514.48.2 **
Applicants who closed with SGA-level earnings (%)8.22.25.7 **

Notes. From Office of Vocational Rehabilitation (OVR) case file data. IPE = individualized plan for employment. Regression models include the predemonstration district mean for the given outcome and client characteristics at application, including age, gender, race, education, primary impairment, previous OVR closure and month of application, which ranged from May 2015 to July 2016.  */ **/ ***indicates significantly different from zero at the.10/.05/.01 level.

Our findings indicate that 59 percent of clients at the SGA Project’s treatment offices had been successfully engaged with VR as of April 2017. However, after controlling for client characteristics at application and district-level differences before the demonstration, we found that this rate is not statistically different from the 54 percent rate we observed among control group clients.

The SGA Project innovations led to a substantial increase in the percentage of cases that closed with competitive employment. The rate of closure with competitive employment as of late April 2017 was nearly 26 percent among clients at the treatment group sites while it was 14 percent at control group sites. We calculated this rate among all applicants and not just those who closed. The regression-adjusted estimates indicate the demonstration increased competitive, integrated employment at closure over this time period by 8.2 percentage points.

Finally, the rate of closure with SGA earnings was 8.2 percent among treatment group clients compared to 2.2 percent among control group clients. The 5.7 percentage-point regression adjusted impact estimate, calculated for all applicants and not just those who closed with employment, implies that the SGA Project innovations led to a nearly three-fold increase in the rate of closure with SGA-level earnings among applicants whose cases had closed by April 2017.

Because OVR still had open cases (33 percent at treatment sites and 39 percent at control sites), the large estimated impacts on employment and SGA-level earnings are likely to change as more cases close. If we assume that open cases at treatment sites close at the pre-demonstration rate for competitive employment, then the employment rate for open cases at the control sites would need to more than double (to about 35 percent) to eliminate the impact of the SGA Project innovations. The rate for the open control group cases needing to double to eliminate the estimated impact suggests that the SGA Project’s positive impact on competitive employment will likely persist after all cases have closed.

3.4Lessons learned

OVR staff described several key lessons learned while implementing the SGA Project innovations. First, counselors became more accepting and better skilled at delivering the enhanced services over time. VR counselors became aware of presumptive eligibility guidelines and more adept at dynamic IPE development strategies. OVR staff also accepted faster pacing as a best practice when they observed clients who were more engaged and saw the first successful VR closures among clients who received faster pacing. Field staff increased the frequency of their collaborations and knowledge exchange with colleagues, most notably via the coordinated team meetings.

Second, while we cannot separately estimate the quantitative impact of any individual enhanced service, OVR staff praised a number of the enhanced service elements in particular. Staff believed that the faster pace of services and the consistent and early involvement of the work incentive coordinators were the innovations’ most important features. Most executive leadership and local managers praised the use of these coordinators as a valuable innovation that was essential to the project’s success. Work incentive coordinators helped reassure clients who frequently expressed apprehension about how earnings would affect their disability benefits. Some staff believed that team meetings also boosted staff morale and proved beneficial for clients. Because these features represented significant departures from the usual services, they likely contributed to the observed impacts, even if we cannot quantify their contributions apart from the other innovations.

3.5Limitations

Readers should note some limitations of this study when interpreting the findings and considering the applicability of the SGA Project innovations to their agencies’ service delivery practices. First, as mentioned earlier, the large share of demonstration cases that were still open at the time we conducted the evaluation mean that the long-term impacts of the demonstration are currently unknown. Second, because all of the innovations were available in treatment sites, we were unable to assess the impacts of each innovation on its own. Third, while the demonstration could have an impact on many outcomes, our findings are limited to the VR service and closure outcomes documented in OVR administrative data.

4Conclusions and implications for VR agencies

The SGA Project innovations appear to have generated early, positive impacts on key service delivery and client outcomes in Kentucky. SSDI-only clients at treatment sites experienced shorter times to IPE development and were more likely to obtain competitive employment with SGA-level earnings than would have occurred in the absence of the SGA Project innovations. Together with the more modest estimated impacts in Minnesota (Honeycutt & Kehn, 2018), these large estimated impacts in Kentucky suggest other agencies may consider adopting features of the SGA Project innovations. We conclude with some implications from the evaluation findings in Kentucky that may be of interest for other agencies.

First, consistent with findings in Minnesota, our findings show that delivering services at a faster pace is feasible. Although some staff expressed concerns about increasing the pace of services, most were able to apply the innovations over time, with no evidence that the accelerated process resulted in negative consequences for staff or clients (Sevak et al., 2017). We recommend that other state VR agencies consider in-service training on faster pacing and dynamic IPE strategies. Although the increased pace of service might not be appropriate for all clients, it is a component of the SGA Project model that any VR agency could adopt or adapt for non-SSDI clients. If implementing a faster pace of service is not feasible because of large caseloads, a VR agency’s staff could attempt a faster pace for clients who would benefit most from it. For example, a faster pace might benefit individuals who are motivated to return to work or clients who have not yet applied for Supplemental Security Income or SSDI. Early intervention strategies applied in other contexts suggest that getting such people into jobs quickly (or helping them retain their jobs) might reduce the chances of their going onto the disability rolls, and work disincentives associated with receiving such benefits might negatively affect their motivation to work and chances of becoming employed (Ben-Shalom, Burns, Contreary, & Stapleton, 2017).

Second, technical assistance and monitoring is essential to ensuring that staff provide innovation services as intended. While we estimate that the demonstration led to large impacts, our finding that not all of the demonstration components were delivered as intended was also true in Minnesota, and it suggests that the impacts could have been larger.

Finally, like VRS in Minnesota, OVR successfully implemented an office-level random assignment evaluation design that might be used by other agencies to rigorously assess the effectiveness of services. Kentucky’s successful implementation of an office-level random assignment design permitted a rigorous test of the SGA Project innovations. Other agencies might use this approach to rigorously assess the effectiveness of new services and programs. Such evidence would further our understanding of which VR service approaches work better than others.

Conflict of interest

None to report.

Acknowledgments

The Institute for Community Inclusion at the University of Massachusetts-Boston and Mathematica Policy Research led the Substantial Gainful Activity Project Demonstration through a grant received from the U.S. Department of Education, Rehabilitation Services Administration. The views and opinions expressed here are those of the authors and do not necessarily reflect the views, opinions, or policies of the Kentucky Office of Vocational Rehabilitation or the Rehabilitation Services Administration. The authors are solely responsible for any errors.

References

1 

Ben-Shalom, Y. , Bruns, S. , Contreary, K. & Stapleton, D. C. (2017, February). Stay-at-work/return-to-work: Key facts, critical information gaps, and current practices and proposals. Mathematica Policy Research. Retrieved from, https://www.mathematica-mpr.com/our-publications-and-findings/publications/stay-at-work-return-to-work-key-facts-critical-information-gaps-and-current-practices-and-proposals

2 

Cameron, A. C. , Gelbach, J. B. & Miller, D. L. (2008) Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics 90(3), 414–427.

3 

Foley, S. M. , Haines, K. & Mock, L. , (2020a). Phase I development of the SGA Model: Use of administrative data and expert opinion to identify key components of the SGA Model. Journal of Vocational Rehabilitation 53(3).

4 

Foley, S. M. , Haines, K. & Mock, L. (2020b). Phase II of the SGA Project: Development of the coordinated team approach intervention. Journal of Vocational Rehabilitation 53(3).

5 

Honeycutt, T. & Kehn, M. (2019), Implementation and impacts of the Substantial Gainful Activity Project demonstration in Minnesota. Unpublished manuscript.

6 

Honeycutt, T. & Stapleton, D. (2013) Striking while the iron is hot: The effect of vocational rehabilitation service wait times on employment outcomes for applicants receiving Social Security disability benefits. Journal of Vocational Rehabilitation 39(2), 137–152.

7 

Marrone, J. , Thelin, R. & Mock, L. (2020) Launching and supporting the SGA Project in Kentucky and Minnesota: Experiences and perspectives of the senior technical assistance team. Journal of Vocational Rehabiltation 53(3).

8 

Martin, F. , Morris, E. , Honeycutt, T. , Livermore, G. , Doubleday, A. , Sevak, P. & Stapleton, D. (2017) Kentucky SGA Project Demonstration: An interim evaluation report. Report submitted to U.S. Department of Education, Rehabilitation Services Administration. Mathematica Policy Research.

9 

Rehabilitation Services Administration. (2016) FY state vocational rehabilitation performance. Retrieved August 1, 2018 from http://www2.ed.gov/programs/rsabvrs/resources/fy2014-state-voc-rehab-performance.pdf

10 

Sevak, P. , Martin, F. , Livermore, G. , Honeycutt, T. & Morris, E. (2017) Kentucky Substantial Gainful Activity (SGA) Project Demonstration: Final evaluation report. Final report submitted to U.S. Department of Education, Rehabilitation Services Administration. Mathematica Policy Research.

11 

Social Security Administration. (2015) Annual statistical report on the Social Security Disability Insurance Program, 2014. (SSA Publication No. 13-11826). Social Security Administration.

12 

Stapleton, D. C. & Martin, F. H. (2012) Vocational rehabilitation on the road to Social Security Disability: Longitudinal statistics from matched administrative data. Report submitted to the Michigan Retirement Research Center. Mathematica Policy Research.

13 

Tremblay, T. , Smith, J. , Xie, H. & Drake, R. E. (2006) Effect of benefits counseling services on employment outcomes for people with psychiatric disabilities. Psychiatric Services 57(6), 816–821.

14 

U.S. Census Bureau. (2018) American fact finder. Retrieved June 1, 2018 from, https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml

15 

U.S. Government Accountability Office. (2007a). Vocational rehabilitation: Earnings increased for many SSA beneficiaries after completing VR services, but few earned enough to leave SSA’s disability rolls. (Publication No. GAO-07-332). Government Accountability Office.

16 

U.S. Government Accountability Office. (2007b). Vocational rehabilitation: Improved information and practices may enhance state agency earnings outcomes for SSA beneficiaries. (Publication No. GAO-07-521). Government Accountability Office.