After-School Program for urban youth: Evaluation of a health careers course in New York City high schools

1.1.Mentoring in Medicine (MIM) addresses an urgent need for minority health professionals and promotes careers in health care for urban youth

Mentoring in Medicine (MIM) is a non-profit organization founded in 2006 by three emergency medicine physicians. MIM seeks to develop non-traditional educational experiences that (1) foster underrepresented minority student interest in the health professions; (2) contribute to the societal goal of increasing workforce diversity; and thereby, (3) improve health services and patient outcomes in underserved minority communities. The core goal is to turn dreamers into health professionals through academic enrichment, leadership development, civic engagement and mentoring. Pursuit of a career in one of the health professions (e.g., medicine, dentistry, nursing or allied health disciplines) is an intended outcome. MIM is based in New York City and also has local activities in the metropolitan Washington, DC area. The focus is on urban youth [22].

From a health policy perspective, MIM is a pipeline program that helps overcome a continuing deficit in the diversity of the applicant pool for health professional schools. A 2014 report from the American Association of Medical Colleges shows that just 4 percent of physicians are African-American; the percentage was 3.3 percent in 2004 [8]. This deficit negatively impacts minority communities where research shows that minority physicians are more likely to specialize in primary care and serve patients who share their backgrounds [26]. Moreover, a cultural connection also produces other benefits including better patient satisfaction and adherence to treatment protocols [8]. Diversifying medicine and the health professions may yield a reduction in health disparities experienced by minority populations [9–11,27].

From an educational policy perspective, MIM introduces students early to career and educational opportunities that will be demanded in the future workforce. It is projected that by 2020, nearly two-thirds of all jobs will require post-secondary education or training beyond high school, with 35 percent requiring a bachelor’s degree and another 30 percent requiring an associate’s degree or some college training [7]. This reality is coupled with the broader national need to increase the number of American students pursuing careers that require preparation in Science, Technology, Engineering and Mathematics (STEM) in a global knowledge economy [29]. Science preparation is a key determinant to success in pursuing and attaining a career in the health professions.

1.2.MIM programming is innovative and versatile, and employs educational strategies that seek to create learning environments best suited to program goals and student needs

MIM conducts three types of programs:

Community-based: The “Yes, I can be a Health Care Professional” Conference is a one-day program offered annually and is now in its 9th year. It attracts up to 2500 students and their parents throughout the metropolitan New York City area. It targets students ranging from middle grades through college who are treated to dynamic and inspiring life storytelling by diverse, successful health professionals who have overcome their own personal challenges and present positive role models from cultural and social backgrounds similar to the students. A mixture of entertainment and education engage and motivate students and parents alike and are an entry point to more than a dozen rotational hands-on participatory demonstration classes offered that day.

Internet-based: The Virtual Science Camp (VSC) is a synchronous e-learning experience that employs a video streaming service enabling lectures to be broadcast live in real-time from a MIM studio [21]. Students log-in remotely from their own computers and can interactively communicate with instructors and fellow students via chat and e-mail. Demonstrations, individual and student group projects, and daily journaling encourage student engagement. It seeks to foster an interest in health care and science and to promote healthy living. Another is the Medical Pathway Program that is both in-person and virtual, offering a holistic 20-week intensive boot camp for premedical college students consisting of sessions with an MCAT test prep coach, a learning specialist, a communications coach, a peak performance coach, and advisors during the medical school application process, including mock interviewing. Another is the Interactive Webinar Series for parents and students that addresses the college admissions process, financial aid, SAT preparation, and other topics that provide the critical information parents and students need to believe that a health career is feasible.

School-based: These programs take two forms, those that are offered as a for-credit elective In-School Program (ISP), and those that are offered as an After-School Program (ASP). The MIM ASP Course and its evaluation is the central focus of this paper.

1.3.What are After-School Programs and how can they make a difference?

After-School Programs (ASPs) are sometimes also referred to in the literature as informal or out-of-school time learning, are distinct from single-focus extracurricular pursuits. They typically operate on a regular basis during non-school hours throughout the academic year, are supervised by adults, offer more than one activity, and involve other youth [17]. The Harvard Family Research Council Project [24,25] reviewed a decade of ASP evaluation research to provide a basis on which to assess the educational potential of ASP, and identify promising practices. They concluded that children and youth who participate can reap a host of positive benefits in several interrelated outcome areas: academic, social–emotional, prevention, and health and wellness. In particular, successful programs focus not just on academic support, but also balance this with enrichment activities that afford opportunities to practice new skills through hands-on, experiential learning that is structured, engaging and enjoyable.

However, rates of participation among economically disadvantaged and minority youth – those who may be most in need of academic assistance – have generally been found to be relatively low for reasons that include inability to pay participant fees, poor access to safe transportation, conflicting obligations, and negative perceptions that such programs are of low quality [17]. An exception may be STEM learning, where children from ethnic backgrounds – historically underrepresented in STEM – have been found to attend an ASP in higher numbers: 24 percent of African-American, 21 percent of Hispanic and 16 percent of Native American, compared to the national average of 15 percent [2].

In the domain of STEM learning, a study of ASP providers and supporters by Krishnamurthi and the Afterschool Alliance [3] found a clear consensus that ASPs help youth to develop an interest in STEM, and develop capacities to productively engage in STEM learning activities. There was also shared agreement that ASPs are best positioned to impact indicators of learning that include participation in STEM learning opportunities; curiosity about STEM topics, concepts or practices; awareness of STEM professions; ability to exercise STEM relevant to life and career skills; and understanding the value of STEM in society. More generally, there is clear empirical support that ASPs can produce a variety of positive benefits in the form of youths’ attitudes and self-perceptions, behavioral adjustment, school grades, and level of academic achievement [13]. A National Research Council study of learning science in informal environments [28] identifies six interrelated aspects of learning to describe what participants do cognitively, socially, developmentally and emotionally in these settings.

But is more student participation necessarily better than less? Measuring the extent of student participation, such as the frequency of attendance or breadth of activities has proven to be an elusive construct in which to demonstrate impacts on academic achievement, as well as behavioral or social–emotional outcomes. Roth, Malone and Brooks-Gunn [31] in a synthesis of the literature focusing on the amount of participation and program effectiveness found few links between the amount of participation and developmental outcomes. They conclude that participation is best viewed as one component of several intersecting influences affecting the association between ASPs and developmental outcomes.

One such influence worthy of further study is engagement (as distinct from intensity, breadth, and duration of participation), and its possible double effect on both participation and impact [15]. Engagement – behavioral, emotional and cognitive – is critical as a contributor to impact. Engaged youth have relationships with caring adults; build competence and a sense of mastery; and reinforce a sense of self-efficacy [6]. Jolly, Campbell and Perlman [23] posit engagement as one of three factors that need to be addressed in achieving student success in the sciences. The other two are capacity building wherein fundamental knowledge necessary to proceed to more rigorous or advanced levels is imparted; and continuity in the form of a pathway that offers resources necessary for advancement. Researchers have found ASPs that also encourage increased parental engagement are an important building block for student success [4]. Each of these elements – engagement, capacity building and staged pathways – is foundational in MIM ASP programming.

1.4.The MIM After-School Program

Mentoring in Medicine’s (MIM) After-School Program (ASP) consists of a curriculum comprising the body’s 11 organ systems presented as discrete modules, along with complementary educational activities including field trips and class projects. The MIM ASP Course evaluated the fall 2014 semester focused on the Gastrointestinal System module.

The Course is designed to capture the imagination and ignite an interest in health and science for disadvantaged youth in order to motivate them to pursue a health or science career. The active experiential learning environment and culturally relevant content reinforces the student’s confidence in their ability to succeed in school, career and life. Along with career mentoring and cognitive achievement, MIM seeks to create an environment conducive to increasing social and emotional learning [13].

The MIM ASP Course is taught in spring and fall semesters at a cohort of MIM selected public and parochial high schools in New York City’s Harlem and the Bronx. The predominantly African-American and Hispanic students are presented with a specially constructed curriculum that is taught by college and graduate students, supervised by science teachers who receive compensation, and volunteer health professionals who impart their expertise and serve as positive role models. The complete MIM ASP Course curriculum covers the body’s 11 organ systems (one organ system per semester), introduces students to a variety of health care careers that relate to each system, and the diseases and conditions that can affect each. Organized field trips to area hospitals, health professional schools and laboratories are included. Evaluations and pre- and post-measures of knowledge gain are conducted at the start and conclusion of each semester. Students, parents and the instructional staff are queried about the program and its potential impact on student academic success and future career choices in health professions.

1.5.MIM’s After-School Program logic model framework

A logic model is a systematic and visual way to present relationships among program resources, activities, intended results and outcomes [32]. A logic model framework was constructed to help assess the performance and effectiveness of the ASP (see Fig. 1).

Fig. 1.

ASP logic model.

Pictured in the model are the Situational factors that characterize the ASP target population who are largely underrepresented minority students in urban settings that have low high school graduation rates, low representation in health careers, and live in communities that are underserved by health care providers. The Goals of the MIM ASP Course are to encourage these students to explore, pursue, prepare for, enroll in, and ultimately graduate from a post-secondary health care education program following high school. This also entails enhancing students’ knowledge, motivation to succeed, and self-efficacy that finds expression in positive character development and resiliency. The Course Inputs consist of a tailored curriculum, selection of schools, instructors and students, parental engagement, and ancillary learning tools such as educational games, labs and field trips. The measurable Outputs include number of students enrolled and completing The Course, number of parents engaged, and completion of evaluation instruments by students, parents and instructors that identify changes in knowledge, attitudes and behaviors. The Outcomes may variously be characterized as Short-Term, observable or occurring during The Course; Mid-Term, occurring after completion of The Course during high school; or Long-Term, occurring after graduation from high school. (Evaluation of Mid- and Long-Term impacts are outside the scope of the present report, and will be addressed in an upcoming longitudinal study.) The linking of Outputs to Outcomes is accomplished by the gathering of Evidence, the Analysis of that evidence as Program Evaluation, and the reporting of results that yield Recommendations for Program Improvement.

1.6.A framework for evaluating After-School Programs

Effective program evaluation does more than simply collect, analyze and provide data. It makes it possible for program managers and stakeholders to learn continually about the effectiveness of a program’s implementation and how to improve it. Coupling evaluation with a logic model offers greater learning opportunities, better documentation of outcomes, and knowledge about what works and why. Having a logic model framework also increases an evaluation’s effectiveness by focusing on posing evaluation questions that have real value, be they questions that help improve a program or questions that prove whether a program worked as planned.

Friedman [16] identifies four phases or types of evaluation that are well suited to assessing the performance, effectiveness, and acceptance of ASP courses in science education: (a) front-end evaluation, implemented as part of the project’s planning stage, is designed to determine an audience’s prior knowledge, expectations, experiences, learning styles and concerns; (b) formative evaluation provides information about how the project can be improved while it is under development; (c) remedial evaluation provides information to improve a project once its design is complete and allows for corrections once the project is underway. Remedial evaluation is an assessment of how all the individual parts are working together and, like formative evaluation, its purpose is to improve educational effectiveness and assure achievement of project goals and objectives; and (d) summative evaluation assesses outcomes or impacts of a fully implemented project. It provides information about project impacts that are tied to its goals and objectives and, ideally, also documents unintended consequences.

Evaluation of the MIM ASP Course as reported here addresses the ongoing collection of evidence that is both Remedial and Summative. Front-end and Formative evaluations were undertaken during previous iterations of The Course in earlier semesters and are reported elsewhere [5]. Currently under planning and development and to be reported later is a longitudinal study of program effectiveness.

1.7.Evaluation questions

Evaluation of the MIM ASP Course entails gathering evidence of two kinds: (a) quantitative Data that look at measurable Program Outcomes; and (b) qualitative Data expressed as Program Feedback. Each type is applied to specific evaluation questions:

Program Outcomes

Knowledge Acquisition: Did students learn The Course material?

Self-Efficacy: Were students more confident about succeeding in their studies and career?

Attitude Toward Health Care: Did students feel more positive about health care?

Engagement with Health Information: Did the students pursue health information?

Program Feedback

How did students and parents feel about The Course?

How did students and staff view The Course’s success in developing skills?

Why did students drop out of The Course?

2.1.The MIM ASP Course being evaluated

The MIM ASP Course and the evaluation methodology were granted an exemption from the IRB process by the Institutional Review Board of the Albert Einstein College of Medicine. The parents gave permission for their students to participate, the students volunteered to attend The Course, and the school administrators (principals) approved The Course being conducted in their schools and the requisite data being collected.

The MIM ASP Course consisted of a curriculum and pedagogical methods that MIM had developed and refined over the last five years. Table 1 characterizes The Course offered in the fall of 2014 which focused on the Gastrointestinal System. The module consisted of the structure, function and diseases of the Gastrointestinal System, including the small and large intestine, liver, gallbladder and pancreas. During the segment on the pancreas, for example, it was noted that nutritionists advise patients with diabetes or who are at high risk for the disease, on their diet and consumption of carbohydrates. Additionally, students received instruction on obesity, healthy eating and how to read food labels. Related biomedical careers were discussed, including physician, nurse, imaging technician, biomedical researcher, public health worker and dietician. Although included in the curriculum, class discussion and student projects, there were no evaluation measures that focused directly on healthy living concepts or health careers, including their specific educational requirements. Consequently, outcomes related to these two areas could not be directly assessed.

The field trip to Touro College of Osteopathic Medicine introduced students to differences in allopathic vs. osteopathic medicine. During the field trip, students learned about two organ systems, examined models and went into the cadaver lab.

The MIM ASP Course curriculum was supplemented by the electronic online resources of the National Library of Medicine (NLM), in particular the Medline Plus website that contains easy to read descriptions of health topics, diseases and conditions (http://www.nlm.nih.gov/medlineplus/). Printed and electronic articles from the NLM Medline Plus Magazine were also provided (www.nlm.nih.gov/medlineplus/magazine). The instructional staff consisted of MIM provided personnel (college students and medical students and health professionals) as well as members of the schools’ science teaching staff. The MIM staff taught content, chaperoned trips, and assisted with labs, while the school teachers integrated The Course into the school culture and assisted with supervision and teaching. The school principals were engaged in initial Course discussions and assisted as needed to resolve school issues occurring during The Course. An important part of the MIM strategy was to engage the students’ parents. The parents were kept informed of the ASP goals, activities, and progress through e-mail newsletters, conferences with workshops, and a series of MIM-developed webinars.

2.2.The schools and students

The NY ASP program has been administered over 70 times since its inception in 2009. Eight different NY high schools have been involved in the MIM ASP Course to date and over 1500 middle and high school students completed The Course. This paper is based on the most recent experience, where 84 students completed the fall 2014 MIM ASP Course.

The five New York high schools included in this fall 2014 study were: All Hallows High School, Cardinal Hayes High School, Knowledge and Power Preparatory Academy International High School, Mount Saint Michael Academy and Frederick Douglass Academy I (middle through high school, but only high school students attended the ASP). All of these schools were in low income neighborhoods, four in the Bronx and one in Manhattan. Three of the schools were all-male parochial schools, and two were co-ed public schools.

The schools were selected to participate in the MIM ASP Course if the Principals and the school staff were eager to participate, were committed to supporting the implementation and evaluation of The Course, and felt that The Course would enhance their STEM goals.

The grade and gender of the student participants are shown in Table 2.

2.3.The information collected

The study employed a number of instruments to collect specific data from the students, parents and instructional staff.

Student Data Collection Instruments:

Parent Data Collection Instrument:

Instructional Staff Feedback:

Table 3 identifies the data collected for, and reported on, in this study. The total number of students originally enrolled in The Course was 124 and 84 completed it. A dropout was operationally defined as a student who missed three or more consecutive classes at the end of the course. Reasons for dropping out are discussed in Section 4.3.

As noted, in Table 3, well over 90% of the requested data was successfully collected.

2.4.Overview of study design and analysis

The study used a quasi-experimental design to draw conclusions about the effect and effectiveness of the MIM ASP Course. Employing a valid experimental–control group design is problematic in the ASP setting. The act of volunteering for The Course implies a level of interest in, motivation for, and commitment to, biology and/or health care that is different, if not greater than, their school cohorts. A meaningful control group for the ASP students would involve not only students with similar demographical and academic variables, but also similar motivation. Even randomly assigning some students to take The Course and others not to take The Course does not preclude the outcomes from being biased by the Hawthorne effect (more positive experience for the ASP students and disappointment for those not selected). A more robust comparison would require that the control group also be given a course – one that differed from the MIM ASP Course on the course factors. Even in that case, providing the “best practices” only in the MIM ASP Course, raises ethical concerns about the treatment of the control group students. In addition to these methodological concerns, the MIM effort had neither the charter nor the resources to conduct a randomized control group study.

In lieu of a control group design, the present study employed a number of pre–post knowledge acquisition, and self-report attitude and behavior measures that used the students as their own control.

The following types of data were used in the analyses:

Significance tests were based on the dependence/independence of the data sets. Where subjects were asked to provide pre- and then post-data, the tests were conducted as paired comparisons (test on dependent measures), in other cases significance tests were employed against their null hypothesis values.

The majority of the information collected was comparative data that was subjected to statistical analysis. These statistical analyses are presented in Section 3 – MIM ASP Course Outcomes. A subset of the information was in the form of descriptive measurements (i.e., percentages lacking a baseline for comparison). These descriptive measurements are presented in Section 4 – MIM ASP Course Feedback.

The use of self-report surveys (vs. non-obtrusive observations) can limit the validity of the findings by introducing recall unreliability and a bias towards socially desirable responses. In an attempt to counter these unwanted effects, the survey instruments were carefully constructed and revised in order to be unambiguous and clearly focused on specific attitudes and behaviors.

3.1.Knowledge acquisition: Did the students learn The Course’s material?

Evidence. A 20-item multiple choice and list-matching test on the Gastrointestinal System was used to measure the students’ knowledge. An example of the knowledge questions on the test is shown below (correct answer in bold).

Digestion takes place in a long tube-like canal called the alimentary canal or the digestive tract. Food travels through these organs in the following order:

All of the 84 students completing the MIM ASP Course were administered the test prior to the discussion of the system and then again at the conclusion of The Course. In prior use the test was found to be a relatively reliable measure of its content (Cronbach’s alpha of 0.66).

After the MIM ASP Course the students were asked to judge the amount of Gastrointestinal System knowledge they had before The Course and how much they felt they had after taking The Course. A 10 point scale, from 10% to 100%, was used to elicit this information.

We also obtained the students’ high school GPAs for use in the analysis.

Analysis. To assess knowledge gain, we performed a paired t-test on the difference found in the students’ pre-test and post-test Gastrointestinal System scores. A t-test was used to determine if the difference in test scores found in our sample of students was large enough to conclude that it was unlikely to have occurred by chance (less than 5%, or 1% of the time) when the average difference between pre–post scores in the entire population of students was zero (no knowledge gain). Students significantly increased (t=18.5, df=83, p<0.01) their understanding of the Gastrointestinal System with the average score increasing from 56% to 86% on the 20-item test.

When asked, the students also recognized that they gained significant knowledge. They estimated knowing 46% of the Gastrointestinal System information prior to the MIM ASP Course and 76% after The Course (t=14.7, df=75, p<0.01).

An analysis of the students’ prior GPAs and their knowledge scores found that 38% of the students who were below the median GPA, actually scored above the class median in post-test knowledge. Furthermore, the correlation between GPA and post-test scores was small and not statistically significant (r=0.18, df=82, p=ns).

Results. It appears that the students learned the MIM ASP Course content and also recognized that they had acquired significant knowledge during The Course. Students across the academic spectrum appeared to have learned The Course content – the GPA was not a predictor of knowledge acquisition.

3.2.Self-efficacy: Were students more confident about succeeding in their studies and career?

Evidence. Self-efficacy can be defined as: An individual’s self-confidence in his/her ability to succeed at difficult tasks and achieve their goals. We measured their perception of self-efficacy by asking the students how the MIM ASP Course affected their self-confidence in six areas. They indicated their change in self-confidence using a 10-point Likert scale (from 1 = No Change to 10 = A Lot of Change, with 5–6 being anchored at Some Change). The questions were:

How much more confident are you that you could learn the material?

How much more confident are you about fitting into a health care field?

How much more confident are you about succeeding in health care training?

How much more confident are you about succeeding in a health care career?

How much more confident are you about contributing to the health of your community?

How much more confident are you about overcoming personal issues to succeed in a career?

Analysis. Self-report protocol can often lead to over-reporting of an affect (e.g., an educational experience). Consequently, our significance test compared the students’ responses to a null hypothesis level 6 (Some Change) on the scale rather than level 1 (No Change). In this way the analysis is more conservative by testing for a significant difference from the Some Change level. The analysis is shown in Table 4.

Table 4 shows the average self-efficacy score for each question, the statistical significance of that score against the null hypothesis of “some impact”, and the confidence interval around the mean score on each question. In general, the narrower the interval the more certain that we can be that the true mean (for the population) is close to the mean found for our sample. In our case, using a 95% confidence interval, we can say with 95% certainty that the actual population mean is within the calculated interval around the sample mean.

Results. The analysis found that the students reported significantly more self-confidence as a result of the MIM ASP Course in all areas – knowledge acquisition, fitting into health care, succeeding in health care training, succeeding in the a health care career, contributing to the health of their community, and overcoming personal issues.

3.3.Attitude toward health care: Did the students feel more positive about health care as a career?

Evidence. At the end of the MIM ASP Course the students were asked to rate the impact of their experience on the following statements using Likert scales:

The Course made me feel a lot more positive about the health care profession (from 1 = Don’t Agree to 5 = Strongly Agree).

The Course made a difference in choosing a health care career (from The Course having 1 = No Impact to 6 = Made it Extremely More Likely).

Analysis. An analysis of the students’ responses to these two questions is shown in Table 5.

In both cases, the students expressed a belief that the MIM ASP Course increased their positive feeling toward, and likely pursuit of, a health care career. Both questions showed statistically significant changes in attitude from the “somewhat agree” or “somewhat more likely” null hypothesis levels. The students were asked at the beginning of The Course how interested they were in pursuing a health care career. Table 6 shows their response to this question crossed with their post-MIM ASP Course attitude towards choosing a health career.

Of the 79 students responding, only three (4%) said that the MIM ASP Course had No Impact on their interest in a health career. Three students (4%) also stated it made the career More Unlikely. 73 students (92%) reported that it had a positive effect and 32 (41%) reported that it made a career in health care Extremely More Likely. The impact appears to be greatest for those who asserted a health care preference at the beginning of the MIM ASP Course relative to those who indicated “maybe” or “no”.

Results. It is common knowledge that individuals providing self-report data are subject to overstating the actual impact of an educational experience. Nevertheless, the magnitude of the results tends to support the conclusion that the MIM ASP Course had a positive effect on the students’ attitude about pursuing a health career.

3.4.Engagement with health information: Did The Course influence the students’ intent to seek out additional health care information?

Evidence. At the end of the MIM ASP Course, the students were asked to indicate if The Course increased their intent to pursue the acquisition of additional health care information. The students responded to the following statements using a 6-point Likert scale (from The Course having 1 = No Impact to 6 = Made it Extremely More Likely). The three areas the students were asked about are:

Analysis. An analysis of the students’ responses to these three questions is shown in Table 7.

In all three cases, the students expressed a belief that the MIM ASP Course increased the likelihood of them seeking out more health care information in the future. All three questions showed statistically significant changes in attitude from the “somewhat more likely” null hypothesis level.

Results. As previously noted, individuals providing self-report data are subject to overstating the actual impact of an educational experience. The students did report that the MIM ASP Course significantly affected their intent to more actively pursue more health care information.

4.1.How did students and parents feel about The Course?

At the end of the MIM ASP Course the students and parents were asked to respond to a number of questions about their assessment of The Course. The students used a 5-point Likert scale (1 = Don’t Agree to 5 = Strongly Agree) to respond to the following questions. The average agreement score is shown on each below:

The Course provided me with a lot of important health care information – 4.2;

The Course was very interesting and well presented – 4.2;

The Course was one of the most important classes I took during the last two years – 3.5;

I would recommend The Course to my friends – 4.3;

When the students were asked if they achieved what they wanted from the MIM ASP Course 81% said “Most” or “Yes”, while only 17% said “Somewhat”, and 3% said “No”.

The parents responded to several questions about their child’s experience:

Did your child get out of The Course what you had hoped? – 83% said “Most”, “All” or “More” than I hoped;

Would you recommend your child take another class in this series? – 85% said “Yes”.

When the parents were asked “Did The Course influence how applicable you feel a health care career is for your child?”, 70 (90%) of the parents said The Course made it “More likely”, “Very much more likely” or “Definitely” want them to pursue. Of these ‘highly affected’ parents 11 (16%) had said before The Course that they were “Not Sure” they wanted their child to pursue a health care career.

By-and-large, the students and parents were pleased with the MIM ASP Course’s composition, presentation and effectiveness.

4.2.How did students and instructional staff view The Course’s success in developing skills?

Each student was required to complete a capstone project about an historical figure in science or health, a health career or a disease. The projects required the students to perform research and to communicate their findings. The class project was quantitatively scored by the instructors. The projects were rated on a 100-point scale. Of the 84 students 83 (99%) completed the projects. A large majority obtained a high grade on the project’s content, quality and presentation. The average score for the students was 85%. A total of 12 (14%) students received scores of 100, while 19 (23%) received scores below 80%. It was not documented how many of these student projects contained content related to newly acquired knowledge about healthy living behaviors, nor evidence of enhanced health literacy that can be linked to specific class discussions.

Qualitative results bearing on skill development came from the students and instructional staff. In the Post-Survey, the students were asked to select the two components of the MIM ASP Course from which they learned the most. Sixty-nine percent (69%) of the responses selected were for Class Presentation Material, Class Discussions, Games and Labs, while Guest Speakers, Internet Resources/Material, Homework and Knowledge Tests only received 14% of the responses.

Eleven (11) members of the instructional staff also rated the effectiveness of eight Course components on a 5-point Likert scale. Instructional Materials, Hands-on Activities and Field Trips received an average rating of at least 4.0 (“Effective”) in impacting the three dimensions of Learning, Student Participation and Interest in Health Care. On the other hand, Instructor Added Material, Guest Speakers and Reading Materials were rated as the least effective (below 4.0) on all three dimensions.

4.3.Why did students drop out of The Course?

All 40 students who dropped out of the MIM ASP Course participated in a structured interview to determine their reasons for dropping The Course. Each student was asked to select one or more reasons. The most frequent reasons for dropping The Course were:

Conflict with other school demands – identified by 68% of the dropped students;

Lost interest – 25% of the dropped students (associated with “too much lecture” in the students’ written responses);

Not enough hands-on activities – 25% of the dropped students.

5.1.Major findings and their implications

Students across the academic spectrum appeared to have learned the MIM ASP Course content as assessed via pre- vs. post-test scores. On the average, students scored 30% higher on the post-test (from 56% to 86%). Furthermore, students’ prior high school GPA was not a significant predictor of their ability to learn The Course material. The students in this analysis demonstrated two behaviors that made it likely they would do well in The Course. First, they all chose to sign-up for The Course, and second, only those individuals who completed The Course were included in the analysis. This implies that the students were interested in, and motivated by, the subject matter. Interest and motivation are key ingredients of academic performance as identified in the National Research Council [28] and the National Science Foundation [16] learning models. Thus, finding lower GPA students with high achievement scores is not surprising [1]. It is encouraging to realize that lower GPA students can be successful in the MIM ASP Course that may motivate them to pursue training in health careers, and that they may ultimately succeed in that endeavor.

Students were also cognizant of their knowledge gains and estimated them to be an average of 30% percentage points above their prior knowledge (from 46% to 76%). This finding reflects on the students’ sense of accomplishment and self-confidence which is important for continued engagement and acquisition of health care knowledge and skills.

In addition, the students reported that the MIM ASP Course significantly increased their self-confidence in their ability to succeed (self-efficacy). The students expressed a significant improvement in five health care related attitudes and an additional increase in their ability to overcome personal issues impacting their career. Self-efficacy is recognized as a prerequisite of mastery and accomplishment that can be affected by after school programs [14]. It appears that the MIM ASP Course experience improved these feelings of self-efficacy. If this feeling persists, the students are more likely to apply themselves to, and be more successful in, their studies [12]. Other research has also found that ASP participation improves self-confidence [13,14].

There is reason to believe that some of these students might apply their reported increases in self-confidence to a career in health care since the students also reported significant positive attitude changes towards choosing a health care career. In addition, the students stated that the MIM ASP Course significantly increased their interest and intent to seek out more information about health care, participate in health care activities, and take more health care courses in high school. Specifically, interest and positive feelings towards STEM subjects in high school have been shown to be predictors of pursuing STEM education in college [12]. As previously noted, while encouraging, these self-reports about intended behaviors are likely overstated and need to be calibrated through additional research measuring actual behaviors.

The instructors, students and their parents all rated the MIM ASP Course highly effective – both in terms of content and presentation [18]. Over 80% of the parents stated their child got out of The Course what the parents’ hoped for and that they would recommend their student take another MIM ASP Course. In addition, 90% of the parents said The Course made it more likely that they would recommend a health care career for their child. The parents’ opinion is particularly noteworthy because students identify their parents as the single most important influencers in their career choice [5]. This impression made on the parents (undoubtedly influenced by discussions with their children) could predispose some otherwise uncommitted students to pursue a health career.

The reasons students identified for dropping the MIM ASP Course provide valuable insight into both the selection of students and potential improvements to the curriculum. The National Research Council [28] and National Science Foundation [16] learning models support the students’ feedback on the importance of hands-on activities (including labs and the class discussions/activities) in getting and keeping the interest of the students. The ASP curriculum continues to integrate more active learning opportunities into The Course.

5.2.Study limitations and future research considerations

Although the paper reports a number of statistically significant findings, the reader should be aware of the study’s limitations. Specifically:

The findings of this study could be strengthened and extended if future research:

It is worth noting that MIM is currently undertaking a multi-year longitudinal study (of both completed and dropped students) to determine the long-term impact of the ASP experience on students’ career choices.

6.1.MIM ASP Course impact and applicability

The MIM ASP Course appears to have achieved its principal educational objectives of providing academic enrichment in human biology and improving attitudes towards a health career for a self-selected population of disadvantaged, underrepresented minority high school students in an urban setting. In particular, the evaluation data for which we have measurable program outcomes indicate that The Course helped the students develop the characteristics necessary for a successful health career, including knowledge acquisition, self-efficacy and positive attitudes toward a career in health.

From both educational and health policy perspectives, the MIM ASP Course shows promise as a model capable of benefiting American society broadly by stimulating high school students’ interest in pursuing health careers, thus, potentially helping to diversify the pipeline of students pursuing a career in medicine, nursing, dentistry and other allied health professions. This may be a step in the process that leads to a reduction in health disparities experienced by minority populations that may one day benefit from these new health care professionals. While the evidence offered here is based on the use of a single learning module (Gastrointestinal System), previous MIM ASP course offerings using other content modules have reported evaluation consistent results [5].

The results achieved are contingent on the MIM ASP Course that was designed to incorporate the social determinants of learning and to create an environment that empowers students while providing a rewarding personal and educational experience. While The Course does not represent the majority of After-School Programs around the country, it does draw upon many of the best practices reported by other investigators working in other after school educational settings [19,24,25,30]. Additionally, MIM has conducted ongoing studies of MIM’s target population of disadvantaged urban youth along with their parents and teachers in order to continually improve the curriculum and impact of The Course on the students [5,22].

For those organizations interested in replicating the MIM ASP Course with similar students in urban settings, MIM has developed a curriculum built around the following best practices that address both educational and socio-cultural considerations. The literature indicates that consideration of the socio-cultural learning factors in minority populations enhances academic success [30].

6.2.Educational best practices

The MIM ASP Course was designed around the characteristics of high quality and effective educational programs [19,24,25,30] in order to capture student interest, foster motivation/engagement, impart knowledge/skills, and change attitudes thus building student confidence in their capability to succeed.

Curriculum/heuristic features of the MIM ASP Course:

The MIM ASP Course was designed to be relevant to the economic, social, community and cultural background of the students. While the basic curriculum covered the anatomy, physiology and pathology of the Gastrointestinal System, The Course also:

MIM educational best practices have evolved over the history of the After-School Program in New York City, which began in 2009, and The Course offerings have expanded. The curriculum has been adapted for middle schools and MIM ASP Courses have been taught in underserved high schools and middle schools in both NYC and Washington, DC. Leveraging the curriculum developed for the ASP, MIM has also developed a more intense two-semester Anatomy and Physiology Course appropriate as an In-School Program (ISP) for-credit elective for high school sophomores, juniors and seniors. Academic rigor, leadership development and community service are fundamental to the unique mission and success of MIM.