Publication Type: Report

Summary | Download

Impact Evaluation for the Parent Child Plus Program

The Newark Trust for Education (NTE) Parent Child Plus (PC+) program is an evidence-based early childhood education program for families in the Newark, NJ. NTE seeks to evaluate performance by conducting analyses of existing data for a cohort of over 80 families, assessed four times over 46 weeks using observational measures of parenting practices and children’s socio-emotional skills. PC+ is intended to result in “improved child behaviors related to social‐emotional development and self‐regulation skills” (Organizational Research Services [ORS], 2010, p. 23).

The Quality-Impact-Equity Design and Methods (QDM) Toolbox (Smith, Peck, Roy, & Smith, 2019; Smith, Peck, & McNeil, 2020) was used to: (a) reconfigure existing measures for Parenting Practice Quality and Child SEL Skill to maximize reliability and validity for measuring socio-emotional skills and learning (SEL); (b) produce holistic profiles of parent and child skill (e.g., “whole child”) at each timepoint; and (c) apply pattern-centered analytics to estimate impact and equity effects of the PC+ program as implemented in Newark. Please note: We define impact in terms of the actual “in-the-world” structure of causes and effects, not in terms of counterfactuals. A brief description of the QDM methodology is provided in Appendix A (see also Smith et al., 2019)..

The PC+ program results reveal an overall impact pattern that suggests both a strong relation between parent and child skills and an effect of home visitors on both parent and child skills. Although, in almost all cases, the children of parents with high or growing parenting skills outperformed children with low or declining parenting skills, many children with parents in the low-skill profile for Parenting Practice Quality still experienced growth in SEL skills. This finding suggests that PC+ is working as it should, with parents and home visitors both having direct effects on child SEL skill growth. To fully demonstrate the impact of the NTE PC+ program given this “triadic” causal flow, we recommend (a) improving measures of PC+ fidelity and (b) including a small no-program sample of parents and children.

Contact QTurn to request Report Appendices A-I.

Impact Evaluation for the Palm Beach County Quality Improvement System 

Quality Improvement System (QIS) exposure moves afterschool programs to higher quality, increasing access to developmentally powerful settings and building children’s social and emotional learning skills. Higher quality is defined in terms of the quality of instruction (i.e., individuation, basic/advanced SEL, enrichment content), the stability of staff tenure, and evidence of children’s SEL skill growth.

In this study, we used performance data generated by Prime Time Inc. in Palm Beach County and fully pattern-centered methodology to describe the chain of causal effects as a cascade of sequential impacts. We sought to answer two specific questions about implementation and children’s SEL skill growth: What is the impact of QIS exposure on program quality (i.e., best practices, low staff turnover, great content), particularly for programs that have lower program quality at baseline? What is the impact of exposure to high program quality on student SEL skills?

Findings demonstrate that (1) QIS exposure causes program quality improvement to occur and (2) exposure to high quality corresponds to SEL skill growth. Specifically, (1.a) quality increased dramatically over three years of exposure to the Palm Beach County QIS; (1.b) programs with Low Quality at QIS entry improved when exposed to even moderate QIS Fidelity; (2.a.) children exposed to higher-quality programs had greater SEL skill maintenance and gains compared to children exposed to lower-quality programs; and (2.b) children with Low SEL Skill at entry made greater gains at all levels of program quality.

This pattern of findings suggests that the Prime Time QIS design is successfully building the quality of services available in the county in substantively meaningful ways – by increasing the quality of instruction, increasing the tenure of staff, and growing SEL skills for students who need it most.

Youth Program Quality Self-Assessment Pilot Study

Summary

Overall 24 sites within 17 grantees participated in the self-assessment pilot study by assembling staff teams to collect data and score the Youth Program Quality Assessment (PQA).

At each site an average of 5 staff spent an average of 13 staff hours to complete the self-assessment process.

Whether using an absolute standard or group norms as a benchmark for interpretation of data from the Youth PQA Self-Assessment Pilot Study (hereafter called the Pilot Study), quality scores were very positive for participating programs and also reflected the tendency of self-assessment scores to be biased toward higher quality levels.

The quality scores followed the same pattern as outside observer scores in other samples, highest on for issues of safety and staff support and lowest on higher order practices focused on interaction and engagement.

Youth PQA data collected using the self-assessment method demonstrated promising patterns of both internal consistency and concurrent validity with aligned youth survey responses.

Two thirds or more of sites reported that the observation and scoring process helped the selfassessment team to have greater insight into the operation of their programs, talk in greater depth about the program quality than usual, and have more concrete understanding of program quality.

Site directors and local evaluators said that the self-assessment process was a source of good conversations about program priorities and how to meet them. In almost all cases, concrete action followed from the self-assessment process.

Site directors and local evaluators demonstrated the ability to improvise the self-assessment method to fit local needs.

Program directors, site coordinators, and local evaluators have used the Youth PQA and statewide Youth PQA data to generate statewide program change models, suggesting that the instrument and data are useful for setting system-level improvement priorities.

Afterschool Learning at a Distance: Key Themes and Promising Practices

Afterschool Learning at a Distance: Key Themes and Promising Practices describes the experiences and practices of Genesee Intermediate School District: Bridges to Success (GISD) Team Leads and direct staff serving children and families after substantially redesigning afterschool programming due to the COVID-19 crisis. While the original evaluation plan was to do a (second) round of in-person observations (no longer possible due to school closures), QTurn conducted 15 staff interviews via zoom. Key themes from the interviews include professional uncertainty made manageable because of their strong organizational culture, the importance of addressing inequity, adjustment to flexible service, and a focus on a whole child, whole family approach.

Promoting Healthy Development of Young People: Outcomes Framework 2.0

In the summer of 2018, the Local Government Association (LGA) in England commissioned the Centre for Youth Impact to produce an outcomes framework to help partners across the English youth sector to develop and agree on mutual aims to support young people in their local areas. The work was in response to LGA’s consultations that led to its vision statement described in the report, Bright Futures: our vision for youth services, published at the end of 2017. In that report, the authors noted:

“A clear outcomes framework can help to effectively monitor the impact of a service at key milestones to spot where things aren’t working and provide opportunities to make changes where needed. It can also support evidence of collective impact across the system.”

The proposed framework was intended to support partners’ efforts to track and understand the short-, medium-, and longer-term impacts of their work on the lives of young people. The framework needed to be simple and adaptable to provision for different groups of young people and for diverse approaches.

This document is an update on the framework and is the result of two phases of work: an initial phase including desk research and widespread consultation with practitioners, commissioners and elected members, and a second phase to test the proposed framework in action. The work was undertaken by the Centre’s network of regional impact leads and its central team.

Measure Once, Cut Twice: Using Data For Continuous and Impact Evaluation in Education Programs

Frustration and confusion often occur when practitioners require detailed information about program processes for continuous quality improvement (CQI) while policy-makers require evidence of outcome effects for accountability and funding.  Impact studies are often preferred over continuous improvement studies, but they seldom offer useful information to practitioners.  Per the conference theme, this situation leads to a worldview that emphasizes the limitations of social science methods for achieving practical purposes and welcomes arbitrary decision making (i.e., Type-2 error) in the absence of better evidence and arguments. 

This paper describes a generic quality-outcomes design (Q-O design) that meets the need for performance measurement methodology for concurrent and integrated impact evaluation and continuous improvement in the same organization; that is, measure once, cut twice. 

Quality-Outcomes Study for Seattle Public Schools Summer Programs

This quality-outcomes study was designed to both (a) describe performance in Seattle Public Schools (SPS) summer learning programs in ways that are useful to staff and (b) provide evaluative evidence (i.e., validity) for an instructional model that includes challenging academic content and responsive instructional practices.

Results from this study were mainly positive yet partially ambiguous. Summer program offerings were well-attended and characterized by high-quality instructional practices, with a majority of students increasing their literacy and math skills during the program. Findings about the association between exposure to more responsiveness instruction (e.g., quality) and academic skill change were mixed.

Results include:

Positive academic skill change was found in the raw data, including for academically at-risk students. Positive change on the academic performance measures used during the summer program was found for 73% of students, and positive change on the academic achievement tests was found for 74% of students from the 2015 to 2016 school year. Standardized effect sizes for the full sample ranged from medium to large (i.e., dz = .56 – .95) across the seven academic skill measures.

Attendance was regular, and instructional responsiveness was consistently high. Summer program attendance for 21 or more days (out of a total possible 27 days) was observed for 77% of students. Analysis of instructional responsiveness using the Summer Learning PQA revealed three profiles of instructional responsiveness at the point of service: high, medium, and low quality. However, compared to other urban samples, the “low” SPS profile is not very low.

Students in SPS summer programs had similar rates of skill change across profiles of instructional responsiveness in the most rigorous models for 3rd and 4th grade students (N = 535); that is, there was insufficient evidence in support of the hypothesized pattern of differential skill change across profiles of instructional quality. However, these results should be interpreted with caution due to the absence of a true low-quality instructional practices subgroup in the sample. Less statistically rigorous but more theoretically well-specified models for the entire K-4 sample (N = 1060) revealed a positive association between instructional quality and academic skill change, despite the lack of a true low-quality subgroup.

Analyses of academically at-risk students revealed similarly mixed results. In the more statistically rigorous models with grades 3-4, students who entered SPS summer programs below proficient on academic achievement tests for the prior school year (2015-16) showed similar rates of academic skill change across profiles of instruction. In the theoretically well-specified models, academically at-risk students showed greater changes in academic skills in summer programs with higher-quality instructional practices.

Evaluation of Afterschool Improvement Process: Oklahoma 21st Century Community Learning Centers

Since 2007, the Oklahoma State Department of Education has operated a quality improvement system (QIS) for its approximately 100 federally-funded 21st Century Community Learning Centers (OK 21CCLC) afterschool programs with the explicit purpose of improving the performance of these service providers. This report draws upon data from 23 performance measures collected annually over multiple annual program cycles to present findings for reliability, validity, performance change, and effect of intervention fidelity on performance change. These analyses were conducted as part of an ongoing effort to: (a) evaluate over-time change in performance that is the central purpose of the QIS and (b) improve the accuracy and usefulness of performance data available to individual organizations that participate in the QIS.

In general, our findings indicate that the Oklahoma Afterschool Improvement Process is performing in accordance with its purposes: using accurate performance data to incentivize improvement in the quality of services.

Findings for the reliability and validity of the measures include:

  • All of the 23 measures demonstrated acceptable levels of reliability.
  • There is evidence for construct validity at each time point and factorial invariance across time points.

Findings for performance improvement include:

  • Nearly all measures incrementally improved during a four year (2010-2013) period, while a subset demonstrated statistically significant growth.
  • For nearly all measures, lower-performing sites at the baseline year (2010-2011) improved most. A subset of models demonstrated statistically significant effects.
  • The indicator with the largest increase over four years was Targeting At-Risk Students, suggesting that even though the students served became more challenging, service quality was also generally improving.

Findings for intervention fidelity include:

  • Higher fidelity of YPQI implementation is positively associated with growth on nearly all performance measures at over half of all year-to-year time increments, in line with the YPQI theory of change

This report is supplement to a series of annual reports submitted to the Oklahoma State Department of Education over eight years. These reports provide the unadjusted information that was used in the models described in this report. The supplement to the annual performance report for the 2013-14 program year (Sniegowski, Gersh, Smith, & Garner, 2015) provides the unadjusted means and descriptive statistics for all of the items and scales in the study.

Design Study for the Summer Learning Program Quality Intervention (SLPQI)

The Summer Learning Program Quality Intervention (SLPQI) is a continuous improvement intervention for summer learning systems and settings. The intervention includes: (a) standards and measures for high-quality instructional practices, (b) data products and technology for meaningful feedback, (c) a plan-assess-improve cycle at each summer site, and (d) supports necessary to design and implement the prior three parts. The SLPQI focuses on instructional practices that build student skills during summer and increase school success during subsequent school years.

The SLPQI was the subject of a four-year Design Study involving 152 providers in seven cities. In the final year of the study, the SLPQI was implemented citywide in Denver, CO; St. Paul, MN; and Seattle, WA (N = 106 sites). This report presents final specification of the SLPQI design, supports, measures, and performance benchmarks.

Key findings include:

The SLPQI was implemented at moderate to high fidelity, at scale, in three citywide systems with local provision of supports. The proportion of sites implementing the SLPQI at high fidelity was high in all three systems, and partnerships of school districts, city agencies, community-based providers, and quality intermediary organizations developed capacity to implement the SLPQI at scale. A large proportion of non-school-based sites were connected with information about students’ success in the prior school year.

Summer program staff positively valued the SLPQI and the assessor-coach role. System leaders, site managers, and assessors reported that implementation of the SLPQI was a good use of their time and a good fit with their work. They also reported that the Summer Learning Program Quality Assessment (PQA) successfully differentiated between higher and lower quality. Staff valued of the assessor-coach who observed, generated performance feedback, and provided coaching for the site manager.

Performance data indicates that instructional quality and student outcomes improved as predicted by the SLPQI theory of change. Performance data indicates that instructional quality improved from 2015 to 2016. Lower-performing sites improved the most, and high performance was sustained. Innovations were focused on identified areas of low quality: student management of their executive skills, motivation, and emotions. Students in higher-quality summer settings had greater academic skill gains in both 2015 and 2016 compared to students participating in lower-quality summer settings.

Recommendations include (a) marketing the SLPQI in cities with strong summer partnerships; (b) marketing SLPQI to school districts that hope to build summer partnerships; (c) continuing efforts to improve the Summer Learning PQA as a standard for high-quality instruction tailored specifically for students with difficult SEL histories, and (d) conducting a randomized efficacy trial for the SLPQI.