Tag: Evaluation Exemplars from QIS Data

Quality-Outcomes Study for Seattle Public Schools Summer Programs

This quality-outcomes study was designed to both (a) describe performance in Seattle Public Schools (SPS) summer learning programs in ways that are useful to staff and (b) provide evaluative evidence (i.e., validity) for an instructional model that includes challenging academic content and responsive instructional practices.

Results from this study were mainly positive yet partially ambiguous. Summer program offerings were well-attended and characterized by high-quality instructional practices, with a majority of students increasing their literacy and math skills during the program. Findings about the association between exposure to more responsiveness instruction (e.g., quality) and academic skill change were mixed.

Results include:

Positive academic skill change was found in the raw data, including for academically at-risk students. Positive change on the academic performance measures used during the summer program was found for 73% of students, and positive change on the academic achievement tests was found for 74% of students from the 2015 to 2016 school year. Standardized effect sizes for the full sample ranged from medium to large (i.e., dz = .56 – .95) across the seven academic skill measures.

Attendance was regular, and instructional responsiveness was consistently high. Summer program attendance for 21 or more days (out of a total possible 27 days) was observed for 77% of students. Analysis of instructional responsiveness using the Summer Learning PQA revealed three profiles of instructional responsiveness at the point of service: high, medium, and low quality. However, compared to other urban samples, the “low” SPS profile is not very low.

Students in SPS summer programs had similar rates of skill change across profiles of instructional responsiveness in the most rigorous models for 3rd and 4th grade students (N = 535); that is, there was insufficient evidence in support of the hypothesized pattern of differential skill change across profiles of instructional quality. However, these results should be interpreted with caution due to the absence of a true low-quality instructional practices subgroup in the sample. Less statistically rigorous but more theoretically well-specified models for the entire K-4 sample (N = 1060) revealed a positive association between instructional quality and academic skill change, despite the lack of a true low-quality subgroup.

Analyses of academically at-risk students revealed similarly mixed results. In the more statistically rigorous models with grades 3-4, students who entered SPS summer programs below proficient on academic achievement tests for the prior school year (2015-16) showed similar rates of academic skill change across profiles of instruction. In the theoretically well-specified models, academically at-risk students showed greater changes in academic skills in summer programs with higher-quality instructional practices.

Evaluation of Afterschool Improvement Process: Oklahoma 21st Century Community Learning Centers

Since 2007, the Oklahoma State Department of Education has operated a quality improvement system (QIS) for its approximately 100 federally-funded 21st Century Community Learning Centers (OK 21CCLC) afterschool programs with the explicit purpose of improving the performance of these service providers. This report draws upon data from 23 performance measures collected annually over multiple annual program cycles to present findings for reliability, validity, performance change, and effect of intervention fidelity on performance change. These analyses were conducted as part of an ongoing effort to: (a) evaluate over-time change in performance that is the central purpose of the QIS and (b) improve the accuracy and usefulness of performance data available to individual organizations that participate in the QIS.

In general, our findings indicate that the Oklahoma Afterschool Improvement Process is performing in accordance with its purposes: using accurate performance data to incentivize improvement in the quality of services.

Findings for the reliability and validity of the measures include:

  • All of the 23 measures demonstrated acceptable levels of reliability.
  • There is evidence for construct validity at each time point and factorial invariance across time points.

Findings for performance improvement include:

  • Nearly all measures incrementally improved during a four year (2010-2013) period, while a subset demonstrated statistically significant growth.
  • For nearly all measures, lower-performing sites at the baseline year (2010-2011) improved most. A subset of models demonstrated statistically significant effects.
  • The indicator with the largest increase over four years was Targeting At-Risk Students, suggesting that even though the students served became more challenging, service quality was also generally improving.

Findings for intervention fidelity include:

  • Higher fidelity of YPQI implementation is positively associated with growth on nearly all performance measures at over half of all year-to-year time increments, in line with the YPQI theory of change

This report is supplement to a series of annual reports submitted to the Oklahoma State Department of Education over eight years. These reports provide the unadjusted information that was used in the models described in this report. The supplement to the annual performance report for the 2013-14 program year (Sniegowski, Gersh, Smith, & Garner, 2015) provides the unadjusted means and descriptive statistics for all of the items and scales in the study.

Design Study for the Summer Learning Program Quality Intervention (SLPQI)

The Summer Learning Program Quality Intervention (SLPQI) is a continuous improvement intervention for summer learning systems and settings. The intervention includes: (a) standards and measures for high-quality instructional practices, (b) data products and technology for meaningful feedback, (c) a plan-assess-improve cycle at each summer site, and (d) supports necessary to design and implement the prior three parts. The SLPQI focuses on instructional practices that build student skills during summer and increase school success during subsequent school years.

The SLPQI was the subject of a four-year Design Study involving 152 providers in seven cities. In the final year of the study, the SLPQI was implemented citywide in Denver, CO; St. Paul, MN; and Seattle, WA (N = 106 sites). This report presents final specification of the SLPQI design, supports, measures, and performance benchmarks.

Key findings include:

The SLPQI was implemented at moderate to high fidelity, at scale, in three citywide systems with local provision of supports. The proportion of sites implementing the SLPQI at high fidelity was high in all three systems, and partnerships of school districts, city agencies, community-based providers, and quality intermediary organizations developed capacity to implement the SLPQI at scale. A large proportion of non-school-based sites were connected with information about students’ success in the prior school year.

Summer program staff positively valued the SLPQI and the assessor-coach role. System leaders, site managers, and assessors reported that implementation of the SLPQI was a good use of their time and a good fit with their work. They also reported that the Summer Learning Program Quality Assessment (PQA) successfully differentiated between higher and lower quality. Staff valued of the assessor-coach who observed, generated performance feedback, and provided coaching for the site manager.

Performance data indicates that instructional quality and student outcomes improved as predicted by the SLPQI theory of change. Performance data indicates that instructional quality improved from 2015 to 2016. Lower-performing sites improved the most, and high performance was sustained. Innovations were focused on identified areas of low quality: student management of their executive skills, motivation, and emotions. Students in higher-quality summer settings had greater academic skill gains in both 2015 and 2016 compared to students participating in lower-quality summer settings.

Recommendations include (a) marketing the SLPQI in cities with strong summer partnerships; (b) marketing SLPQI to school districts that hope to build summer partnerships; (c) continuing efforts to improve the Summer Learning PQA as a standard for high-quality instruction tailored specifically for students with difficult SEL histories, and (d) conducting a randomized efficacy trial for the SLPQI.

Bringing in the Community: Partnerships and Quality Assurance

As a matter of policy, 21st Century Community Learning Centers rely heavily on community organizations to provide a variety of instructional programs. In this way, 21st Century sites tap the depth and breadth of knowledge available in their communities to provide non-traditional learning experiences that can better meet young participants’ need for engagement and relevance than can a simple extension of school-day routine. However, the inclusion of multiple partners along with school-based site staff at any given 21st Century site means that the quality of instruction can be extremely uneven. How do school districts that receive 21st Century grants, and the coordinators of each of their sites, ensure high quality across a wide variety of offerings led by staff from many different organizations? To begin to answer this question, we first explored the extent to which 21st Century sites in Michigan are actually partnering with community organizations. We then researched selected high-quality sites to arrive at an instructional partnerships model of quality assurance practices whose wide adoption could have significant impact on 21st Century policy and on the youth development field as a whole.

Also available for download at Research Gate.