Legacy content

Explore the reports, books, and other publications produced by the QTurn team and provides a foundation of our work and passion.

Theory and Evidence for Quality Improvement Systems and Lower-Stakes Accountability in the Out-of-School Time Sector

This decade-long project was focused on scaling an education policy innovation called quality improvement systems (QIS). QIS are a type of performance-based accountability system (PBAS) that is guided by a lower-stakes design for QIS participant experience. Lower-stakes designs optimize staff expertise and performance using positive-reinforcing informational incentives that are a good with collaborative and data-driven organizational cultures in the education fields. Place-based QIS designs have been successful and sustained in conditions of public sector outsourcing, mixed public-private networks, and performance measurement that characterize “new public management.”

Quality in the out-of-school time sector: Insights from the Youth PQA validation study
Design theory and formative evidence to describe setting dynamics at two distinct levels, (1) the organization-level where professional learning communities are formed and (2) point-of-service level (e.g., classrooms) where professional staff and learners meet.
Download from Research Gate
Quality accountability: Improving fidelity of broad developmentally focused interventions
Design theory and formative evidence to define (1) the system/network-level within which organization and point-of-service settings are materially nested and (2) the lower-stakes QIS design that networks and organizations should implement for improved performance.
Download from Research Gate
Final report on the Palm Beach quality improvement system pilot: Model implementation and program quality improvement in 38 after-school programs
Mixed methods study of QIS implementation and effects in afterschool programs in Palm Beach County, Fl.
Download Final Report
Quality accountability in the out of school time sector
Download Chapter
Continuous quality improvement in afterschool settings: Impact findings from the Youth Program Quality Intervention study
Multi-site randomized controlled trial that identifies a substantively large and statistically significant cross-level cascade of QIS effects from network to organization to point-of-service instruction.
Download Full Report
Understanding the “how” of effective quality improvement: Lessons from the Rhode Island Program Quality Intervention
Case study of scaled QIS implementation in Rhode Island with focus on manager skills and implementation at the organization level.
Download from Research Gate
Building citywide systems for quality: Guide and case studies for afterschool leaders
How-to guide for cities drawing on experience from implementation of QIS in 40+ cities and states.
Download Guide
Moving the needle on moving the needle: Next stage technical guidance for performance based accountability systems in the expanded learning field with a focus on performance levels for the quality of instructional services
This paper includes: (1) An update to lower-stakes accountability theory and nomenclature with four case studies, (2) empirical evidence about reliability of quality ratings composed to represent a program site using longitudinal data from the Palm Beach county QIS, and (3) guidance about how to align benchmarks and incentives in the lower-stakes accountability designs.
Download Full Report

Pattern-Centered Methodology for Cross-level Effects and Within-Level Interactions

This project explored the potential of pattern-centered methodology to provide a cost-effective approach to addressing s two key challenges for performance data produced through QIS: (1) Effects are produced through cross-level cascade from network to organization to point-of-service to individual learner (e.g., unit of analyses problems for linear models) and (2) point of service level environments (e.g., classrooms) entail a complex integration of higher order interactions (e.g., Cronbach’s hall of mirrors problem of unidentified subgroups). These papers identify a key performance indicator – membership in the low-quality profile for quality of instruction – and then use pattern-centered methods to link low quality instruction to both manager practices and child/youth engagement with instruction.

Managing for positive youth development: Linking profiles of after-school program management to profiles of staff instructional performance
This paper uses pattern centered methods to describe the association between four collaborative management practices and the quality of instruction available to youth.
Download from Research Gate
Quality at the point of service: profiles of practice in after-school settings
This paper uses pattern centered methods to identify three different quality profiles – high, medium, and low – that characterize staff instruction in out-of-school time programs.
Download from Research Gate
Linking after-school instructional practices to youth engagement: A pattern-centered approach
This paper uses pattern centered methods to describe the association between the quality of instruction available and youth’s level of mental engagement with that instruction.
Download From Research Gate
Measuring youth skills in expanded learning systems: Case study for reliability and validity of the YDEKC skill measures and technical guidance for local evaluators
This paper uses pattern-centered methods to increase the usefulness of information available from survey-based skill measures, with a focus on using survey-based skills measures to detect skill change over time.
Download From Research Gate

SEL Performance Standards, Measures, and Benchmarks

This project used a mixed methods approach to identify best practice standards, develop a suite of performance measures, and establish performance benchmarks – for teacher practices and youth skills in the area of social and emotional learning (SEL).

Afterschool quality systems
This paper describes areas of broad consensus in the out-of-school time field, including the growing focus on SEL skills.
Download Publication
Preparing youth to thrive: Promising practices in social and emotional learning
The guidebook describes SEL standards and organizational and curriculum features for a set of eight exemplary SEL programs.
Download Guidebook
Preparing youth to thrive: Methodology and findings from the social and emotional learning challenge
This technical report describes methodology and findings for (1) best-practice SEL standards, (2) validation of a suite of SEL performance measures for use in QIS, and (3) performance benchmarks for out-of-school time programs focused on building SEL skills with vulnerable children/youth.
Download Full Report
Evaluation of Program Quality and Social and Emotional Learning in American Youth Circus Organization Social Circus Programs
This report exemplifies use of the suite of SEL measures and benchmarks in the American Youth Circus Organizations network.
Download Full Report

Evaluation Exemplars from QIS Data

This project includes papers from school-based QIS - 21st CCLC systems in Michigan and Oklahoma; school district-based summer learning programs Seattle, Denver, and St. Paul - to demonstrate the types of performance evaluations that are possible using QIS data.

Bringing in the community: Partnerships and quality assurance in 21st century community learning center
This paper describes community partnerships in Michigan’s 21st CCLC programs.
Download From Research Gate
Oklahoma afterschool improvement process: Evaluation of 5 year trajectories
This paper describes validity of QIS performance measures and longitudinal change over four years in Oklahoma 21st CCLC programs.
Request Report
Design study for the Summer Learning Program Quality Intervention (SLPQI): Final-year intervention design and evaluation results
This paper describes implementation and outcomes for QIS in school-based summer learning programs in multiple cities.
Download Full Report
Quality-outcomes study for Seattle Public Schools summer programs, 2016 program cycle
This paper compares outcomes for struggling students in high quality instructional settings to outcomes for similar students in low quality instructional settings.
Download Full Report

Policy and Evidence

This project includes papers discuss the challenges for average-effects impact designs social-address covariates to detect effects in afterschool programs. Performance measurement designs are offered as cost-effective alternative source of evidence.

Framing an evidence based decision about 21st CCLC: How do we see the value?
Download Publication
Continuous Improvement Theory and Methodology for Lower Stakes Accountability Models
Request Report

Observational Assessment Methodology

This project includes papers on validation of observational measures of instruction in afterschool, child care, and youth programs.

Full findings from the Youth PQA validation study
Download Publication
Findings from the PQA self-assessment pilot in Michigan 21st century learning centers
Download Publication
Development and early validation evidence for an observational measure of high quality instructional practice for science, technology, engineering and mathematics in out-of-school time settings: The STEM supplement to the Youth Program Quality Assessment
Request Report