Explore the reports, books, and other publications produced by the QTurn team and provides a foundation of our work and passion.
Theory and Evidence for Quality Improvement Systems and Lower-Stakes Accountability in the Out-of-School Time Sector
This decade-long project was focused on scaling an education policy innovation called quality improvement systems (QIS). QIS are a type of performance-based accountability system (PBAS) that is guided by a lower-stakes design for QIS participant experience. Lower-stakes designs optimize staff expertise and performance using positive-reinforcing informational incentives that are a good with collaborative and data-driven organizational cultures in the education fields. Place-based QIS designs have been successful and sustained in conditions of public sector outsourcing, mixed public-private networks, and performance measurement that characterize “new public management.”
Moving the needle on moving the needle: Next stage technical guidance for performance based accountability systems in the expanded learning field with a focus on performance levels for the quality of instructional services
December 1, 2013
This paper introduces the nomenclature of performance-based accountability systems (PBAS) to the expanded learning field, provides a policy case study for a countywide system in southern Florida and uses data from that system to explore the issue of quality thresholds. We present an expanded design standard to guide development and improvement of PBAS policies and further develop a theory of lower-stakes accountability to guide effective use of incentives of various types. Findings suggest that (1) the PBAS framework defines critical concepts and improves our ability to describe existing quality improvement systems, (2) the Youth Program Quality Assessment (Youth PQA) can be used to produce a program rating of sufficient reliability for use in a PBAS, and (3) that the Palm Beach County PBAS design is an exemplar for expanded learning policies.
Citation: Smith, C., Akiva, T., McGovern, G. and Peck, S.C. (2014), Afterschool quality. New Directions for Youth Development, 2014: 31-44. https://doi.org/10.1002/yd.20111
Building citywide systems for quality: Guide and case studies for afterschool leaders
January 12, 2012
How-to guide for cities drawing on experience from implementation of QIS in 40+ cities and states.
Citation: Yohalem, N., Devaney, E., Smith, C., & Wilson-Ahlstrom, A. (2012). Building Citywide Systems for Quality: A Guide and Case Studies for Afterschool Leaders.
Continuous quality improvement in afterschool settings: Impact findings from the Youth Program Quality Intervention study
January 1, 2012
Multi-site randomized controlled trial that identifies a substantively large and statistically significant cross-level cascade of QIS effects from network to organization to point-of-service instruction.
Citation: Akiva, T., Sugar, S.A., Smith, C., Pearson, L.M., Peck, S.C., Denault, A., & Blazevski, J. (2012). Continuous Quality Improvement in Afterschool Settings: Impact findings from the Youth Program Quality Intervention Study.
Understanding the “how” of effective quality improvement: Lessons from the Rhode Island Program Quality Intervention
January 1, 2012
Case study of scaled QIS implementation in Rhode Island with focus on manager skills and implementation at the organization level.
Citation: Devaney, E., Smith, C., & Wong, K.K. (2012). Understanding the “How” of Quality Improvement: Lessons from the Rhode Island Program Quality Intervention. Afterschool Matters.
Final report on the Palm Beach quality improvement system pilot: Model implementation and program quality improvement in 38 after-school programs
November 17, 2008
Mixed methods study of QIS implementation and effects in afterschool programs in Palm Beach County, Fl.
Citation: Smith, C., Akiva, T., Blazevski, J., & Pelle, L. (2008). Final report on the Palm Beach Quality Improvement System pilot: Model implementation and program quality improvement in 38 after-school programs. Forum for Youth Investment, David P, Weikart Center for Youth Program Quality.
Quality accountability: Improving fidelity of broad developmentally focused interventions
January 1, 2008
Design theory and formative evidence to define (1) the system/network-level within which organization and point-of-service settings are materially nested and (2) the lower-stakes QIS design that networks and organizations should implement for improved performance.
Citation: Smith, C.M., & Akiva, T. (2008). Quality Accountability: Improving Fidelity of Broad Developmentally Focused Interventions.
Quality in the out-of-school time sector: Insights from the Youth PQA validation study
January 1, 2006
Design theory and formative evidence to describe setting dynamics at two distinct levels, (1) the organization-level where professional learning communities are formed and (2) point-of-service level (e.g., classrooms) where professional staff and learners meet.
Pattern-Centered Methodology for Cross-level Effects and Within-Level Interactions
This project explored the potential of pattern-centered methodology to provide a cost-effective approach to addressing s two key challenges for performance data produced through QIS: (1) Effects are produced through cross-level cascade from network to organization to point-of-service to individual learner (e.g., unit of analyses problems for linear models) and (2) point of service level environments (e.g., classrooms) entail a complex integration of higher order interactions (e.g., Cronbach’s hall of mirrors problem of unidentified subgroups). These papers identify a key performance indicator – membership in the low-quality profile for quality of instruction – and then use pattern-centered methods to link low quality instruction to both manager practices and child/youth engagement with instruction.
Measuring youth skills in expanded learning systems: Case study for reliability and validity of the YDEKC skill measures and technical guidance for local evaluators
January 2, 2013
This paper uses pattern-centered methods to increase the usefulness of information available from survey-based skill measures, with a focus on using survey-based skills measures to detect skill change over time.
Managing for positive youth development: Linking profiles of after-school program management to profiles of staff instructional performance
November 17, 2012
This paper uses pattern centered methods to describe the association between four collaborative management practices and the quality of instruction available to youth.
Quality at the point of service: profiles of practice in after-school settings
April 24, 2010
This paper uses pattern centered methods to identify three different quality profiles – high, medium, and low – that characterize staff instruction in out-of-school time programs.
Citation: Smith, C., Peck, S.C., Denault, A., Blazevski, J. and Akiva, T. (2010), Quality at the Point of Service: Profiles of Practice in After-School Settings. American Journal of Community Psychology, 45: 358-369. https://doi.org/10.1007/s10464-010-9315-z
Linking after-school instructional practices to youth engagement: A pattern-centered approach
January 1, 2010
This paper uses pattern centered methods to describe the association between the quality of instruction available and youth’s level of mental engagement with that instruction.
Citation: Akiva, T., Pearson, L., Sugar, S. A., Peck, S. C., Smith, C., & Denault, A. (2010). Linking after-school instructional practices to youth engagement: A pattern-centered approach. Paper presented at the Society for Research on Adolescence Biennial Meeting.
SEL Performance Standards, Measures, and Benchmarks
This project used a mixed methods approach to identify best practice standards, develop a suite of performance measures, and establish performance benchmarks – for teacher practices and youth skills in the area of social and emotional learning (SEL).
Evaluation of Program Quality and Social and Emotional Learning in American Youth Circus Organization Social Circus Programs
January 12, 2018
This report exemplifies use of the suite of SEL measures and benchmarks in the American Youth Circus Organizations network.
Citation: Smith, C., Roy, L., Peck, S., & Macleod, C. (2017). Evaluation of program quality and social and emotional learning in American Youth Circus Organization social circus programs. Retrieved January 2018 from American Youth Circus Organization website: http://www.americancircuseducators.org/sel/.
Preparing youth to thrive: Methodology and findings from the social and emotional learning challenge
August 20, 2016
This technical report describes methodology and findings for (1) best-practice SEL standards, (2) validation of a suite of SEL performance measures for use in QIS, and (3) performance benchmarks for out-of-school time programs focused on building SEL skills with vulnerable children/youth.
Citation: Smith, C., McGovern, G., Peck., S.C., Larson, R., Hillaker, B., Roy, L. (2016). Preparing Youth to Thrive: Methodology and Findings from the Social and Emotional Learning Challenge. Forum for Youth Investment, Washington, D.C.
Afterschool quality systems
December 23, 2014
This paper describes areas of broad consensus in the out-of-school time field, including the growing focus on SEL skills.
Citation: Smith, C., Akiva, T., McGovern, G. and Peck, S.C. (2014), Afterschool quality. New Directions for Youth Development, 2014: 31-44. https://doi.org/10.1002/yd.20111
Preparing youth to thrive: Promising practices in social and emotional learning
January 24, 2012
The guidebook describes SEL standards and organizational and curriculum features for a set of eight exemplary SEL programs.
Citation: Charles Smith et al., Preparing Youth to Thrive: Promising Practices for Social and Emotional Learning (Washington, DC: Forum for Youth Investment, 2016).
Evaluation Exemplars from QIS Data
This project includes papers from school-based QIS - 21st CCLC systems in Michigan and Oklahoma; school district-based summer learning programs Seattle, Denver, and St. Paul - to demonstrate the types of performance evaluations that are possible using QIS data.
Design study for the Summer Learning Program Quality Intervention (SLPQI): Final-year intervention design and evaluation results
January 1, 2017
This paper describes implementation and outcomes for QIS in school-based summer learning programs in multiple cities.
Oklahoma afterschool improvement process: Evaluation of 5 year trajectories
January 1, 2017
This paper describes validity of QIS performance measures and longitudinal change over four years in Oklahoma 21st CCLC programs.
Quality-outcomes study for Seattle Public Schools summer programs, 2016 program cycle
January 1, 2016
This paper compares outcomes for struggling students in high quality instructional settings to outcomes for similar students in low quality instructional settings.
Policy and Evidence
This project includes papers discuss the challenges for average-effects impact designs social-address covariates to detect effects in afterschool programs. Performance measurement designs are offered as cost-effective alternative source of evidence.
Framing an evidence based decision about 21st CCLC: How do we see the value?
March 27, 2015
In this policy commentary, we do some reasoning about how 21st CCLC produces value and discuss the limitations in one particular way of seeing that value – the intent-to-treat impact evaluation design.
Observational Assessment Methodology
This project includes papers on validation of observational measures of instruction in afterschool, child care, and youth programs.
Development and early validation evidence for an observational measure of high quality instructional practice for science, technology, engineering and mathematics in out-of-school time settings: The STEM supplement to the Youth Program Quality Assessment
November 24, 2012
This report describes the process of development of the STEM supplement to the Youth Program Quality Assessment (Youth PQA; HighScope, 2005) and preliminary reliability and validity evidence based on data collected during Afterzone Summer Scholars program.
Citation: Smith, C., Hallman, S., Hillaker, B., Sugar, S., McGovern, G., & Devaney, E. (2012). Development and early validation evidence for an observational measure of high quality instructional practice for science, technology, engineering and mathematics in out-of-school time settings: The stem supplement to the youth program quality assessment (pp. 1–25). Ypsilanti, MI: The David P. Weikart Center for Youth Program Quality.
Findings from the PQA self-assessment pilot in Michigan 21st century learning centers
September 11, 2005
Overall 24 sites within 17 grantees participated in the self-assessment pilot study by assembling staff teams to collect data and score the Youth Program Quality Assessment (PQA). Youth PQA data collected using the self-assessment method demonstrated promising patterns of both internal consistency and concurrent validity with aligned youth survey responses.
Citation: Smith, C. (2005b). Findings from the self-assessment pilot in Michigan 21st Century Learning Centers. Ypsilanti, MI: High/Scope Educational Research Foundation.
Full findings from the Youth PQA validation study
August 15, 2005
Original validation study for the Youth Program Quality Assessment (Youth PQA). The Youth PQA was developed to assess the quality of instruction in afterschool and other education settings.
Citation: Smith, C., & Hohmann, C. (2005). Full findings from the Youth PQA validation study. Ypsilanti, MI: High/Scope Educational Research Foundation.