Publication Type: Report

Summary | Download

Preparing Youth to Thrive: Methodology and Findings from the SEL Challenge

The Social and Emotional Learning (SEL) Challenge was undertaken in pursuit of two ambitious goals: To identify promising practices for building SEL skills with vulnerable adolescents, and to develop technical supports for use of these SEL practices at scale in thousands of out-of-school time (OST) settings. The study design included a qualitative methodology, expert practitioners, and performance studies at each of eight exemplary programs. The products of the Challenge—standards for SEL practice and the suite of SEL performance measures—is designed to help OST programs focus deeply on SEL practice, assess their strengths, and improve the quality and effectiveness of their services using a continuous improvement approach.

By focusing systematically at a granular level of adult and youth behavior, the Challenge content supports use in practice-oriented settings and systems—youth programs, school day classrooms, mentorships, residential treatment, apprenticeships, workplace, families—where the qualities of adult-youth interaction and learning are a primary concern. We hope that local policy makers and funders will use the Challenge as a template for identifying the exemplary SEL services already available in their communities and make sure that they are adequately recognized, resourced, and replicated.

The promising practices [were] featured in a Field Guide, Preparing Youth to Thrive: Promising Practices for Social and Emotional Learning (Smith, McGovern, et al., 2016), a companion website, and a suite of tools and technical assistance. This report, Preparing Youth to Thrive: Methodology and Findings from the SEL Challenge, describes how the partnership carried out the work of the Challenge and what we learned as a result. Findings from the SEL Challenge include:

  1. The Challenge methodology successfully identified exemplary SEL offerings and produced 34 standards, 78 practice indicators, and 327 vignettes for building SEL skills with vulnerable youth.
  2. The suite of performance measures developed for the Challenge is feasible to implement and demonstrates sufficient reliability and validity for both continuous improvement and evaluation uses.
  3. The performance studies indicate that the exemplary offerings were exceptionally high quality compared to other OST programs and that youth skills improved in all six SEL domains. Skill growth also occurred for the higher risk groups. Benchmarks for SEL performance include: (a) Diverse staff and youth, intensive participation, and expert adult guidance; (b) Collaborative organizational cultures; (c) Exceptionally high quality instruction and youth engagement; (d) A consistent pattern of positive SEL skill growth across measures, offerings, and risk status.
  4. The offerings shared an OST-SEL intervention design: project-based learning with intensive co-regulation.

The Discussion section addresses generalizability of findings, cautions about SEL measurement, and study limitations.

Preparing Youth to Thrive: Promising Practices for Social Emotional Learning

Executive Summary

The Social and Emotional Learning (SEL) Challenge was designed to identify promising practices for building skills in six areas: emotion management, empathy, teamwork, initiative, responsibility, and problem-solving. The Challenge was a partnership between expert practitioners (youth workers, social workers, teachers) delivering exemplary programs in eight unique communities, a team of researchers, and a national foundation.

Although each of the exemplary out-of-school-time (OST) programs that were studied uses a different curriculum, their approaches to building social and emotional skills have important similarities, and these are the subject of the guide. This guide presents 32 standards and 58 indicators of SEL practice in six domains as well as four curriculum features that were shown to be foundational for supporting SEL practices.

For teens, social and emotional learning helps build resiliency and a sense of agency—skills critical for navigating toward positive futures of their own design. Social and emotional skills are the skills for action that help youth on that path. These skills go by several names: 21st-century skills, soft skills, and character education, and are experiential learning, positive youth development, etc. We focused on translating the “action” that staff and youth see in exemplary out-of-school-time programs into plain language. The guide sets things to share widely and in plain language how professionals can embed practices that support social and emotional learning with greater intentionality.

This guide is designed to start conversations about the kinds of social and emotional skills readers hope will grow in the adolescents they know and care about and to support the adult practices that help these skills to grow. We hope that readers will use the guide to create and pursue their own action plans for implementing SEL in their OST programs and networks. The guide is designed for readers to use on their own terms, not as a book to be read front-to-back—advice to readers is provided at the end of the introduction.

Moving the Needle on “Moving the Needle”

Summary

This paper introduces the nomenclature of performance-based accountability systems (PBAS) to the expanded learning field, provides a policy case study for a countywide system in southern Florida and uses data from that system to explore the issue of quality thresholds. We present an expanded design standard to guide development and improvement of PBAS policies and further develop a theory of lower-stakes accountability to guide effective use of incentives of various types. Findings suggest that (1) the PBAS framework defines critical concepts and improves our ability to describe existing quality improvement systems, (2) the Youth Program Quality Assessment (Youth PQA) can be used to produce a program rating of sufficient reliability for use in a PBAS, and (3) that the Palm Beach County PBAS
design is an exemplar for expanded learning policies.

General recommendations for PBAS designs include:

  • PBAS design should differentiate roles and link performance measures to incentives targeted at specific management and service delivery roles.
  • PBAS designs should include program ratings for multiple service domains linked to a mix of higher- and lower-stakes incentives.
  • PBAS should emphasize participants’ understanding of performance levels and sense of fairness while evolving toward higher-stakes incentives over time.

Detailed recommendations for Weikart Center clients using the Youth Program Quality Intervention and related Program Quality Assessments as the basis for an expanded learning PBAS design include:

  • Recommendations for best practice in each element of the seven elements in PBAS design
    standard.
  • Detailed description of a composition map for program ratings and performance levels for nine commonly used measures in expanded learning PBAS.
  • A PBAS design exemplar based on the Palm Beach County case describing specific combinations four types of incentives (financial, customer review, supervisory review, access to data) with two types of performance levels (high and low) and nine program ratings to achieve an optimal, lower-stakes, PBAS design with higher-stakes elements.

Measuring Youth Skills in Expanded Learning Systems: Case Study for Reliability and Validity of YDEKC Skill Measures and Technical Guidance for Local Evaluators

Weikart Center Expanded Learning Initiative, Technical Working Paper #4 for the Ready by 21 Project at the Forum for Youth Investment

Summary

  • YDEKC has made great progress toward development of skill measures for expanded learning service providers that serve multiple purposes of community positioning, performance improvement, and proof of program effectiveness. Already YDEKC’s efforts have advanced the field toward the most important questions: What are the important skills of interest for the expanded learning field? How do expanded learning settings cause change in these skills?
  • The current set of YDEKC measures (Table 1) are valuable for positioning in relation to community goals because they state the intentions of YDEKC providers. These measures utilize scales that are reliable (defined as internal consistency) but have weak evidence for construct validity because many of the scales and items are highly correlated.
  • An improved set of skill measures (Table 8) can be extracted from the YDEKC skill measures with more sufficient evidence of reliability, construct validity, and additional evidence for convergent validity. This structure was replicated in important subgroups in the YDEKC sample, including middle school youth, high school youth, and at-risk youth.
  • Additional evidence for convergent validity includes:
    • External measures of program quality are positively associated with youth reports of the program fit for skill building.
    • Youth reports of the program fit for skill building are positively associated with most of the other youth skill measures.
    • Measures related to managing academic work are positively associated with youth reports on school success measures, including grades and attendance in the past month.
  • YDEKC data can be used to create multi-variate skill profiles that better reflect the integrated nature of skill learning and demonstration. These profiles indicate that a subgroup of youth in lower skill profiles can be identified and that these youth are spread across nearly all YDEKC partner organizations.
  • Due to within-program heterogeneity of skills, program averages should not be used. However, all measures considered here have substantial negative skew or ceiling effects, which limits the usefulness of these measures for multiple time point designs.
  • We recommend a three-step method that addresses the integrated nature of skill learning as well as the use of youth skill measures that have lower construct validity and ceiling effects: (a) identify dimensionality in the data to best reflect the independent components of an individual’s integrated skill set, (b) use pattern-centered methods to identify independent profiles or subgroups of individuals defined by similar skill sets, and (c) collect the data at multiple time points for youth in the lower skill profiles at baseline.
  • We carried out a similar set of analyses using data from the YDEKC school survey, finding substantial positive evidence for the reliability and construct validity of these measures (see Appendix D).

Building Citywide Systems for Quality: A Guide and Case Studies for Afterschool Leaders

Summary

High-quality programming is essential in order for afterschool efforts to generate positive effects for youth. This guide can help those working to create better, more coordinated afterschool programming get started building a quality improvement system (QIS), or further develop existing efforts. A quality improvement system is an intentional effort to raise the quality of afterschool programming in an ongoing, organized fashion.

Components of an Effective Quality Improvement System 

Shared definition of quality: There should be general agreement on what constitutes a high-quality program.

Lead organization: Not having a clear leader can cause confusion. Lead organizations can be stand-alone technical assistance organizations, intermediaries, city agencies, funding entities, or policy/advocacy organizations.

Engaged stakeholders: a QIS is more likely to be effective, sustainable, and scalable if a defined group of organizations is supportive.

Continuous improvement model: Effective models typically include a standard for high-quality performance, an assessment tool, and aligned improvement supports such as planning, coaching, and training.

Information system(s): Quality improvement systems generate data. These data can include administrative audits, tracking of QIS participation and engagement, or data about program attendance and/or child outcomes. The QIS should capture and store such information.

Guidelines and incentives for participation: Guidelines brings coherence and relevance to the system.

Adequate resources: Funding is necessary for sustainability. 

System-building Stages and Tasks

The guide describes a series of tasks organized into three broad stages.

  1. Plan and Engage. Specific tasks include assessing readiness, forming a work group, making the case, engaging stakeholders, identifying a lead organization, defining quality, clarifying purpose, considering information needs, and determining costs and potential resources.
  2. Design and Build. This step takes the process from the conceptual to the practical. Specific tasks include designing the continuous improvement model that programs will experience, developing system-level supports for the model, recruiting pilot sites, and piloting the continuous improvement cycle.
  3. Adjust and Sustain. Specific tasks include refining the continuous improvement model and system supports, building capacity of the lead organization, engaging new programs and sectors, evaluating, and embedding and sustaining the system.

In addition to a QIS Capacity Self-Assessment Tool, the guide also includes case studies of efforts by six communities to build quality improvement systems.

Key Takeaways

  • Developing a quality improvement system for afterschool and youth development programs in a community is important, complex work.
  • Although quality improvement systems vary, mature, effective systems share some common components and characteristics. These include a shared definition of quality, engaged stakeholders, and adequate resources.
  • There are three stages for developing a quality improvement system: Plan and engage, design and build, and adjust and sustain. 

The STEM supplement to the Youth Program Quality Assessment

Introduction

curricula from the fields of environmental science, technology, engineering, and mathematics (“STEM”) at 10 sites(one partner delivered the same curriculum at two sites). The offerings were organized at 10 school-based Afterzone sites and each offering included field work in the local Providence region. Across the 10 sites, STEM curricula were delivered to a total of approximately 250 middle school students (about 25 students per Afterzone section).

In order to evaluate the Afterzone Summer Scholars model and collect information for future improvement, PASA (a) hired an external evaluator for the project; (b) committed to providing continuous improvement supports to participating program managers and content providers (quality assessment and coaching); and (c) formed an evaluation advisory board to monitor the development and implementation of the external evaluation. In addition, PASA contracted with the David P. Weikart Center for Youth Program Quality (Weikart Center) at the Forum for Youth Investment to develop an observation-based measure of instructional practices to support continuous improvement during STEM programming. This report describes the process of development of the STEM supplement to the Youth Program Quality Assessment (Youth PQA; HighScope, 2005) and preliminary reliability and validity evidence based on data collected during Afterzone Summer Scholars program.

Continuous Quality Improvement in Afterschool Settings: Impact Findings from the Youth Program Quality Intervention Study

Abstract

Background: Out-of-school time programs can have positive effects on young people’s development; however, programs do not always produce such effects. The quality of instructional practices is logically a key factor but quality improvement interventions must be understood within a multi-level framework including policy, organization, and point of service if they are to be both effective and scalable.

Purpose: To evaluate the effectiveness of the Youth Program Quality Intervention (YPQI), a data-driven continuous improvement model for afterschool systems. Research questions include:

  • Does the YPQI increase managers’ focus on instruction and the use of continuous improvement practices by site-based teams?
  • Does the YPQI improve the quality of afterschool instruction?
  • Does the YPQI increase staff tenure?
  • Can the YPQI be taken to scale across programs that vary widely in terms of structure, purposes, and funding and using resources available to public agencies and community-based organizations?
  • Will afterschool organizations implement the YPQI under lower stakes conditions where compliance with the model is focused on the improvement process rather than attainment of pre-determined quality ratings?

Participants: Eighty-seven afterschool sites in five diverse afterschool networks participated in the study. Each site employed the equivalent of one full-time program manager and between two and ten direct staff; had an average annual enrollment of 216 youth; and had an average daily attendance of 87 youth.

Research Design: This is a cluster randomized trial. Within each of the five networks, between 17 and 21 sites were randomly assigned to an intervention (N=43) or control group (N=44). Survey data were collected from managers, staff, and youth in all sites at baseline prior to randomization (spring 2006), at the end of the implementation year of the study (spring 2007) and again at the end of the follow-up year (spring 2008). External observers rated instructional practices at baseline and at the end of the implementation year. Implementation data were collected from both intervention and control groups. Hierarchical linear models were used to produce impact estimates.

Findings: The impacts of the YPQI on the central outcome variables were positive and statistically significant. The YPQI produced gains in continuous improvement practices with effect sizes of .98 for managers and .52 for staff. The YPQI improved the quality of staff instructional practices, with an effect size of .55. Higher implementation of continuous improvement practices was associated with higher levels of instructional quality, with effects nearly three times greater than the overall experimental impact. Level of implementation was sustained in intervention group sites in the follow-up year.

Conclusions: This study demonstrates that a sequence of continuous improvement practices implemented by a site-based team – standardized assessment of instruction, planning for improvement, coaching from a site manager, and training for specific instructional methods – improves the quality of instruction available to children and youth. The YPQI produces a cascade of positive effects beginning with the provision of standards, training, and technical assistance, flowing through managers and staff implementation of continuous improvement practices, and resulting in effects on staff instructional practices. Evidence also suggests that participation in the YPQI may increase the length of staff tenure and that YPQI impacts are both sustainable and scalable.

Palm Beach Quality Improvement System Pilot: Final Report

This report on the Palm Beach County Quality Improvement System (QIS) pilot provides evaluative findings from a four-year effort to imagine and implement a powerful quality accountability and improvement policy in a countywide network of after-school programs. The Palm Beach QIS is an assessment-driven, multi-level intervention designed to raise quality in after-school programs, and thereby raise the level of access to key developmental and learning experiences for the youth who attend. At its core, the QIS asks providers to identify and address strengths and areas for improvement based on use of the Palm Beach County Program Quality Assessment (PBC-PQA)—a diagnostic and prescriptive quality assessment tool – and then to develop and enact quality improvement plans. Throughout this process training and technical assistance are provided by several local and national intermediary organizations.

We present baseline and post-pilot quality ratings for 38 after-school programs that volunteered to participate in the Palm Beach QIS pilot over a two-year cycle. This data is the routine output from the QIS system and is designed to support evaluative decisions by program staff and regional decision-makers. In addition to the typical QIS output, we also provide as much detail as possible about the depth of participation in the various elements of the improvement initiative and offer a few opinions about what worked.

Primary findings include:

  • Quality changed at both the point of service and management levels. During the QIS quality scores changed substantially at both the point of service and management levels, suggesting that the delivery of key developmental and learning experiences to children and youth increased between baseline and post-pilot rounds of data collection.
    • Point-of-service quality increased most substantially in areas related to environmental supports for learning and peer interaction, but positive and statistically significant gains were evidenced in all assessed domains of quality.
    • The incidence of organizational best practices and policies increased in all assessed management-level domains, especially staff expectations, family connections and organizational logistics.
  • Planning strategies that targeted specific improvement areas were effective. Pilot sites registered larger quality gains on point of service metrics that were aligned with intentionally selected areas for improvement. This indicates that the quality improvement planning process effectively channels improvement energies.
  • Site managers and front line staff participated in core elements of the QIS at high rates. Relative to other samples, participation by front line staff was especially high, suggesting that the core tools and practices of the QIS are reasonably easy for site managers to introduce into their organizations.
  • The core tools and practices of the QIS were adopted at high rates. Thirty-five of 38 sites (92%) completed the self-assessment process and 28 sites (74%) completed all of the steps necessary to submit a quality improvement plan.

Several secondary questions posed by stakeholders or relevant to policy were also explored. These secondary findings must be treated with caution since they are drawn from a small sample and, in some cases, less than perfect data sources. Secondary findings include:

  • The low stakes approach to accountability within the QIS model appears to have increased provider buy in. Through review of secondary documents and quantitative data, the QIS emphasis on partnership rather than external evaluation achieved buy-in from pilot group providers for the self-assessment and improvement planning process.
  • The self-assessment and improvement planning sequence was associated with change in quality scores. Programs that participated in the self-assessment process were more likely than those that did not to experience improvement in their quality scores.
  • Structural characteristics such as organization type, licensing status, supervisor education and experience levels were not strongly related to point-of-service quality. This suggests that the variables most often manipulated by reform initiatives are, at best, weak drivers of setting quality and thus less-than-ideal policy targets. Put another way, these several program “credentials”, while reasonably easy to measure, were poor proxies for quality.

Findings from the Self-Assessment Pilot in Michigan 21st Century Learning Centers

Overall 24 sites within 17 grantees participated in the self-assessment pilot study by assembling staff teams to collect data and score the Youth Program Quality Assessment (PQA).

At each site an average of 5 staff spent an average of 13 staff hours to complete the self-assessment process.

Whether using an absolute standard or group norms as a benchmark for interpretation of data from the Youth PQA Self-Assessment Pilot Study (hereafter called the Pilot Study), quality scores were very positive for participating programs and also reflected the tendency of self-assessment scores to be biased toward higher quality levels.

The quality scores followed the same pattern as outside observer scores in other samples, highest on for issues of safety and staff support and lowest on higher order practices focused on interaction and engagement.

Youth PQA data collected using the self-assessment method demonstrated promising patterns of both internal consistency and concurrent validity with aligned youth survey responses.

Two thirds or more of sites reported that the observation and scoring process helped the self-assessment team to have greater insight into the operation of their programs, talk in greater depth about the program quality than usual, and have more concrete understanding of program quality.

Site directors and local evaluators said that the self-assessment process was a source of good conversations about program priorities and how to meet them. In almost all cases, concrete action followed from the self-assessment process.

Site directors and local evaluators demonstrated the ability to improvise the self-assessment method to fit local needs.

Program directors, site coordinators, and local evaluators have used the Youth PQA and statewide Youth PQA data to generate statewide program change models, suggesting that the instrument and data are useful for setting system-level improvement priorities.

Original Validation of the Youth Program Quality Assessment (Youth PQA)

Summary

The Youth Program Quality Assessment (PQA) is an assessment of best practices in afterschool programs, community organizations, schools, summer programs, and other places where you have fun, work, and learn with adults. The Youth PQA creates understanding and accountability focused on the point of service — where youth and adults come together to coproduce developmental experiences. The ultimate purposes of the Youth PQA are empowering staff to envision optimal programming and building motivation of youth to participate and engage. As an approach to assessment at the systems level, the Youth PQA links accountability to equity by focusing on access to high-quality learning environments for all youth who enroll. As a research tool, the Youth PQA improves measurement of instructional process in places where young people learn.

The Youth PQA consists of seven sections or subscales, each bearing on one dimension of program quality critical for positive youth development: safe environment, supportive environment, interaction, engagement, youth-centered policies and practices, high expectations, and access. Administration of the Youth PQA employs direct observation of youth program activities for its first four sections and a structured interview with a program director for its remaining three sections. The instrument can be used by outside observers to produce the most precise data or as a program self-assessment directed toward generation of rich conversations among staff.

The Youth PQA Validation Study was a 4-year effort to develop and validate a tool to assess program quality in youth settings. Through the process of instrument development, dozens of expert practitioners and researchers were brought together to provide input on the tool. In total, the validation study encompassed 59 organizations in Michigan and more than 300 Youth PQA observations and interviews conducted in programs serving 1,635 youth. Most of these youth programs were afterschool programs that met weekly or daily over several months. The average age of youth in the sample was 14 years, and more than half were attending programs in an urban context.

The Youth PQA Validation Study employed multiple, independent data sources, including interviews with program administrators, observations in youth work settings, surveys of program youth, expert opinions, and verified reports of staff training. The study’s primary concurrent measure of program quality was the Youth Survey from Youth Development Strategies, Inc. All Youth Survey data were independently collected and prepared for analysis by Youth Development Strategies, Inc.

In general, findings from the study demonstrate that the Youth PQA is a valid, reliable, and highly usable measure of youth program quality. Principle findings include:

  1. The Youth PQA measurement rubrics are well calibrated for use in a wide range of youth serving organizations. Average scores fall near the center of the scale and are spread across all five scale points.
  2. Pairs of data collectors were able to achieve acceptable levels of inter-rater reliability on most of the Youth PQA’s measurement constructs.
  3. The Youth PQA scales subscales are reliable measures of several dimensions of quality. Key subscales demonstrated acceptable levels of internal consistency in two samples.
  4. The Youth PQA can be used to assess specific components of programs and is not just a single global quality rating. In repeated factor analyses on two waves of Youth PQA data, the subscales were validated as separate, distinguishable constructs.
  5. Youth PQA quality ratings reflect youth reports about their own experiences in the same program offerings. Youth PQA scores demonstrate concurrent validity through significant positive correlation with aligned scores from the YDSI Youth Survey.
  6. The Youth PQA measures dimensions of quality that are related to positive outcomes for youth such as youth sense of challenge and growth from the youth program. Youth PQA scores demonstrate predictive validity in multivariate and multilevel models of the data, controlling for youth background variables.
  7. Staff in 21st Century afterschool programs find the instrument to have face validity, to be applicable to their current work, and to be a foundation for purposeful change in their programs.