Publication Year: 2013

Summary | Download

Moving the Needle on “Moving the Needle”

Summary

This paper introduces the nomenclature of performance-based accountability systems (PBAS) to the expanded learning field, provides a policy case study for a countywide system in southern Florida and uses data from that system to explore the issue of quality thresholds. We present an expanded design standard to guide development and improvement of PBAS policies and further develop a theory of lower-stakes accountability to guide effective use of incentives of various types. Findings suggest that (1) the PBAS framework defines critical concepts and improves our ability to describe existing quality improvement systems, (2) the Youth Program Quality Assessment (Youth PQA) can be used to produce a program rating of sufficient reliability for use in a PBAS, and (3) that the Palm Beach County PBAS
design is an exemplar for expanded learning policies.

General recommendations for PBAS designs include:

  • PBAS design should differentiate roles and link performance measures to incentives targeted at specific management and service delivery roles.
  • PBAS designs should include program ratings for multiple service domains linked to a mix of higher- and lower-stakes incentives.
  • PBAS should emphasize participants’ understanding of performance levels and sense of fairness while evolving toward higher-stakes incentives over time.

Detailed recommendations for Weikart Center clients using the Youth Program Quality Intervention and related Program Quality Assessments as the basis for an expanded learning PBAS design include:

  • Recommendations for best practice in each element of the seven elements in PBAS design
    standard.
  • Detailed description of a composition map for program ratings and performance levels for nine commonly used measures in expanded learning PBAS.
  • A PBAS design exemplar based on the Palm Beach County case describing specific combinations four types of incentives (financial, customer review, supervisory review, access to data) with two types of performance levels (high and low) and nine program ratings to achieve an optimal, lower-stakes, PBAS design with higher-stakes elements.

Measuring Youth Skills in Expanded Learning Systems: Case Study for Reliability and Validity of YDEKC Skill Measures and Technical Guidance for Local Evaluators

Weikart Center Expanded Learning Initiative, Technical Working Paper #4 for the Ready by 21 Project at the Forum for Youth Investment

Summary

  • YDEKC has made great progress toward development of skill measures for expanded learning service providers that serve multiple purposes of community positioning, performance improvement, and proof of program effectiveness. Already YDEKC’s efforts have advanced the field toward the most important questions: What are the important skills of interest for the expanded learning field? How do expanded learning settings cause change in these skills?
  • The current set of YDEKC measures (Table 1) are valuable for positioning in relation to community goals because they state the intentions of YDEKC providers. These measures utilize scales that are reliable (defined as internal consistency) but have weak evidence for construct validity because many of the scales and items are highly correlated.
  • An improved set of skill measures (Table 8) can be extracted from the YDEKC skill measures with more sufficient evidence of reliability, construct validity, and additional evidence for convergent validity. This structure was replicated in important subgroups in the YDEKC sample, including middle school youth, high school youth, and at-risk youth.
  • Additional evidence for convergent validity includes:
    • External measures of program quality are positively associated with youth reports of the program fit for skill building.
    • Youth reports of the program fit for skill building are positively associated with most of the other youth skill measures.
    • Measures related to managing academic work are positively associated with youth reports on school success measures, including grades and attendance in the past month.
  • YDEKC data can be used to create multi-variate skill profiles that better reflect the integrated nature of skill learning and demonstration. These profiles indicate that a subgroup of youth in lower skill profiles can be identified and that these youth are spread across nearly all YDEKC partner organizations.
  • Due to within-program heterogeneity of skills, program averages should not be used. However, all measures considered here have substantial negative skew or ceiling effects, which limits the usefulness of these measures for multiple time point designs.
  • We recommend a three-step method that addresses the integrated nature of skill learning as well as the use of youth skill measures that have lower construct validity and ceiling effects: (a) identify dimensionality in the data to best reflect the independent components of an individual’s integrated skill set, (b) use pattern-centered methods to identify independent profiles or subgroups of individuals defined by similar skill sets, and (c) collect the data at multiple time points for youth in the lower skill profiles at baseline.
  • We carried out a similar set of analyses using data from the YDEKC school survey, finding substantial positive evidence for the reliability and construct validity of these measures (see Appendix D).