Category: Publication

Impact Evaluation for the Parent Child Plus Program

This Executive Summary was developed to facilitate discussion and decision making. The detailed findings and method are available in the full report, Impact Evaluation for the Parent Child Plus Program, Newark Trust for Education (2020), and the technical appendices available upon request.

The Newark Trust for Education (NTE) Parent Child Plus (PC+) program is an evidence-based early childhood education program for families in the city of Newark, NJ. NTE seeks to evaluate PC+ by conducting analyses of existing data for a cohort of 89 families, assessed 4 times over 46 weeks using observational measures of parenting practices and children’s socio-emotional skills.

The PC+ program is intended to result in “improved child behaviors related to social-emotional development and self-regulation skills” (Organizational Research Services [ORS], 2010, p. 23). Data generated as part of the PC+ program implementation include ratings by home visitors using the inhome observation assessment instruments, Child Behavior Traits (CBT) and Parent and Child Together (PACT), that “were specifically developed to evaluate the social-emotional status of low-income children age two to four and the positive parent-child interaction behaviors of low-income parent-child dyads, respectively”.

The home visit curriculum is delivered in the family home using four major strategies: (a) a strengths-based approach, (b) using parents/caregivers as teachers, (c) focusing on children’s learning and development outcomes, and (d) involving siblings and family members to reinforce the child’s learning experience. The PC+ home visits are made up of the following elements: A program schedule (VISM) for the home visit curriculum that thematically pairs books (in English, Spanish, or Portuguese) and toys together for a family to receive for a given program cycle. A manualized curriculum (VISM Guide) that explains key concepts, related to the books and toys received by the family, for the parent/caregiver to focus on after the home visit. A contract that obligates the parent/caregiver to maintain the predetermined home visit schedule and be present during the home visits. The standard home visit is a half-hour, during which the parent/caregiver and child practice specific positive interactions around the use of the books and toys provided by the PC+ program. A storage bin is provided to the family by PC+ for keeping books and toys organized.

Impact Evaluation for the Parent Child Plus Program

The Newark Trust for Education (NTE) Parent Child Plus (PC+) program is an evidence-based early childhood education program for families in the Newark, NJ. NTE seeks to evaluate performance by conducting analyses of existing data for a cohort of over 80 families, assessed four times over 46 weeks using observational measures of parenting practices and children’s socio-emotional skills. PC+ is intended to result in “improved child behaviors related to social‐emotional development and self‐regulation skills” (Organizational Research Services [ORS], 2010, p. 23).

The Quality-Impact-Equity Design and Methods (QDM) Toolbox (Smith, Peck, Roy, & Smith, 2019; Smith, Peck, & McNeil, 2020) was used to: (a) reconfigure existing measures for Parenting Practice Quality and Child SEL Skill to maximize reliability and validity for measuring socio-emotional skills and learning (SEL); (b) produce holistic profiles of parent and child skill (e.g., “whole child”) at each timepoint; and (c) apply pattern-centered analytics to estimate impact and equity effects of the PC+ program as implemented in Newark. Please note: We define impact in terms of the actual “in-the-world” structure of causes and effects, not in terms of counterfactuals. A brief description of the QDM methodology is provided in Appendix A (see also Smith et al., 2019)..

The PC+ program results reveal an overall impact pattern that suggests both a strong relation between parent and child skills and an effect of home visitors on both parent and child skills. Although, in almost all cases, the children of parents with high or growing parenting skills outperformed children with low or declining parenting skills, many children with parents in the low-skill profile for Parenting Practice Quality still experienced growth in SEL skills. This finding suggests that PC+ is working as it should, with parents and home visitors both having direct effects on child SEL skill growth. To fully demonstrate the impact of the NTE PC+ program given this “triadic” causal flow, we recommend (a) improving measures of PC+ fidelity and (b) including a small no-program sample of parents and children.

Contact QTurn to request Report Appendices A-I.

Impact Evaluation for the Palm Beach County Quality Improvement System 

Quality Improvement System (QIS) exposure moves afterschool programs to higher quality, increasing access to developmentally powerful settings and building children’s social and emotional learning skills. Higher quality is defined in terms of the quality of instruction (i.e., individuation, basic/advanced SEL, enrichment content), the stability of staff tenure, and evidence of children’s SEL skill growth.

In this study, we used performance data generated by Prime Time Inc. in Palm Beach County and fully pattern-centered methodology to describe the chain of causal effects as a cascade of sequential impacts. We sought to answer two specific questions about implementation and children’s SEL skill growth: What is the impact of QIS exposure on program quality (i.e., best practices, low staff turnover, great content), particularly for programs that have lower program quality at baseline? What is the impact of exposure to high program quality on student SEL skills?

Findings demonstrate that (1) QIS exposure causes program quality improvement to occur and (2) exposure to high quality corresponds to SEL skill growth. Specifically, (1.a) quality increased dramatically over three years of exposure to the Palm Beach County QIS; (1.b) programs with Low Quality at QIS entry improved when exposed to even moderate QIS Fidelity; (2.a.) children exposed to higher-quality programs had greater SEL skill maintenance and gains compared to children exposed to lower-quality programs; and (2.b) children with Low SEL Skill at entry made greater gains at all levels of program quality.

This pattern of findings suggests that the Prime Time QIS design is successfully building the quality of services available in the county in substantively meaningful ways – by increasing the quality of instruction, increasing the tenure of staff, and growing SEL skills for students who need it most.

Socio-Emotional Skills, Quality, and Equity: The Multilevel Person-in-Context ~neuroperson (MPCn) Framework

Evaluation evidence about the relations among children’s prior history, engagement in program settings, resulting SEL skill growth, and the ultimately desired transfer outcomes (e.g., agency to succeed in other settings) has been sporadic and fragmented. One reason for this may be that the positivist theory and methodology used by most researchers and evaluators is poorly suited to the formative explanations that guide continuous quality improvement (CQI) processes. As a result, we lack nuanced impact models that address questions about how and how much, or the information necessary for organizational decision-making. QTurn’s Quality-Outcomes Design and Methods (Q-ODM) toolbox (Peck & Smith, 2020) was created to address these fundamental problems in the evaluation of education settings, with a specific focus on out-of-school time (OST; afterschool, child care, drop-in, mentoring, tutoring, etc.) programs.

In this white paper, we introduce a theoretical framework designed to describe the integrated set of mental and behavioral parts and processes (i.e., schemas, beliefs, and awareness) that are socio-emotional skills and that produce both basic and advanced forms of agency. With improved definitions and understanding of SEL skills, and the causes of SEL skill growth, we hope to improve reasoning about programs and policies for socio-emotional supports in any setting where children spend time. Perhaps most importantly, we hope to inform policy decisions and advance applied developmental science by improving the accuracy and meaningfulness of basic data on children’s SEL skill growth.

Place-Based Ecological Stewardship

The Southeastern Michigan Stewardship Coalition (SEMIS) is a high impact and low-cost school reform for adults and students to build their repertoire of ecological stewardship skills – SEL, STEM and civic – and experience agency from the practice of ecological stewardship in their place. This white paper was developed to (1) describe how SEMIS promotes social and emotional learning, (2) iterate content and language with SEMIS stakeholders, (3) and to make a compelling and scientifically grounded case for expansion of the work.

Realist(ic) Evaluation Tools for OST Programs: The Quality-Outcomes Design and Methods Toolbox

Socio-emotional learning (SEL) skills are a partial but necessary cause of children’s developmental outcomes, and SEL skill growth is a key objective for nearly all out-of-school time (OST) programs. The Quality-Outcomes Design and Methods (Q-ODM) toolbox holds an integrated set of tools to measure and model children’s SEL skills, including how they change during, and in response to, OST programs (e.g., afterschool, school-age child care, workforce and career preparation, arts, sports). The Q-ODM toolbox helps organizational managers and evaluators to feasibly and cost-effectively adopt pattern-centered measures and models that produce actionable information for both continuous quality improvement (CQI) and impact evaluation.

The Q-ODM toolbox addresses practical questions about SEL skills and skill growth, such as: What is high-quality SEL support? How much SEL skill change does our program cause in each cycle? How much program quality does it take for stressed children to fully engage? Does our work create equity effects? The tools are divided into three groups: Design Tools, Analytic Tools, and Feedback Tools. These tools increase dramatically the value of CQI feedback for staff and the power of the analytic models used to evaluate program impact and equity effects for participating children. The Q-ODM toolbox was designed to empower internal and local evaluators to conduct rigorous and meaningful impact evaluations using existing resources (e.g., while they are implementing their current CQI systems). These tools will be particularly welcomed by evaluators currently struggling with positivist thinking and methods.

Measuring Socio-Emotional Skill, Impact, and Equity Outcomes

The positivist theory and methodology used by most researchers and evaluators is poorly suited for addressing the formative explanations that guide continuous quality improvement (CQI) processes and the nuanced impact models that pertain to questions about how and how much. QTurn’s Quality-Outcomes Design and Methods (Q-ODM) toolbox (Peck & Smith, 2020b) was created to address fundamental problems in the evaluation of out-of-school time (OST) programs (e.g., afterschool, child care, drop-in, mentoring, tutoring, etc.). In this white paper, we extend from a framework for individual socio-emotional (SEL) skills (Peck & Smith, 2020a) to address several issues in the applied measurement of individual SEL skills.

We present steps to (a) identify the real objects we seek to represent with measurement and models (i.e., the parts of an individual’s SEL skill set and the type and amount of skill change that is likely to occur during the program) and (b) produce SEL skill indicators and measures that are feasible and valid for both CQI and impact evaluation uses. With improved reasoning and evidence about the parts of SEL skill and individual skill change, we hope to help organizations produce local evidence and advocate both internally and externally for improved OST policies and increased investment.

Youth Program Quality Self-Assessment Pilot Study

Summary

Overall 24 sites within 17 grantees participated in the self-assessment pilot study by assembling staff teams to collect data and score the Youth Program Quality Assessment (PQA).

At each site an average of 5 staff spent an average of 13 staff hours to complete the self-assessment process.

Whether using an absolute standard or group norms as a benchmark for interpretation of data from the Youth PQA Self-Assessment Pilot Study (hereafter called the Pilot Study), quality scores were very positive for participating programs and also reflected the tendency of self-assessment scores to be biased toward higher quality levels.

The quality scores followed the same pattern as outside observer scores in other samples, highest on for issues of safety and staff support and lowest on higher order practices focused on interaction and engagement.

Youth PQA data collected using the self-assessment method demonstrated promising patterns of both internal consistency and concurrent validity with aligned youth survey responses.

Two thirds or more of sites reported that the observation and scoring process helped the selfassessment team to have greater insight into the operation of their programs, talk in greater depth about the program quality than usual, and have more concrete understanding of program quality.

Site directors and local evaluators said that the self-assessment process was a source of good conversations about program priorities and how to meet them. In almost all cases, concrete action followed from the self-assessment process.

Site directors and local evaluators demonstrated the ability to improvise the self-assessment method to fit local needs.

Program directors, site coordinators, and local evaluators have used the Youth PQA and statewide Youth PQA data to generate statewide program change models, suggesting that the instrument and data are useful for setting system-level improvement priorities.

Youth Report of Socio-Emotional Skills (YRSS)

QTurn’s Quality-Outcomes Design and Measure (Q-ODM) toolbox includes a comprehensive suite of evidence-informed measures designed to assess the quality and impact of out-of-school time (OST) programs. This manual provides information about a self-report survey measure designed to assess children’s and adolescents’ socio-emotional skills. We also provide some technical information about the measure and general guidance about how to administer, score, and interpret it.

The Youth Report of Socio-Emotional Skills (YRSS) was developed from extensive practitioner input and academic research about the socio-emotional behavior of school-aged children and adolescents1 and the socio-emotional mental skills likely to influence that behavior. The YRSS items were created, selected, and/or adapted from wide range of similar measurement instruments that were designed to assess youth’s beliefs about their own socio-emotional skills.

The YRSS can be used within the context of most types of OST program offerings. It can be completed by youth shortly after they first enter a program offering, in order to assess their baseline socio-emotional skills, and shortly after or near the end of a program period, in order to assess socioemotional skill growth. We generally recommend using the YRSS as a secondary outcome variable for typical OST program impact studies, and over longer periods of time between baseline and follow up, because we believe that it is less likely to detect socio-emotional skill growth than measures that are based on behavioral observations of youth (like the Adult Rating of Youth Behavior).

Please keep in mind that the YRSS does not provide a clinical assessment of children or youth. It is intended only for lower-stakes planning, improvement, and program evaluation purposes (e.g., where low scores signal areas of focus and support for children and youth but not failure, sanctions, or other disciplinary action).