Category: White Paper

Realist(ic) Evaluation Tools for OST Programs: The Quality-Outcomes Design and Methods (Q-ODM) Toolbox

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email
Download

Socio-emotional learning (SEL) skills are a partial but necessary cause of children’s developmental outcomes, and SEL skill growth is a key objective for nearly all out-of-school time (OST) programs. The Quality-Outcomes Design and Methods (Q-ODM) toolbox holds an integrated set of tools to measure and model children’s SEL skills, including how they change during, and in response to, OST programs (e.g., afterschool, school-age child care, workforce and career preparation, arts, sports). The Q-ODM toolbox helps organizational managers and evaluators to feasibly and cost-effectively adopt pattern-centered measures and models that produce actionable information for both continuous quality improvement (CQI) and impact evaluation.

The Q-ODM toolbox addresses practical questions about SEL skills and skill growth, such as: What is high-quality SEL support? How much SEL skill change does our program cause in each cycle? How much program quality does it take for stressed children to fully engage? Does our work create equity effects? The tools are divided into three groups: Design Tools, Analytic Tools, and Feedback Tools. These tools increase dramatically the value of CQI feedback for staff and the power of the analytic models used to evaluate program impact and equity effects for participating children. The Q-ODM toolbox was designed to empower internal and local evaluators to conduct rigorous and meaningful impact evaluations using existing resources (e.g., while they are implementing their current CQI systems). These tools will be particularly welcomed by evaluators currently struggling with positivist thinking and methods.

Citation: Peck, S. C., & Smith, C., (2020). Realist(ic) Evaluation Tools for OST Programs: The Quality-Outcomes Design and Methods (Q-ODM) Toolbox. [White Paper #3].

Download

Measuring Socio-Emotional Skill, Impact, and Equity Outcomes

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email
Download

The positivist theory and methodology used by most researchers and evaluators is poorly suited for addressing the formative explanations that guide continuous quality improvement (CQI) processes and the nuanced impact models that pertain to questions about how and how much. QTurn’s Quality-Outcomes Design and Methods (Q-ODM) toolbox (Peck & Smith, 2020b) was created to address fundamental problems in the evaluation of out-of-school time (OST) programs (e.g., afterschool, child care, drop-in, mentoring, tutoring, etc.). In this white paper, we extend from a framework for individual socio-emotional (SEL) skills (Peck & Smith, 2020a) to address several issues in the applied measurement of individual SEL skills.

We present steps to (a) identify the real objects we seek to represent with measurement and models (i.e., the parts of an individual’s SEL skill set and the type and amount of skill change that is likely to occur during the program) and (b) produce SEL skill indicators and measures that are feasible and valid for both CQI and impact evaluation uses. With improved reasoning and evidence about the parts of SEL skill and individual skill change, we hope to help organizations produce local evidence and advocate both internally and externally for improved OST policies and increased investment.

Citation: Smith, C., & Peck, S. C. (2020). Measuring Socio-Emotional Skill, Impact, and Equity Outcomes [White Paper #2].

Download

Socio-Emotional Skills, Quality, and Equity: The Multilevel Person-in-Context~neuroperson (MPCn) Framework

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email
Download

Evaluation evidence about the relations among children’s prior history, engagement in program settings, resulting SEL skill growth, and the ultimately desired transfer outcomes (e.g., agency to succeed in other settings) has been sporadic and fragmented. One reason for this may be that the positivist theory and methodology used by most researchers and evaluators is poorly suited to the formative explanations that guide continuous quality improvement (CQI) processes. As a result, we lack nuanced impact models that address questions about how and how much, or the information necessary for organizational decision-making. QTurn’s Quality-Outcomes Design and Methods (Q-ODM) toolbox (Peck & Smith, 2020) was created to address these fundamental problems in the evaluation of education settings, with a specific focus on out-of-school time (OST; afterschool, child care, drop-in, mentoring, tutoring, etc.) programs.

In this white paper, we introduce a theoretical framework designed to describe the integrated set of mental and behavioral parts and processes (i.e., schemas, beliefs, and awareness) that are socio-emotional skills and that produce both basic and advanced forms of agency. With improved definitions and understanding of SEL skills, and the causes of SEL skill growth, we hope to improve reasoning about programs and policies for socio-emotional supports in any setting where children spend time. Perhaps most importantly, we hope to inform policy decisions and advance applied developmental science by improving the accuracy and meaningfulness of basic data on children’s SEL skill growth.

Citation: Peck, S. C., & Smith, C., (2020). Socio-Emotional Skills, Quality, and Equity: The Multilevel Person-in-Context~neuroperson (MPCn) Framework. [White Paper #1].

Download