Content and materials from the session Youth Voice for Climate Readiness in Schools, on October 13th, 2023 at the Michigan Afterschool Collaborative – Leadership Summit.
QTurn 101 – Outcome Measures
VIDEO 3: An introduction to QTurn’s new measures developed to promote more accurate, feasible and low cost evidence about impact and equity in child development settings.
QTurn 101 – More on Schema
VIDEO 2: Using the Neuroperson Framework to see through socio-emotional blind spots and demystify the reality of trauma for child development settings.
QTurn 101 – Neuroperson Framework
VIDEO 1: QTurn’s Neuroperson Framework helps to see children’s mental and behavioral skills more clearly, in support of attachment-aware trauma-informed (AATI) updates to best practice and quality.
Realist(ic) Evaluation Tools for OST Programs: The Quality-Outcomes Design and Methods (Q-ODM) Toolbox
Socio-emotional learning (SEL) skills are a partial but necessary cause of children’s developmental outcomes, and SEL skill growth is a key objective for nearly all out-of-school time (OST) programs. The Quality-Outcomes Design and Methods (Q-ODM) toolbox holds an integrated set of tools to measure and model children’s SEL skills, including how they change during, and in response to, OST programs (e.g., afterschool, school-age child care, workforce and career preparation, arts, sports). The Q-ODM toolbox helps organizational managers and evaluators to feasibly and cost-effectively adopt pattern-centered measures and models that produce actionable information for both continuous quality improvement (CQI) and impact evaluation.
The Q-ODM toolbox addresses practical questions about SEL skills and skill growth, such as: What is high-quality SEL support? How much SEL skill change does our program cause in each cycle? How much program quality does it take for stressed children to fully engage? Does our work create equity effects? The tools are divided into three groups: Design Tools, Analytic Tools, and Feedback Tools. These tools increase dramatically the value of CQI feedback for staff and the power of the analytic models used to evaluate program impact and equity effects for participating children. The Q-ODM toolbox was designed to empower internal and local evaluators to conduct rigorous and meaningful impact evaluations using existing resources (e.g., while they are implementing their current CQI systems). These tools will be particularly welcomed by evaluators currently struggling with positivist thinking and methods.
Citation: Peck, S. C., & Smith, C., (2020). Realist(ic) Evaluation Tools for OST Programs: The Quality-Outcomes Design and Methods (Q-ODM) Toolbox. [White Paper #3].
Measuring Socio-Emotional Skill, Impact, and Equity Outcomes
The positivist theory and methodology used by most researchers and evaluators is poorly suited for addressing the formative explanations that guide continuous quality improvement (CQI) processes and the nuanced impact models that pertain to questions about how and how much. QTurn’s Quality-Outcomes Design and Methods (Q-ODM) toolbox (Peck & Smith, 2020b) was created to address fundamental problems in the evaluation of out-of-school time (OST) programs (e.g., afterschool, child care, drop-in, mentoring, tutoring, etc.). In this white paper, we extend from a framework for individual socio-emotional (SEL) skills (Peck & Smith, 2020a) to address several issues in the applied measurement of individual SEL skills.
We present steps to (a) identify the real objects we seek to represent with measurement and models (i.e., the parts of an individual’s SEL skill set and the type and amount of skill change that is likely to occur during the program) and (b) produce SEL skill indicators and measures that are feasible and valid for both CQI and impact evaluation uses. With improved reasoning and evidence about the parts of SEL skill and individual skill change, we hope to help organizations produce local evidence and advocate both internally and externally for improved OST policies and increased investment.
Citation: Smith, C., & Peck, S. C. (2020). Measuring Socio-Emotional Skill, Impact, and Equity Outcomes [White Paper #2].
Socio-Emotional Skills, Quality, and Equity: The Multilevel Person-in-Context~neuroperson (MPCn) Framework
Evaluation evidence about the relations among children’s prior history, engagement in program settings, resulting SEL skill growth, and the ultimately desired transfer outcomes (e.g., agency to succeed in other settings) has been sporadic and fragmented. One reason for this may be that the positivist theory and methodology used by most researchers and evaluators is poorly suited to the formative explanations that guide continuous quality improvement (CQI) processes. As a result, we lack nuanced impact models that address questions about how and how much, or the information necessary for organizational decision-making. QTurn’s Quality-Outcomes Design and Methods (Q-ODM) toolbox (Peck & Smith, 2020) was created to address these fundamental problems in the evaluation of education settings, with a specific focus on out-of-school time (OST; afterschool, child care, drop-in, mentoring, tutoring, etc.) programs.
In this white paper, we introduce a theoretical framework designed to describe the integrated set of mental and behavioral parts and processes (i.e., schemas, beliefs, and awareness) that are socio-emotional skills and that produce both basic and advanced forms of agency. With improved definitions and understanding of SEL skills, and the causes of SEL skill growth, we hope to improve reasoning about programs and policies for socio-emotional supports in any setting where children spend time. Perhaps most importantly, we hope to inform policy decisions and advance applied developmental science by improving the accuracy and meaningfulness of basic data on children’s SEL skill growth.
Citation: Peck, S. C., & Smith, C., (2020). Socio-Emotional Skills, Quality, and Equity: The Multilevel Person-in-Context~neuroperson (MPCn) Framework. [White Paper #1].
The Afterschool Learning at a Distance: Key Themes and Promising Practices
The Afterschool Learning at a Distance: Key Themes and Promising Practices describes the experiences and practices of Genesee Intermediate School District: Brides to Success’ (GISD) Team Leads and direct staff serving children and families after substantially redesigning afterschool programming due to the COVID-19 crisis. While the original evaluation plan was to do a (second) round of in-person observations (no longer possible due to school closures), QTurn conducted 15 staff interviews via zoom. Key themes from the interviews include professional uncertainty made manageable because of their strong organizational culture, the importance of addressing inequity, adjustment to flexible service, and a focus on a whole child, whole family approach.
Citation: Smith, C., Smith, L., Roy, L., & Peck, S. C. (2020). Afterschool learning at a distance: key themes and promising practices [Grantee Evaluation Report]. Ypsilanti, MI: QTurn.
The multilevel person in context ~ neuroperson (MPCn) model: Guidance for quality improvement systems (QIS) focused on socio-emotional skill growth and transfer outcomes
Reviewing theory and research on socio-emotional learning (SEL) highlights the importance of a wide range of psychological and behavioral skills, ranging from very specific psychological processes that occur on the order of milliseconds (e.g., updating working memory) to broad patterns of behavior that occur over minutes, days, and months (e.g., teamwork and relationship skills). Attempts to organize this vast array of skills into a coherent theoretical or measurement framework has yielded dozens of unique but overlapping frameworks. For example, a recent review of SEL theory, research, and practice by the American Institutes for Research (Berg et al., 2017) found over 100 different SEL frameworks.
The multilevel person-in-context model of youth development programs (Smith, McGovern, Peck, et al., 2016) facilitates thinking about how the SEL skills being developed at the point of service (POS) are both (a) embedded within the wider context of policy decisions, family background, and out-of-school time (OST) program quality and (b) related subsequently to shorter-term youth outcomes (e.g., SEL skill growth) and longer-term youth achievements (e.g., graduation and employment). The multilevel person-in-context model, and the corresponding neuroperson model that focuses on the structure and dynamics of SEL skill growth (described below), together constitute the multilevel person-in-context, neuroperson (MPCn) framework that was developed to improve the precision, validity, and comprehension of performance data used in lower-stakes quality improvement systems (QIS) in the OST sector (Smith, McGovern, Larson, et al., 2016; Smith, McGovern, Peck, et al., 2016; Smith et al., 2019).
Citation: Peck, S. C., Smith, C., & Smith, L. (2019). The multilevel person-in-context ~ neuroperson (MPCn) model: Guidance for quality improvement systems (QIS) focused on socio-emotional skill growth and transfer outcomes. Ypsilanti, MI: QTurn.
Promoting Healthy Development of Young People: Outcomes Framework 2.0
In the summer of 2018, the Local Government Association (LGA) in England commissioned the Centre for Youth Impact to produce an outcomes framework to help partners across the English youth sector to develop and agree on mutual aims to support young people in their local areas. The work was in response to LGA’s consultations that led to its vision statement described in the report, Bright Futures: our vision for youth services, published at the end of 2017. In that report, the authors noted:
“A clear outcomes framework can help to effectively monitor the impact of a service at key milestones to spot where things aren’t working and provide opportunities to make changes where needed. It can also support evidence of collective impact across the system.”
The proposed framework was intended to support partners’ efforts to track and understand the short-, medium-, and longer-term impacts of their work on the lives of young people. The framework needed to be simple and adaptable to provision for different groups of young people and for diverse approaches.
This document is an update on the framework and is the result of two phases of work: an initial phase including desk research and widespread consultation with practitioners, commissioners and elected members, and a second phase to test the proposed framework in action. The work was undertaken by the Centre’s network of regional impact leads and its central team.