Introduction to QTurn White Papers

We, at QTurn, are pleased to share the first three, in a series of four, white papers. White Paper 1, Socio-Emotional Skills, Quality, and Equity (Peck & Smith, 2020), provides a translational framework for understanding our relatively unique view of the key parts of a socio-emotional skill set. In short, we develop a case for supplementing the traditional focus on student beliefs and behavior with a much more extensive focus on students’ emotional life and the attention skills necessary for becoming the primary authors of their own development.

You can download White Paper 1 from our website or ResearchGate. We’ve also published a blog describing what we think are some of the important points and implications of White Paper 1.

Although our work is anchored in the wide and deep range of developmental supports that are currently evident in the out-of-school time (OST) field, we view the “neuroperson” model described in White Paper 1 as applying to all adults and children in all settings. Quoting from the paper:

We introduce a theoretical framework designed to describe the integrated set of mental and behavioral parts and processes (i.e., schemas, beliefs, and awareness) that are socio-emotional skills and that produce both basic and advanced forms of agency. With improved definitions and understanding of SEL skills, and the causes of SEL skill growth, we hope to improve reasoning about programs and policies for socio-emotional supports in any setting where children spend time. Perhaps most importantly, we hope to inform policy decisions and advance applied developmental science by improving the accuracy and meaningfulness of basic data on children’s SEL skill growth. (p. 3)

The series of white papers will define what exactly we do and believe at QTurn. After the translational framework is explained in White Paper 1, White Paper 2 – Measuring Socio-Emotional Skill, Impact, and Equity Outcomes (Smith & Peck, 2020a) – provides guidance for selecting feasible and valid SEL skill measures. White Paper 3 –  Realist(ic) Evaluation Tools for OST Programs – integrates the SEL framework and measures with a pattern-centered approach to both CQI and impact evaluation. White Paper 4 – Citizen Science and Advocacy in OST (Smith & Peck, 2020b) – presents an alternative evidence-based approach to improving both the impact and equity of OST investments. Over the next few weeks, we’ll be releasing blogs related to White Papers 2 and 3.

We’ll also be updating our website as we go along and hope to be joined in the blogging by a couple of expert clients in Flint and London. That’s it for now. We look forward to sharing further information in the coming months and would love to receive any feedback you think might help further the cause of supporting OST staff and students.

Reflections on White Paper 1

In conjunction with the release of White Paper 1 this week – A Framework for Socio-Emotional Skills, Quality, and Equity – we want to mention a few of the highlights:

What are socio-emotional skills? In our view, a person’s socio-emotional skills are integrated sets of mental and behavioral parts and processes (i.e., schemas, beliefs, and awareness); these integrated systems are socio-emotional skills and produce both basic and advanced forms of agency.

Why are socio-emotional skills important? Socio-emotional skills have a compounding effect on many developmental outcomes that has been described as dynamic complementarity (Heckman, 2007); that is, socio-emotional skills beget other types of skills. Children and adults operating at high levels of SEL skill can more easily get on to the business of learning what the context has to offer. Settings that do not address SEL skills can become a further cause of educational inequity.

Why are organizations and policies struggling to implement socio-emotional skill reforms? A recent review found over 100 different frameworks describing SEL skills and supports (Berg et al., 2017). This cacophony of words and concepts undermines the shared understanding and language necessary for coordinated action, both within organizations doing the work and among evaluators producing the evidence.[i] Confusion about what constitutes SEL skill, and how “skill” may or may not differ from many other concepts – such as, competence, abilities, traits, attitudes, and mindsets – undermines scientific progress and slows policy processes that rely on at least approximate consensus around shared meanings and objects of measurement.

How can the QTurn socio-emotional skills framework help increase the effectiveness of reform? By defining, naming, and sorting out the key parts of integrated SEL skill sets, we can much more effectively measure and model both changes in socio-emotional skills and, ultimately, impacts on outcomes and equity. In White Paper 2, we extend from the socio-emotional skills framework described in White Paper 1 to corresponding guidance for measuring socio-emotional skills with increased precision, accuracy, and sensitivity.

We’ll be back with more soon…

 


[i] Given the extent of diversity across SEL frameworks, Jones et al. (2019) developed resources to help stakeholders understand the unique strengths of different frameworks as well as the alignment between core elements of these different frameworks. The general conclusions from this work are (a) there is currently no single consensus framework that is obviously more scientifically or practically valid than any or all of the others, and (b) the use of the same terms by different frameworks where presumably referring to different things (i.e., jingle fallacies), and the use of different terms by different frameworks where presumably referring to the same things (i.e., jangle fallacies), are abiding challenges faced by stakeholders charged with making funding, evaluation, training, performance, measurement, and analysis decisions. Our approach is designed to help solve these problems.

Realist(ic) Evaluation Tools for OST Programs: The Quality-Outcomes Design and Methods (Q-ODM) Toolbox

Socio-emotional learning (SEL) skills are a partial but necessary cause of children’s developmental outcomes, and SEL skill growth is a key objective for nearly all out-of-school time (OST) programs. The Quality-Outcomes Design and Methods (Q-ODM) toolbox holds an integrated set of tools to measure and model children’s SEL skills, including how they change during, and in response to, OST programs (e.g., afterschool, school-age child care, workforce and career preparation, arts, sports). The Q-ODM toolbox helps organizational managers and evaluators to feasibly and cost-effectively adopt pattern-centered measures and models that produce actionable information for both continuous quality improvement (CQI) and impact evaluation.

The Q-ODM toolbox addresses practical questions about SEL skills and skill growth, such as: What is high-quality SEL support? How much SEL skill change does our program cause in each cycle? How much program quality does it take for stressed children to fully engage? Does our work create equity effects? The tools are divided into three groups: Design Tools, Analytic Tools, and Feedback Tools. These tools increase dramatically the value of CQI feedback for staff and the power of the analytic models used to evaluate program impact and equity effects for participating children. The Q-ODM toolbox was designed to empower internal and local evaluators to conduct rigorous and meaningful impact evaluations using existing resources (e.g., while they are implementing their current CQI systems). These tools will be particularly welcomed by evaluators currently struggling with positivist thinking and methods.

Citation: Peck, S. C., & Smith, C., (2020). Realist(ic) Evaluation Tools for OST Programs: The Quality-Outcomes Design and Methods (Q-ODM) Toolbox. [White Paper #3].

Measuring Socio-Emotional Skill, Impact, and Equity Outcomes

The positivist theory and methodology used by most researchers and evaluators is poorly suited for addressing the formative explanations that guide continuous quality improvement (CQI) processes and the nuanced impact models that pertain to questions about how and how much. QTurn’s Quality-Outcomes Design and Methods (Q-ODM) toolbox (Peck & Smith, 2020b) was created to address fundamental problems in the evaluation of out-of-school time (OST) programs (e.g., afterschool, child care, drop-in, mentoring, tutoring, etc.). In this white paper, we extend from a framework for individual socio-emotional (SEL) skills (Peck & Smith, 2020a) to address several issues in the applied measurement of individual SEL skills.

We present steps to (a) identify the real objects we seek to represent with measurement and models (i.e., the parts of an individual’s SEL skill set and the type and amount of skill change that is likely to occur during the program) and (b) produce SEL skill indicators and measures that are feasible and valid for both CQI and impact evaluation uses. With improved reasoning and evidence about the parts of SEL skill and individual skill change, we hope to help organizations produce local evidence and advocate both internally and externally for improved OST policies and increased investment.

Citation: Smith, C., & Peck, S. C. (2020). Measuring Socio-Emotional Skill, Impact, and Equity Outcomes [White Paper #2].

Socio-Emotional Skills, Quality, and Equity: The Multilevel Person-in-Context~neuroperson (MPCn) Framework

Evaluation evidence about the relations among children’s prior history, engagement in program settings, resulting SEL skill growth, and the ultimately desired transfer outcomes (e.g., agency to succeed in other settings) has been sporadic and fragmented. One reason for this may be that the positivist theory and methodology used by most researchers and evaluators is poorly suited to the formative explanations that guide continuous quality improvement (CQI) processes. As a result, we lack nuanced impact models that address questions about how and how much, or the information necessary for organizational decision-making. QTurn’s Quality-Outcomes Design and Methods (Q-ODM) toolbox (Peck & Smith, 2020) was created to address these fundamental problems in the evaluation of education settings, with a specific focus on out-of-school time (OST; afterschool, child care, drop-in, mentoring, tutoring, etc.) programs.

In this white paper, we introduce a theoretical framework designed to describe the integrated set of mental and behavioral parts and processes (i.e., schemas, beliefs, and awareness) that are socio-emotional skills and that produce both basic and advanced forms of agency. With improved definitions and understanding of SEL skills, and the causes of SEL skill growth, we hope to improve reasoning about programs and policies for socio-emotional supports in any setting where children spend time. Perhaps most importantly, we hope to inform policy decisions and advance applied developmental science by improving the accuracy and meaningfulness of basic data on children’s SEL skill growth.

Citation: Peck, S. C., & Smith, C., (2020). Socio-Emotional Skills, Quality, and Equity: The Multilevel Person-in-Context~neuroperson (MPCn) Framework. [White Paper #1].

Guidance for Out-of-school time Learning at a Distance (GOLD)

The Guidance for Out-of-School Time Learning at a Distance (GOLD) is a program quality assessment for 21st Century Community Learning Centers (21st CCLC) and other community-based programs (e.g., school-age childcare, YMCA, 4 H, Boys & Girls Clubs) that have transitioned virtual, socially distanced in-person, and blended service models. For these new models, the GOLD will:

Help staff document local standards for quality and make improvements

Help leaders to assess organizational readiness and demonstrate accountability

Help funders and intermediaries target supports

By explicitly engaging family or caregiver strengths, assuring flexible supports, and sharing accurate information about the future (e.g., plans for school and OST in the coming months), the GOLD was specifically designed to address both the young person’s socio-emotional wellness and the conditions of academic learning. The four GOLD domains of quality are: I. Family Centered Engagement, II. Individual Learning Environment, III. Distance Programming, and IV. Planning with Children, Families, Caregivers, and Schools.

Manual

Tool Introduction, Protocol, Standards and Indicators, FAQ, Promising Practices, Method for development and Contributors.

Download

Self Assessment Form

All 27 indicators and self-assessment rubric form. This document can be an be completed digitally or manually.

Download

Database of resources

These resources provide additional information and offer insight and best practice for educators.

Open

These materials were developed under a grant awarded by the Michigan Department of Education.

index

Citation: Smith, C., Roy, L., Smith, L., Sutton, M., Peck, S. C., & Porter, K. (2020). Guidance for Out-of-School Time Learning at a Distance: Standards and Self-Assessment Manual. Lansing, MI: Michigan Afterschool Partnership and QTurn LLC.

Subjective Well-Being (SWB)

In OST organizations that focus on student’s socio-emotional skills and wellbeing, pandemic conditions created new urgency to staff’s desire to know how students and caregivers were doing. While many OST organizations know how to ask families about basic needs created by the pandemic – health care, housing, food, transportation – sometimes staff are less comfortable checking in on mental health issues.

The concept of subjective-well-being (SWB) has traditionally been assessed using measures of happiness and satisfaction with life. Most of the measurement instruments developed to assess SWB use several items to assess each of several dimensions. However, for simplicity and efficiency, we recommend using only one item for each of the three dimensions: positive feelings, negative feelings, and life satisfaction.

Positive Feelings

On a scale of 1 to 10: During the past month, how often have you had positive feelings (e.g., happiness, joy, interest, calmness, and love)?

Negative Feelings

On a scale of 1 to 10: During the past month, how often have you had negative feelings (e.g., sadness, fear, boredom, anxiety, and hate)?

Life Satisfaction

On a scale of 1 to 10: During the past month, how satisfied have you felt with your life?

Newark Trust for Education Project

The Newark Trust for Education (NTE) Parent Child Plus (PC+) program is an evidence-based early childhood education program for families in the Newark, NJ. NTE seeks to evaluate performance by conducting analyses of existing data for a cohort of over 80 families, assessed four times over 46 weeks using observational measures of parenting practices and children’s socio-emotional skills. PC+ is intended to result in “improved child behaviors related to social‐emotional development and self‐regulation skills” (Organizational Research Services [ORS], 2010, p. 23).

The Quality-Impact-Equity Design and Methods (QDM) Toolbox (Smith, Peck, Roy, & Smith, 2019; Smith, Peck, & McNeil, 2020) was used to: (a) reconfigure existing measures for Parenting Practice Quality and Child SEL Skill to maximize reliability and validity for measuring socio-emotional skills and learning (SEL); (b) produce holistic profiles of parent and child skill (e.g., “whole child”) at each timepoint; and (c) apply pattern-centered analytics to estimate impact and equity effects of the PC+ program as implemented in Newark. Please note: We define impact in terms of the actual “in-the-world” structure of causes and effects, not in terms of counterfactuals. A brief description of the QDM methodology is provided in Appendix A (see also Smith et al., 2019)..

The PC+ program results reveal an overall impact pattern that suggests both a strong relation between parent and child skills and an effect of home visitors on both parent and child skills. Although, in almost all cases, the children of parents with high or growing parenting skills outperformed children with low or declining parenting skills, many children with parents in the low-skill profile for Parenting Practice Quality still experienced growth in SEL skills. This finding suggests that PC+ is working as it should, with parents and home visitors both having direct effects on child SEL skill growth. To fully demonstrate the impact of the NTE PC+ program given this “triadic” causal flow, we recommend (a) improving measures of PC+ fidelity and (b) including a small no-program sample of parents and children.

Impact Evaluation Report

Download

Summary Report

Download

Report Appendices A-I

Request

Child SEL Skill Infographic

Download

Parent Practice Quality Infographic

Download

Citation for the full report: Peck, S. C., & Smith, C. (2020). Impact Evaluation for the Parent Child Plus Program, Newark Trust for Education. Ypsilanti, MI: QTurn.

The Afterschool Learning at a Distance: Key Themes and Promising Practices

The Afterschool Learning at a Distance: Key Themes and Promising Practices describes the experiences and practices of Genesee Intermediate School District: Brides to Success’ (GISD) Team Leads and direct staff serving children and families after substantially redesigning afterschool programming due to the COVID-19 crisis. While the original evaluation plan was to do a (second) round of in-person observations (no longer possible due to school closures), QTurn conducted 15 staff interviews via zoom. Key themes from the interviews include professional uncertainty made manageable because of their strong organizational culture, the importance of addressing inequity, adjustment to flexible service, and a focus on a whole child, whole family approach.

Citation: Smith, C., Smith, L., Roy, L., & Peck, S. C. (2020). Afterschool learning at a distance: key themes and promising practices [Grantee Evaluation Report]. Ypsilanti, MI: QTurn.

BLACK LIVES MATTER STATEMENT

The consultants of QTurn stand in solidarity with protesters demanding justice for George Floyd, Breonna Taylor, Ahmaud Arbery, Tony McDade, and the many others who have lost their lives to police violence and apartheid policies. To the organizers, protesters, activists, and educators who have thoughtfully and urgently called for systematic change and accountability; we stand with you.

Racism is learned and internalized, and so too is anti-racism. We are committed to helping educators create spaces where socio-emotional skills are used to disrupt racist ideas and policies and where socio-emotional supports are a foundation for learning anti-racist beliefs and practices. We are committed to helping educators produce objective information and to suspend judgement until students and teachers are understood on their own terms.

WE COMMIT TO ACTION, AS WE ALWAYS HAVE, IN THE FIGHT FOR FREEDOM AND LIBERATION OF ALL PEOPLE.