Publication – Page 2

Adult Rating of Youth (ARY)

QTurn’s Quality-Outcomes Design and Measure (Q-ODM) toolbox includes a comprehensive suite of evidence-informed measures designed to assess the quality and impact of out-of-school time (OST) programs. This manual provides information about an observational rating instrument designed to assess children’s and adolescents’ core socio-emotional skill sets. We also provide some technical information about the measure and general guidance about how to administer, score, and interpret it.

The Adult Rating of Youth (ARY) was developed by first selecting and modifying items from the Adult Rating of Youth Behavior that pertained specifically to schemas or awareness and then developing additional items aligned primarily to schemas and awareness. Each item was refined by iterating variations in wording and examples with experienced OST staff. The ARY is generally applicable to, and can be used for, any and all types of OST program offerings.

The ARY can also be used with children and youth classified as having an individualized education program (IEP) or similar designation in the same way you would use it for anyone else. We generally recommend using the ARY as the primary outcome variable for typical OST program impact studies because we believe that it is the most likely to detect socio-emotional skill growth.

Please keep in mind that the ARY does not provide a clinical assessment of children or youth. It is intended only for lower-stakes planning, improvement, and program evaluation purposes (e.g., where low scores signal areas of focus and support for children and youth but not failure, sanctions, or other disciplinary action).

Teacher Practices Instrument (TPI)

QTurn’s Quality-Outcomes Design and Measure (Q-ODM) toolbox includes a comprehensive suite of
evidence-informed measures designed to assess the quality and impact of out-of-school time (OST) programs. This manual provides information about an observational rating instrument designed to assess teachers’ and OST staff’s instructional practice quality at the point of service. We also provide some technical information about the measure and general guidance about how to administer, score, and interpret it.

The Teacher Practices Instrument (TPI) was developed by the researchers at QTurn, in collaboration with expert practitioners and numerous OST organizations by creating observational rating items focused on the warmth, responsivity, scaffolding, and encouragement of teachers and OST staff where interacting with children and youth at the point of service. Each item was refined by iterating variations in wording and examples with experienced teachers and OST staff.

The TPI is generally applicable to, and can be used for, any and all types of school classrooms and OST
program offerings. The TPI can also be used with children and youth classified as having an individualized education program (IEP) or similar designation in the same way you would use it for anyone else. We generally recommend using the TPI as the primary measure of teacher’s and OST staff’s instructional practice quality for impact studies because we believe that it is accurately reflects the kinds of practices most likely to influence children’s and youth’s socio-emotional skill growth. Please keep in mind that the TPI does not provide a clinical assessment of classroom teachers or OST staff. It is intended only for lower-stakes planning, improvement, and program evaluation purposes (e.g., where low scores signal areas of focus and support for teachers and staff but not failure, sanctions, or other disciplinary action).

Subjective Well-Being (SWB)

In OST organizations that focus on student’s socio-emotional skills and wellbeing, pandemic conditions created new urgency to staff’s desire to know how students and caregivers were doing. While many OST organizations know how to ask families about basic needs created by the pandemic – health care, housing, food, transportation – sometimes staff are less comfortable checking in on mental health issues.

The concept of subjective-well-being (SWB) has traditionally been assessed using measures of happiness and satisfaction with life. Most of the measurement instruments developed to assess SWB use several items to assess each of several dimensions. However, for simplicity and efficiency, we recommend using only one item for each of the three dimensions: positive feelings, negative feelings, and life satisfaction.

Positive Feelings

On a scale of 1 to 10: During the past month, how often have you had positive feelings (e.g., happiness, joy, interest, calmness, and love)?

Negative Feelings

On a scale of 1 to 10: During the past month, how often have you had negative feelings (e.g., sadness, fear, boredom, anxiety, and hate)?

Life Satisfaction

On a scale of 1 to 10: During the past month, how satisfied have you felt with your life?

Parent Practice Quality

QTurn’s Multilevel Person-in-Context ~neuroperson (MPCn) model was used to select PACT items corresponding to four different kinds of parenting practices that promote children’s SEL skill growth. As described in the document in more detail, we use the terms Warmth, Responsivity, Scaffolding, and Attention because they reflect the core aspects of parenting practice necessary for calibrating socialization environments to the current skill levels of participating children. Although the basic idea is that parenting practices that combine moderate difficulty, positive affect, adult modeling, and co-participation in the learning task promote child engagement in the learning process, hence SEL skill development and integration.

Guidance for Out-of-School Time Learning at a Distance

The Guidance for Out-of-School Time Learning at a Distance (GOLD) is a program quality assessment for 21st Century Community Learning Centers (21st CCLC) and other community-based programs (e.g., school-age childcare, YMCA, 4 H, Boys & Girls Clubs) that have transitioned virtual, socially distanced in-person, and blended service models. For these new models, the GOLD will:

Help staff document local standards for quality and make improvements

Help leaders to assess organizational readiness and demonstrate accountability

Help funders and intermediaries target supports

By explicitly engaging family or caregiver strengths, assuring flexible supports, and sharing accurate information about the future (e.g., plans for school and OST in the coming months), the GOLD was specifically designed to address both the young person’s socio-emotional wellness and the conditions of academic learning. The four GOLD domains of quality are:

I. Family Centered Engagement
II. Individual Learning Environment
III. Distance Programming
IV. Planning with Children, Families, Caregivers, and Schools

These materials were developed under a grant awarded by the Michigan Department of Education.

Afterschool Learning at a Distance: Key Themes and Promising Practices

Afterschool Learning at a Distance: Key Themes and Promising Practices describes the experiences and practices of Genesee Intermediate School District: Bridges to Success (GISD) Team Leads and direct staff serving children and families after substantially redesigning afterschool programming due to the COVID-19 crisis. While the original evaluation plan was to do a (second) round of in-person observations (no longer possible due to school closures), QTurn conducted 15 staff interviews via zoom. Key themes from the interviews include professional uncertainty made manageable because of their strong organizational culture, the importance of addressing inequity, adjustment to flexible service, and a focus on a whole child, whole family approach.

Management Practices Self-Assessment (MPSA)

The Management Practices Self-Assessment (MPSA) is a measurement instrument created to empower Michigan 21st CCLC Project Directors, who manage at least one program site. The MPSA provides Project Directors with the opportunity to check-in with themselves and their team by identifying practices and policies that are well-documented and working well, or that could be updated or improved, and areas where they need additional assistance to reach their full potential.

Child SEL Skills

This two-page information sheet explains youth profiles for SEL skills (low, moderate, and exemplary) as they relate to schemas, beliefs, and attention. Created as a supplement to QTurn’s impact report for the Newark Trust for Education Project (NTE) Parent Child Plus (PC+) program, this overview explains how various measurements are tied to key outcomes.

Management Practices Self-Assessment (MPSA) Guidebook

The Management Practices Self-Assessment (MPSA) is a measurement instrument created to empower Michigan 21st CCLC Project Directors to be (a) reflective and intentional about organizational practices and culture; (b) quality improvement leaders at their sites; (c) engaged with youth, families, and their communities; and (d) well-informed about state-wide requirements and guidance. The MPSA provides a set of items for management best practices in 21st CCLC administrative and program settings. MPSA provides Project Directors with the opportunity to check-in with themselves and their team by identifying practices and policies that are well-documented and working well, or that could be updated or improved, and areas where they need additional assistance to reach their full potential.

Before completing the MPSA, consider several ways in which the measurement process and resulting data can be used:

Planning for the upcoming year. Project Directors can use this tool to document practices and procedures that are already in place, to create or update protocols and policies, to schedule trainings throughout the year, and as part of the quality improvement planning process. For new Grantee Directors, the MPSA is an important support to start-up planning with staff and school personnel. The MPSA is also a good way to request assistance from the state and technical assistance (TA) providers.
Reflecting on the past progress. When used at the beginning and at the end of the year, Project Directors can reflect on the progress they were able to make over the year and track issues that remain to be addressed. The MPSA also allows Project Directors to identify which practices or efforts were not effective and where additional TA or support would be helpful.
Preserving institutional knowledge. Project Directors can use the MPSA to document the location of written policies, procedures, and practices in a single reference document. This allows Project Directors to explain practices to staff and program partners and supports smooth transitions from one Project Director to another.

Although the primary purpose of the MPSA is to facilitate Project Director self-reflection, a second purpose is to continuously improve 21st CCLC management skills and systems. Once a year, all Project Directors will be asked to provide the results of the MPSA to MDE by uploading a completed copy to the Grant Electronic Monitoring System (GEMS). The results will be analyzed and used to guide MDE in selecting trainings, conference sessions, and other professional development opportunities throughout the ensuing year. At no time will results be used by MDE for grantee assessment.

The MPSA is made up of 4 domains, 8 scales, and 25 items of management best practices in 21st CCLC program settings and should take no longer than 2 hours to complete. The MPSA requires the user to list the best practice; answer questions about implementation and the need for technical assistance; and provide evidence, documentation, and notes. For more detailed instruction, see the MPSA Protocol.

Note: These materials were developed under a grant awarded by the Michigan Department of Education.

The Multilevel Person in Context ~ neuroperson (MPCn) Model

Reviewing theory and research on socio-emotional learning (SEL) highlights the importance of a wide range of psychological and behavioral skills, ranging from very specific psychological processes that occur on the order of milliseconds (e.g., updating working memory) to broad patterns of behavior that occur over minutes, days, and months (e.g., teamwork and relationship skills). Attempts to organize this vast array of skills into a coherent theoretical or measurement framework has yielded dozens of unique but overlapping frameworks. For example, a recent review of SEL theory, research, and practice by the American Institutes for Research (Berg et al., 2017) found over 100 different SEL frameworks.

The multilevel person-in-context model of youth development programs (Smith, McGovern, Peck, et al., 2016) facilitates thinking about how the SEL skills being developed at the point of service (POS) are both (a) embedded within the wider context of policy decisions, family background, and out-of-school time (OST) program quality and (b) related subsequently to shorter-term youth outcomes (e.g., SEL skill growth) and longer-term youth achievements (e.g., graduation and employment). The multilevel person-in-context model, and the corresponding neuroperson model that focuses on the structure and dynamics of SEL skill growth (described below), together constitute the multilevel person-in-context, neuroperson (MPCn) framework that was developed to improve the precision, validity, and comprehension of performance data used in lower-stakes quality improvement systems (QIS) in the OST sector (Smith, McGovern, Larson, et al., 2016; Smith, McGovern, Peck, et al., 2016; Smith et al., 2019).

This publication is not available for download. Please contact us to request access.