Continuous Quality Improvement and Evaluation in 2020: A Plan for 21st Century Community Learning Centers

During times of crisis when programs are under tremendous pressures, evaluation and assessment can be challenging. Programs enter triage mode, putting their limited time and energy into the most urgent tasks. This heightens the need for evaluation that reduces strain and improves capacity. When conditions that created the crisis are long-lasting, like the coronavirus pandemic, it becomes necessary to revisit and restore vital activities that may have been moved to the backburner, but to do this successfully often requires intelligent redesign. How can the same needs be met in a new way? How can evaluation and assessment be adapted to succeed in challenging conditions?

QTurn has developed a comprehensive evaluation plan for afterschool programs at a moment when redesign of the service and delivery of the redesigned service are happening at the same time. This plan, ”Afterschool Evaluation Plan 2020,” was developed to address the unique needs of programs in the 2020-2021 school year – and support compliance with the specific requirements for the 21st Century Community Learning Centers. An evaluation plan for 2020 must remove burdens rather than adding to them and make life easier not harder. Aware of these needs, QTurn’s design includes short, validated assessment tools, guidance available through online trainings, and a reassuring, therapeutic approach referred to as the “lower stakes” model.

A Lower Stakes Approach

The lower stakes model is a strong component of QTurn’s work. Lower stakes means that the results of assessments are used to support and inform, and program staff are able to interpret the meaning of their own individual and group performance data. In a lower stakes model, the results of assessments do not influence funding or prompt sanctions. Instead, low assessments scores are opportunities for mutual learning, support, and growth. 

While this approach has been integral to the work of QTurn’s founder, Charles Smith, for decades, lower stakes is especially critical in the 2020-2021 school year as program staff strive to adapt to a new normal. The AEP 2020 is intentionally designed to alleviate stress and confusion and help staff adapt to rapid change and achieve shared meaning.

User-Friendly Assessment Tools

The Afterschool Evaluation Plan 2020 includes three assessment tools designed for remote or in-person programming (or a combination of both). The first measures fidelity to best practices at the management level. The second captures quality at the point-of-service (and applies to home learning environments). The third charts the growth of social and emotional learning (SEL) skills among youth. The tools are short and easy to use. Designed to work together as part of the cycle, they also support impact evaluation.

Management Practices Self Assessment (MPSA).  The MPSA was developed with extensive input from 21st CCLC program project directors and aligns with the core requirements for 21st CCLC programs in Michigan. With 24 indicators forming eight standards in four domains, the tool requires less than two hours for program managers to complete.

Guidance for Out-of-School Time Learning at a Distance (GOLD) Self-Assessment. Site managers and staff complete a self-assessment produced with extensive input from other expert  21st CCLC site managers. GOLD contains 27 indicators that form 11 standards in four domains. The four domains represent point-of-service quality in the individual learning environment.

Adult Rating of Youth Behavior (ARYB).  Each child is rated on the ARYB in November and April. By completing the assessment at two time points (earlier in the school year then again toward the end of the year), the ARYB is able to capture growth in social and emotional skills across the school year. The ARYB has 30 items that form six skill domains, including emotion management, teamwork, and responsibility.

Additional Features:

Cost-Effective.  The Afterschool Evaluation Plan (AEP) 2020 can be adapted to a wide range of cost structures. The Guidebooks and scoring forms are available for free to all users. Additionally, the demands on time are low. Completing the assessments can require as little as 2 hours for program directors and 3 hours for site managers and staff per year.

Guidance Through Online Training.  QTurn will offer live online trainings covering the use of the MPSA, GOLD, and ARYB. Support also includes online training that equip leaders and staff to do data informed planning.

Emphasis on School-Day Alignment.  QTurn’s AEP 2020 helps programs pivot toward greater integration with schools during a time when school has become more challenging for many children.

Support for Impact Evaluation.  Finally, data obtained using the assessment tools can be used to evaluate the overall impact of programs, particularly across multiple programs.

To adopt the AEP 2020, begin by downloading the assessment tools and resources.

For support with implementation of the AEP 2020, please contact the QTurn Team.

Introducing QTurn’s White Paper #1: A Theoretical Framework for Connecting Positive Child Outcomes to Quality Out-of-School Time Programs

In the first white paper released by QTurn, Smith and Peck introduce a theoretical framework for understanding how high-quality out-of-school time (OST) programs lead to positive outcomes for children and youth. The Multilevel Person-in-Context~neuroperson (MPCn) framework outlines the nature and growth of social and emotional learning (SEL) skills for children in OST settings. The framework is also the first component of QTurn’s Quality-Outcomes Design and Methods (Q-ODM) Toolbox.

White Paper #1

Q-ODM Infographic

Inner Processes Driving Outward Behavior

A key feature of the MPCn framework is that it describes how observable behavior is connected to inner mental processes. Understanding the influence and dynamics of past experiences, beliefs, attitudes, goals, and attention, for example, are necessary for explaining how programs promote positive social, emotional, and academic outcomes. Research and evaluation in the OST field has understandably focused on observable behavior. However, a framework that elucidates the inner causes and connections among the mental processes driving outward behavior is needed to guide policy and program improvement. (The authors contrast this approach with positivist theory and methodology. For an overview and critique of positivism, see the reference section at the bottom of the page.

In the MPCn framework, the term “neuroperson” encompasses three types of mental processes that influence children’s behavior in OST programs: schemas, beliefs, and awareness. These are described in the red portions of the figure below.

First, children enter settings with schemas. These are learned, automatic ways of viewing other people and events. Schemas can also be described as concepts or mental frameworks for organizating information. For example, a child may have a schema for “caregiver” that identifies caregivers as adults who are kind and attentive and meet basic needs for things like food and affection.

When something happens that triggers a schema, that schema can influence the child’s ability to regulate their emotions. It also shapes the quality of their attachments to others. For example, if a child’s schema for caregiver is based on past experiences with abusive adults, an adult who raises their voice may activate this schema, evoking anxiety and avoidance. The inclusion of schemas in their framework highlights the need to consider baseline SEL skills (e.g., attachment styles) and prior stress or trauma as children enter a program. Learn more about schemas at VeryWellMind.

Second, children enter OST settings with beliefs about themselves and the world. These combine to form attitudes, goals, and plans and influence how they view and respond to other people and events. Beliefs can change over the course of a year, but they may be formed either automatically or through self-reflection.

Third, how children focus their awareness influences whether they will respond to a situation habitually or skillfully, with intention. The authors use the term awareness to refer to executive functions and describe how awareness interacts with schemas and beliefs.

Settings Within Settings

Another feature of the framework is its recognition of the nested contexts in which children and youth develop (illustrated in the right half of the figure). A child in an OST program interacts with counselors and other front-line staff. Children are directly affected by the high-quality practices delivered by staff. These staff are in turn affected by management practices occuring at the program level. Their job satisfaction, for example, may influence how they perform at the point-of-service. Programs are shaped by the communities in which they operate, and communities are influenced by city, state, and national policies and events. In this way, children are indirectly affected by multiple settings.

In their paper, the authors describe the causal pathways from baseline skills through point-of-service quality to SEL skill growth and longer-term outcomes.

Improving Outcomes for the Most Vulnerable

Of particular interest in QTurn’s work is how SEL skills improve for the most vulnerable children, those who enter programs with lower skills. The framework describes SEL “equity effects” in programs where vulnerable children experience growth in SEL skills similar to or exceeding their more typically developing peers. Equity effects are explored through QTurn’s unique analytical methods, another major component of the Q-ODM Toolbox.

The MPCn framework can help OST programs understand and respond to children who have had adverse life experiences. The framework also emphasizes the role of conscious awareness in shaping behavior. Unconscious schemas and beliefs contribute to unconscious, automatic behavior and difficulty self-regulating. In contrast, the ability to focus awareness contributes to intentional behavior necessary for learning and the pursuit of long-term goals.

Supports and guidance from QTurn are rooted in the MPCn framework. Subsequent white papers build upon this framework, presenting guidance for research and evaluation design, measurement, and analysis.

References

Breen, L. J., & Darlaston-Jones, D. (2010). Moving beyond the enduring dominance of positivism in psychological research: Implications for psychology in Australia. Australian Psychologist, 45, 67–76. doi: 10.1080/00050060903127481. Link: https://researchonline.nd.edu.au/cgi/viewcontent.cgi?referer=https://www.google.com/&httpsredir=1&article=1002&context=arts_conference

Teo T. (2018) The Consequences of “Positivism” in Psychology. In: Outline of Theoretical Psychology. Palgrave Studies in the Theory and History of Psychology. Palgrave Macmillan, London. Link: https://www.researchgate.net/publication/325015126_The_Consequences_of_Positivism_in_Psychology

Houghton, T. (2009). Does positivism really work in the social sciences. Retrieved from
https://www.e-ir.info/2011/09/26/does-positivism-really-%E2%80%98work%E2%80%99-in-the-social-sciences/

Grice, J.W., Barrett, P.T., Cota, L., Felix, C.M., Taylor, Z., Garner, S., Medellin, E., & Vest, A. (2017). Four Bad Habits of Modern Psychologists. Behavioral Sciences, 7. Link: https://pdfs.semanticscholar.org/3406/6ef50009e1d974bca64e235473d4842ce07a.pdf?_ga=2.190158765.789181764.1595443326-236300916.1595443326

Defining High Quality for Afterschool Learning During the COVID-19 Pandemic​

Since the onset of the coronavirus pandemic, as schools grappled with the sudden demand for remote instruction, afterschool and other out-of-school time (OST) programs have been striving to adapt. As programs expand their remote offerings and gradually reopen, there is a need for guidance. Delivering the same high-quality supports to children and youth at home is a formidable challenge, and returning to in-person programming requires adjustment to a new normal. While the details of this new normal are still uncertain, one thing is clear. The coronavirus has not disappeared. Support for longer-term adaptations is needed.

In response, QTurn has released Guidance for Out-of-School Time Learning at a Distance (GOLD), a set of standards and indicators with a corresponding self-assessment rubric. The standards describe promising practices for afterschool learning at a distance that harmonize with existing standards and competencies. The rubric provides leaders with a means for assessing program readiness in order to provide responsive training and technical assistance. Funders and intermediary organizations may also use the standards and assessment to identify system-level supports for achieving high quality.

Standards and Self-Assessment Manual

Self-Assessment Rubric

The GOLD helps program managers and staff adopt a new model for quality at the point-of-service, a point which shifted to the home learning environment and has only recently begun a slow, uneasy shift back. The goal is to help OST programs continue providing high-quality supports to children and families at a time when they are most needed.

Addressing New Pressures on Afterschool Programs

Like others around the world, the afterschool field entered uncharted territory this spring. When schools shut down, many programs closed completely. Some shifted their focus. For example, staff at ourBRIDGE for Kids in Charlotte, North Carolina, delivered packaged meals to school-age children. In other cities, afterschool counselors have connected with children online and offered tutoring, remote activities, and various social and emotional supports. Providers have faced novel demands stemming from family stress, barriers to virtual learning, and a difficult transition back to in-person learning.

Family Stress

Almost two-thirds of parents report shouting, yelling, or screaming at their children at least once in the past two weeks, according to a study by the University of Michigan. More than half say that financial concerns and social isolation have interfered with their parenting. Parents and caregivers have been under tremendous strain. The ubiquitous rise in stress prompted a response from the World Health Organization which released advice for families and a series of COVID-19 parenting posters.

Comments from parents in the University of Michigan study:

“I feel I’ve been somewhat distant even though I’ve been spending more time with the kids than ever.”

“They are confused. They don’t understand fully the dangers of a pandemic. They want to play with their friends and are getting annoyed with the same routine at home, and playing only with each other. We are also rationing food and household items, and they are anxious and scared by that. I can see it in their faces and their volunteering to make sacrifices for us.”

Like afterschool providers, they’ve raced to learn how to keep their children safe and engaged.

On recommendations for staying safe, one afterschool provider said, “Some of our parents can watch it on the news and not understand a word of it. Some of out parents don’t read real well, so if they read about it, they are not understanding it. They need someone to be able to explain to them maybe in different terms. So where it’s understandable for them and explain it to the kids at home.”

Barriers to Virtual Learning

Adding to this stress, not all children have had access to the technology needed to participate in virtual programming. The home learning environment varies considerably from one household to another, exacerbating the digital divide and presenting hurdles to equity. Even if the child has access to a smartphone, the phone may belong to a parent, or they may be competing with other siblings for screentime. Some children also lack the basic materials, like blank paper or corn starch, necessary for engaging in hands-on expanded learning activities at home.

 “I mean, it’s all over the map. Some are using phones if they don’t have a phone, some are using Chromebooks. I know the school did offer the Chromebooks to people but from my experience, so for whatever reason Zoom does not agree with Chromebooks very well. It’s not real fluid. I just I’ve talked to the parents. And they have internet but their connection isn’t fast enough to support Zoom, which is entirely possible. And I’ve had kids get on and then their phones died because Zoom just eats up a whole battery.”

From the program side, converting lessons designed for in-person programming into engaging, remote learning is a daunting task under ordinary circumstances. In a field that emphasizes cooperative learning and group activities, social distancing may seem like the antithesis of the afterschool experience.

 “Because you’re just showing your head on the zoom. Sometimes you can’t really tell what is going on. Or if they are having other issues, and sometimes on zoom, they aren’t going to talk to you about it. But, if you were at school, they would say ‘hey we need to talk’ or you can pull them aside and ask them what’s going on. I think it’s just that that sometimes not physically being able to see their bodies, their body reactions how they react to questions. You don’t see them if you’re being there in person.”

Promoting social and emotional learning in the absence of in-person interactions requires some creative solutions. In the midst of an ongoing crisis, children need the support of caring adults more than ever.

“I have one little girl who will call me every day, and I pick up, because what else am I doing. And she’ll tell me about the bird she saw on the walk, or a drawing she did, or a video. And she just loves to check in and say hi and then I see her on the zoom calls too.”

The Road Back

When children do return to programs in person, the need for new safety measures, policies, and practices have resulted in a rough road back. Due to rising infection rates, schools are considering the possibility of reopening with a hybrid model that combines in-person and remote learning.

“Simply re-creating the pre-shutdown norm will be neither possible nor desirable.”
 – Erika Christakis, For Schools, the List of Obstacles Grows and Grows, The Atlantic

Sustaining Positive Outcomes for Children and Youth

Under ordinary circumstances, to have positive social, emotional, and academic benefits for children and youth, afterschool programs must be of high quality. Lower-quality programs generally do not lead to positive outcomes. For this reason, quality standards have been a guide for programs across the country since 1998 when the National Afterschool Association developed the Standards for Quality School-Age Care. States have continued to adopt and improve quality standards to meet changing needs in their communities.

Existing standards cover safety and health, relationships, youth engagement, equity, and access. While rooted in principles of positive youth development that apply no matter where learning takes place, existing standards developed for in-person programming do not immediately translate to distance learning. Existing standards illustrate what high-quality afterschool learning looks like… under ordinary circumstances.

But what does quality afterschool learning at a distance look like? Knowing the importance of high quality for supporting youth development, can high quality continue during a nationwide quarantine and transition back to in-person services? How has the role of afterschool staff changed? How can programs adapt?

To answer these questions, the Genesee Intermediate School District (GISD) Bridges to Success team partnered with QTurn. Led by Charles Smith, an expert on afterschool quality with more than 20 years of experience in the field, QTurn interviewed 15 afterschool staff from nine sites as well as an administrator from the GISD office. Throughout each step of the development process, expert practitioners provided extensive input and feedback.

Summary of the Guidance

Guidance for Out-od-School Time Learning at a Distance addresses social, emotional, and academic learning and takes a whole-child approach. It includes 10 standards and 27 indicators. The self-assessment can be completed in less than two hours.

QTurn identified four domains of quality for OST programming at a distance. The 10 quality standards fall into these domains, which address the new pressures on afterschool programs to connect with children and provide content in new environments.

  1. Family- and Caregiver-Centered Engagement
  2. Individual Learning Environment
  3. Distance Programming
  4. Planning with Children, Families, Caregivers, and Schools

The first domain, Family- and Caregiver-Centered Engagement, outlines new roles for afterschool staff as advocates and resources for families. In addition to helping families and caregivers understand critical information for thriving in quarantine, a trauma-informed, strengths-based approach to family engagement is recommended.

The second domain, Individual Learning Environment, addresses barriers to virtual learning. Programs are encouraged to use a variety of methods, technologies, times, and languages to connect with families. Supporting virtual learning means helping families create a workspace for children, equipping students with educational supplies, and coaching families on the use of key technologies or apps.

The third domain, Distance Programming, identifies strategies for continued social and emotional learning (SEL) despite the disconnect inherent in video conferencing. Practices that support staff wellness also contribute to their ability to model SEL skills for children and complete social and emotional check-ins. Included in this domain is an emphasis on school-day alignment, collaborative leadership, and partnerships between program and school day staff.

Finally, Guidance offers support for the transition back to in-person learning through the fourth domain, Planning with Children, Families, Caregivers, and Schools. Planning must include strategies for following social distancing guidelines as well as social and emotional supports for students’ well-being as they resume learning.

The rubric, which corresponds to the standards and indicators encompassed by the above domains, provides programs with the means to assess readiness. After completing the self-assessment, program staff are encouraged to decide which indicators to focus on and set action steps for improving practice or updating policy.

Coming Back Stronger

Afterschool programs and intermediary organizations have endured a period of intense uncertainty around the future of afterschool. Programs throughout the country are in different stages of reopening, but shutdowns have left a permanent mark on the afterschool field. Having witnessed the widespread interruption of a whole sector of youth services, many afterschool programs and staff may never fully return to “business as usual.”

The upheaval of the past few months, however, presents opportunities to build resilience within the field. When children and youth are most in need of the supports unique to afterschool, programs will be equipped to respond.