Month: August 2021

In early 2020, COVID-19 rates were soaring. Masks, cleaning supplies, and clear information were in short supply. This was especially true for schools across the country. Teachers, parents, and students were unsure about what was going to happen next. On Thursday, April 1, 2020 (in-person) school was still in session in the Genesee Intermediate School District (GISD) and the students and staff of the 21^st CCLC Brides to Success afterschool programs were looking forward to a 4-day weekend. On Friday, April 2, Governor Whitmer released executive order 2020-35, immediately suspending school for the remainder of the school year and drastically changing how delivery of school and afterschool services would be provided (e.g., shifting from in-person to remote interactions with children and youth).

This was the second year QTurn partnered with GISD’s Bridges to Success programs, and we quickly realized two things. First, our original continuous quality improvement (CQI) cycle (with a heavy focus on Socio-Emotional Learning) was, in spirit, more relevant than ever but, in implementation, completely inappropriate. Second, the setting-based tools (such as the Youth and School-Age PQA) available to evaluators in the OST field in April of 2020 could not provide valid program quality data and could not demonstrate how afterschool programs, like Bridges to Success, were pivoting to meet the needs of children.

During this pivot point, we discovered the four “rules” for designing and implementing a compassionate evaluation. It was clear that it felt unethical (to us and to the Bridges to success leadership) to ask staff to partake in an external evaluation processes ill-fit for the transitional physical setting or within the greater context of learning during a global pandemic. Our partners didn’t need the added pressure of an external observation while they were still figuring out what it meant to offer virtual and non-virtual programming to students and families.

By the time schools were closed in Michigan, QTurn was in the midst of developing a self-assessment tool for evaluating program quality during the COVID pandemic, which would eventually become the Guidance for OST Learning at a Distance (GOLD). The development of the GOLD, funded by the Michigan Afterschool Association (MAA) and the Michigan Department of Education (MDE), was the culmination of interviews, workshops and reviews with over 25 youth development and OST experts from the State of Michigan. Because the GOLD materials were not released for another month to the public, the QTurn team and Bridges to Success leadership decided to use the 27 best practices described in the GOLD as a framework for coding a series of interviews in order to tell the story of Bridges to Success’ response to the pandemic.

Over the course of two weeks, the QTurn team interviewed 15 afterschool staff, from 9 GISD afterschool sites, and 1 administrator from the GISD office. Each interview was structured around the following five questions:

• What is the experience of transitioning from in-person to distance programming?
• What are you hearing from students and families?
• What are the barriers to students’ virtual learning?
• Where are you experiencing success?
• Where could you be successful with more support?

Some calls were quick, lasting only 35 minutes. Some were over an hour. We asked questions, and we listened, asked follow up questions, and listened more. Every conversation was an intense opportunity for direct staff, program administrators, and team leads to tell someone outside of their world what was going on. We heard many sentiments filled with hope and gratitude, confusion and uncertainty of their impact, moments of fear and sadness, and overwhelming concern for the students and families in their programs.

Although each of the afterschool sites used their own approach to providing services intended to facilitate learning at a distance, with no systematic coordination across sites, five key themes emerged from our analysis of the interview responses:

1. Staff were unsure about how to define some aspects of program quality and/or professional practice within the context of learning at a distance; particularly, how to most effectively monitor children’s (a) socio-emotional well-being, (b) academic effort and progress, and (c) attendance.
2. GISD Bridges to Success leadership style and organizational culture were important sources of support for staff experiencing programmatic uncertainty and professional disequilibrium.
3. Learning at a distance both exacerbates and clarifies inequities. GISD responded by providing a diverse set of programming options, such as: (a) virtual communication and supports, (b) non-virtual communication and supports, and (c) supports for adults supporting younger children.
4. GISD continued to deliver a whole-child curriculum that provided supports for safety, fun, academic work, and socio-emotional skill building.
5. Increased flexibility of staff schedules was necessary to meet student and family needs, even though it often increased the length of staff workdays.

When we started the 2020-2021 school year, we realized again that the setting-based assessments required by the Michigan Department of Education were going to once again offer incomplete data on the impact of programming being offered by Bridges to Success. The afterschool model was no longer face-to-face general enrichment programming. Bridges to Success was still responding to families in crisis, so their approach was a casework model focused on socio-emotional and academic support. Knowing this, we conducted external (virtual) program evaluations, using the MDE recommended PQA items plus two scales from the SEL PQA (emotion management and empathy), But, the PQA data alone felt incomplete. To supplement, we again interviewed site leads and coded the transcripts to the GOLD. By interviewing site leads based on general questions, and letting them talk, we were able to learn about not only their experiences but also what they were most concerned with and focused on. The GOLD demonstrated what the PQA alone could not: that lots of SEL programming was being done but mostly one-on-one with children and families, outside of their regularly-scheduled virtual programming

In early spring, 2021, QTurn and Bridges to Success staff came back together to decide how we could work together for the rest of the year. Schools were opening again, and in-person afterschool programming would be offered again. We decided that would do external observations using the SEL PQA and use a custom distance-learning external assessment tool that was designed specifically for GISD.

By adding the GOLD into our CQI plan, we were able to really define the quality and breadth of services. No two sites were operating the exact same way – but every site was working with their school to meet the needs of children. Utilization of setting-based assessment tools not designed for virtual learning (or non-virtual distanced learning) was only scratching the surface of what programs like Bridges to Success accomplished on 2020-21. And by working with our partners and by centering compassion, our evaluation not only articulated, but honored the heroic effort and continuous dedication of the Bridges to Success program to their communities during a difficult year.

Compassion has a lot of definitions, but most have to do with recognition of suffering, action to alleviate suffering, and tolerance of discomfort during the action.[i] By April of 2020, we knew that our afterschool partners in Genesee County (including the city of Flint) Michigan, and many of the children and families that they served, were suffering. A significantly higher proportion than usual of those families were in a crisis-mode. For afterschool educators, the learning environment had moved, and the means of delivering programs had changed dramatically. A “pivot” was required.

When our partners told us how evaluation could help, they emphasized a compassionate approach to the work that would address suffering in multiple ways: by reducing workload related to evaluation, by providing an evaluation design that was of timely value in the current moment of challenge, and by wherever possible reflecting back to staff their own incredible commitment and ingenuity in meeting those challenges.

We translated this desire for an experience of compassion into a few rules about method:

Rule 1 was make it quick. We knew that staff were in crisis mode and that time was precious. We eliminated all data collection responsibilities for staff. Staff had only to schedule dates for observers, sit for a 45-minute interview, review the report during a 70-minute training, and then review the report again during a subsequent 15 minute portion of an all-staff meeting. This meant less than 4-hours of total time for a site coordinator engaging in required evaluation activities between September and December 2021.

Rule 2 was prioritize local expertise. When program practices and objectives are changing rapidly (the pivot mentioned above), prior evaluation designs (including measures) are of reduced validity. This was true simply because it was no longer the same service – models varied widely both within and across programs.[ii] We asked open-ended questions about what was and wasn’t working and then coded text segments to existing program standards for program fidelity, instructional quality, and students’ socio-emotional skills. In this way, we identified standards that were applicable in the new situation, named new priorities in those terms, and respected site managers as expert sources of data about what works.

Rule 3 was ask about what is changing (and reflect strengths). Afterschool staff told us they felt like their professional tools became outdated overnight, by the pandemic, and it was not a good feeling. We spent our moments of access to leaders and site managers asking how it was going, letting them give voice to however it was going by taking the conversation wherever it went, and then by intentionally reflecting strengths back to them. Although this therapeutic aspect of our service may feel a bit uncomfortable to some evaluators, the situation required it as an aspect of method. As evaluators, we were “giving value to” leaders’ and site managers’ experiences by letting them flow some ideas and emotion while answering our questions.

Rule 4 was write it down. By asking staff about practices and coding their transcribed responses into categories, we were identifying sentences written by program staff that describe specific local best practices in specific local terms. By identifying and writing down local best practices in the words of program staff, the evaluator helps speed up the development of shared mental models about what the pivoted service is. This helps service providers demonstrate accountability in the sense of “this is actually what we did every day.” It also makes it possible for leaders to pivot the service more easily in the future by returning to documentation for crisis- or emergency-management.

---

[i] Strauss, C., Lever Taylor, B., Gu, J., Kuyken, W., Baer, R., Jones, F., & Cavanagh, K. (2016). What is compassion and how can we measure it? A review of definitions and measures. Clinical Psychology Review, 47, 15–27. https://doi-org.proxy.lib.umich.edu/10.1016/j.cpr.2016.05.004

[ii] Roy, L. & Smith, C. (2021). YouthQuest Interim Evaluation Report. [Grantee Evaluation Report]. QTurn. and Smith, C. & Roy, L. (2021). Best Practices for Afterschool Learning at a Distance: GISD Bridges to Success. [Grantee Evaluation Report]. QTurn.