Compassion has a lot of definitions, but most have to do with recognition of suffering, action to alleviate suffering, and tolerance of discomfort during the action.[i] By April of 2020, we knew that our afterschool partners in Genesee County (including the city of Flint) Michigan, and many of the children and families that they served, were suffering. A significantly higher proportion than usual of those families were in a crisis-mode. For afterschool educators, the learning environment had moved, and the means of delivering programs had changed dramatically. A “pivot” was required.
When our partners told us how evaluation could help, they emphasized a compassionate approach to the work that would address suffering in multiple ways: by reducing workload related to evaluation, by providing an evaluation design that was of timely value in the current moment of challenge, and by wherever possible reflecting back to staff their own incredible commitment and ingenuity in meeting those challenges.
We translated this desire for an experience of compassion into a few rules about method:
Rule 1 was make it quick. We knew that staff were in crisis mode and that time was precious. We eliminated all data collection responsibilities for staff. Staff had only to schedule dates for observers, sit for a 45-minute interview, review the report during a 70-minute training, and then review the report again during a subsequent 15 minute portion of an all-staff meeting. This meant less than 4-hours of total time for a site coordinator engaging in required evaluation activities between September and December 2021.
Rule 2 was prioritize local expertise. When program practices and objectives are changing rapidly (the pivot mentioned above), prior evaluation designs (including measures) are of reduced validity. This was true simply because it was no longer the same service – models varied widely both within and across programs.[ii] We asked open-ended questions about what was and wasn’t working and then coded text segments to existing program standards for program fidelity, instructional quality, and students’ socio-emotional skills. In this way, we identified standards that were applicable in the new situation, named new priorities in those terms, and respected site managers as expert sources of data about what works.
Rule 3 was ask about what is changing (and reflect strengths). Afterschool staff told us they felt like their professional tools became outdated overnight, by the pandemic, and it was not a good feeling. We spent our moments of access to leaders and site managers asking how it was going, letting them give voice to however it was going by taking the conversation wherever it went, and then by intentionally reflecting strengths back to them. Although this therapeutic aspect of our service may feel a bit uncomfortable to some evaluators, the situation required it as an aspect of method. As evaluators, we were “giving value to” leaders’ and site managers’ experiences by letting them flow some ideas and emotion while answering our questions.
Rule 4 was write it down. By asking staff about practices and coding their transcribed responses into categories, we were identifying sentences written by program staff that describe specific local best practices in specific local terms. By identifying and writing down local best practices in the words of program staff, the evaluator helps speed up the development of shared mental models about what the pivoted service is. This helps service providers demonstrate accountability in the sense of “this is actually what we did every day.” It also makes it possible for leaders to pivot the service more easily in the future by returning to documentation for crisis- or emergency-management.
[i] Strauss, C., Lever Taylor, B., Gu, J., Kuyken, W., Baer, R., Jones, F., & Cavanagh, K. (2016). What is compassion and how can we measure it? A review of definitions and measures. Clinical Psychology Review, 47, 15–27. https://doi-org.proxy.lib.umich.edu/10.1016/j.cpr.2016.05.004
[ii] Roy, L. & Smith, C. (2021). YouthQuest Interim Evaluation Report. [Grantee Evaluation Report]. QTurn. and Smith, C. & Roy, L. (2021). Best Practices for Afterschool Learning at a Distance: GISD Bridges to Success. [Grantee Evaluation Report]. QTurn.