Fall 2022 Evolving the Standard for High Quality

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email
3,191 words | 21 min

In the prior two blogs, we talked about how a group of master afterschool teachers at 25 afterschool sites in Genesee County have responded to the pandemic years. One important area of evolution was the new service models that put staff in the field visiting homes and meeting parents at drop off sites, producing a wide variety of online activity content, taking activities outside and, for the first time, implementing a whole new regime of personally protective equipment (PPE) and cohorting practices. This was a lot of new stuff! 

The other area of innovation was in the practices that were delivered inside of afterschool programs during the return to in-person schooling for the 2021-2022 and 2022-2023 school years. This was a finding from the 2021-2022 end-of-year evaluation reports for YouthQuest:  

Finding 3. YouthQuest has a compelling afterschool impact story about mental health prevention…in most cases staff teams achieved a “very-low or no-error rate” – there were no moments of missed opportunity. Every challenge was noticed by adults and almost all were addressed in the moment…. Every comment from a child merited an authentic response from an adult, and every adult continuously set and reinforced high expectations for socio-emotional skills.  [See report for YouthQuest]  

There was a similar finding in the Bridges end-of-year report. And we’re not kidding about every instance: In our data, intentional co-regulatory events were happening at a rate of 10 per hour per staff person. The master afterschool teachers at these sites were standing up their best practices and skills to respond to an increasing proportion of stressed children. 

AATI Practices 

One of the key insights of the trauma-informed movement is to not mistake “misbehavior” (i.e., children not participating in the desired way) for children’s intentions. Children and youth who cannot keep their bodies in the activity space, who are frequently very worried about others, who simply can’t focus are likely enacting unconscious behavioral scripts that come from earlier ages and more stressful circumstances.  

This means that, when children have been triggered in some way, they are projecting past-relevant emotions onto the current moment, and they are also acting out the behavior scripts that went with those past emotions. It is not because they want to or choose to. They don’t think about it. Its automatic. 

We also use the term attachment-aware, which means being aware of the attachment skills that children bring into afterschool settings. Adult and youth attachment skills are what make up relationships between parent and child or teacher and child. In afterschool settings, the teacher brings the warmth and intent to care skills, and the child brings their prior experience at relationships with their own primary caregivers. It’s tough for an adult who is unknown to the child to effectively jump in to “have relationship,” or what we would call “co-regulate,” with children who are in the middle of acting out an energized script from an earlier time. It requires trust to help children change the focus of their awareness in that moment. Being “attachment aware” makes it possible to do the trauma-informed work of co-regulation with a “very-low or no-error rate.” 

The formula is this: Adults are able to help children interfere with cycles of reactivity to the extent that a trusting relationship is achieved between the child and adult. In the next several sections, we tell four stories: Two about building children’s attachment skills and two about how teachers use their relationships to help students interfere with reactivity and integrate uncomfortable thoughts and feelings.  

Attunement  

One best practice that we observed across networks was staff who were prepared to completely change expectations for the activity depending on where the children were at emotionally. The priority was to reach a calm and respectful connection between the adult and each student. A great example was Keke, one of the most “Zen” youth workers I’ve ever observed, working with a group of eight 4th and 5th grade students to get started on an activity about storytelling. What’s important is Keke’s decision that attunement had to come first. 

When, after 15 minutes of trying to get started, Keke realized that for perhaps half of the children there was simply no way to settle down and start the activity. It was time to go to the floor mat and get a little bit more physical with a game of hot potato spelling words. This kept things in the general area of literacy but lowered the task demands – storytelling was too hard in that moment. 

This process was a struggle too, but it allowed children to interact with a warm adult and be able to fidget and laugh (sitting on the floor together in a circle) while that adult was warmly but unflinchingly asking each child what’s going on for them and asking the rest of the children to respectfully witness the statements and feelings of the child speaking.  

This process took about 40 minutes and, by the end, several emotionally energized topics had been discussed (e.g., having enough money to buy bracelets at school, why not to physically attack someone who has mocked you, using hand sanitizer to feel safe from COVID germs while passing the hot potato, how to comply with any of several community agreements). Perhaps most importantly, one student who rarely pays compliments paid a compliment to another and the group, and all thought this was significant.  

Everyone was more settled and feeling more connected by the end of hot potato spelling words. The storytelling never happened, a few spelling words were spelled, but most of the eight children had the warm experience of being the focus of an adult who genuinely wanted to know what they were thinking and feeling in that moment. 

Collecting Children  

Attunement is the process of an adult and child sharing eye contact and the adult reading the child’s emotional state and either mirroring what we call the broaden and build skill state or responding to a narrow and constrain skill state. In either case, attunement requires proximity to children who are willing to attune. At Brandon Nuby’s Holmes STEM Academy, collecting children operates as the state of the art – and uses an incentive/rewards system as the foundation of the practice.  

To set the stage: Mr. Nuby had a group of perhaps 20 middle school aged newcomers to his program in fall 2022, particularly a large group of 7th grade boys. When I observed this group in fall 2022, the program had some moderate chaos, what some would call a tough group. However, because Mr. Nuby understands these 7th graders as little replicas of his own child self, he responds calmly with constant love and structure.  

Part of that structure is the YouthQuest Bucks program – a way of earning bucks that can be used to buy things in the YQ Bucks Store and during amazing field trips that are a core part of the YouthQuest model. As Mr. Nuby stated in our fall 2022 interview, “I’ve got these kids coming to the program with YouthQuest bucks – we’ve been at our attendance limit almost every day, and I’ve been working on them.” 

The socio-emotional skill growth, from year to year, was in evidence to me: A youth basketball game in the gym required constant supervision and frequently devolved into argument in fall 2021; that same group of students was able to self-organize into a 10-minute basketball game with few disruptions in spring 2022; by fall 2022, that same basketball game lasted 25 minutes and did not require a single adult support.  

While it’s true that behavioral reward systems are effective for issues of basic regulation, like coming to the program every day, these systems are not what makes the basketball games get successively more well organized. This is the point of telling Mr. Nuby’s story. Mr. Nuby gets the kids attention with the YQ bucks – but it’s the attunement conversation that accompanies the exchange of bucks that matters for more advanced socio-emotional skill building. With every buck comes a conversation with Mr. Nuby or a member of this team who is handing out the bucks. Often, the staff are walking around with a handful of bucks at entry, and at other times during the program, collecting children.  

Every student gets bucks at entry for coming each day, and each transaction involves a look in the eyes and a conversation about their feelings and how the day can go – a moment of emotional attunement for every student. The students, for their part, want to be collected, and look forward to the ritual of earning the YQ bucks as you can tell from the quote below (a composite from two eighth grade and one sixth grade student) 

Most like the bucks because they have a lot of them stacked up. It’s very reasonable in how much we get each day; if you act up, your pocket gets hit. It’s like a good idea to get students to act right cuz, at the end, they get to go to a store to buy stuff… actual things that you would buy from a store…. If you have been acting up, he’ll tell you he is disappointed that you are going to get your money taken… and then, if you pick up the trays, he says thank you and good job. I’ve still got money from prior years at youth quest. 

Mr. Brandon Nuby and Students at YQ Bucks Store, Brownell Elementary, Flint, Michigan 

1-on-1 Nurturance 

In the parenting literature, there are two terms that capture two important types of responsive adult behavior when parenting: synchrony and nurturance. Synchrony is when parents use encouragement to help children quickly generate positive emotions so they can return to the task, right in the moment (e.g., a child’s attention wanders, and an adult is there to bring it immediately back).  

Nurturance is when parents use encouragement to help children downregulate negative emotion. When a parent decides that their child can’t keep going on the task, and child needs to stop for a check-in, it’s referred to as an adult’s act of nurturance. Nurturance, in this case, means making it 1-to-1 attunement for a moment and changing the child’s attention by modeling a broaden and build skill state and/or changing the circumstance in some way. These acts of co-regulation break the connection between the current circumstance (the trigger) and the child’s wandering attention, boredom, or overflowing emotion. 

The 1-on-1 moment of nurturance was used most skillfully and intentionally in Fall 2022 at Brownell Elementary, where Miranda Manns has been site team leader the for the past 5 years. Miranda is a master afterschool teacher with extensive experience working in the positive youth development and trauma-informed approaches during her professional journey from parent volunteer to afterschool staff to program manager. When asked about the children who were struggling to manage their own bodies and emotions, the kids with their narrow and constrain skill engaged, Miranda replied:  

We’re trying to get them to manage emotions and reminding them of things they had used in the past that worked when they were sad… and just always letting them know it’s okay to have negative feelings as well as negative attitudes but we have to manage them… something may have happened during the school day and they are still ruminating on it when they come in.  

So, what happens at the intersection of stressed children in emotional overflow and staff who understand the attachment and trauma science? The first thing is completely consistent warmth and emotional containment from the staff. It didn’t matter if a single child was having a moment of meltdown, or if a group of children were having a moment of moderate chaos, the staff contained any reactivity they may have felt and continued to express high expectations with calm and caring body language and tone of voice.  

The second thing that happened was lots of 1-on-1s where adults gave their mostly undivided attention to a child experiencing emotional reactivity and behavioral impulse. During our three-hour observation, nearly fifteen 1-on-1s were required by the five members of the staff team. Because our observation protocol included follow up questions, we know those 1-on-1s guided children toward mindfulness about their emotions because staff asked children what was happening for them in that moment, how they were feeling, and why. 

Brownell also has a space for when the 1-on-1 needs a next step: the blue couch in the staff office. Miranda says:  

“It’s almost like a healer or something, but once they sit on that couch…  I’m not sure if its because I offered it or because they are actually able to get mindful and be at peace. The blue couch is the healer in this place, and I ask them if they need the famous blue couch or if they want to talk to Miss Manns about it. 

Miranda Manns and Staff at Brownell Elementary, Flint, Michigan. 

Mindful Tools  

As we saw with the 1-on-1s and blue couch at Brownell, staff were guiding children in distress toward mindful responses to their own overflowing or uncomfortable feelings and behaviors. Mindfulness is all about being aware of some specific things in the present moment. This also means not thinking about the past or future – which is what people in our society spend most of their time doing – and why it’s good to spend time building your present-focusing muscles.  

Mindfulness interventions are suggested for children and adults with multiple ACEs because survivors are more likely to be triggered into narrow and constrain mental skill states (e.g., invasive thoughts, hypervigilance, avoiding people/places/topics, hyperarousal) that accompany the familiar behaviors of withdrawal, frustration, and disruption of self/others. Children’s capacity for openness, curiosity, and play are “hijacked” by an activated narrow and constrain skill state. 

There are a few general mindfulness practices that are frequently recommended to manage narrow and constrain skill states in education settings. The first practice is focusing awareness for the purpose of building your present-focusing muscles by, for example, practicing mindfulness exercises as a group in afterschool programs. The second important mindfulness practice is learning how to focus on feelings and sensations inside of the body. This ability to focus on internal sensations is also called interoceptive awareness. These two mindfulness practices have lots of application in educational settings. 

Danielle, at Pierce Elementary School, uses mindfulness practices weekly with students. Danielle’s explicit goal is to teach interoceptive skills that staff and children can use during the 1-on1 moments of nurturance, right after behavior has gone off track and the adult approaches. But the muscle building part is the preparation for use, and this happens with the whole group. When the interoceptive and other strategies are taught and practiced with all of the children, the tools become part of the culture and can be reinforced by peers as well.  

Here is a snippet of what it sounds like when Danielle is doing the whole group mindfulness work at Pierce. In this case, children are spread out on the gym floor, and they have just listened to some very chill music and a guided meditation about how to become an observer of your own emotional states using the analogy of a feelings-fish swimming in your own brain-body pond: 

  • Teacher States: …Feelings are fish – the fish are your feelings that can be angry, sleepy, happy, nervous… Your job is to be the pond. You are the pond, and you watch the feelings swim around. You can just notice those feelings. Can you be the pond? What do you notice?  
  • Sometimes we feel like we are angry – like the fish. But we are not the fish. We are the pond. So be the pond. When you feel like you are only an angry fish – remind yourself that you are not an angry fish; you are the pond. You are not your angry feelings; you are just noticing them. 
  • Teacher Question: So does anyone have any thoughts? Student Responses: I liked breathing in and breathing out (and other students echo). I liked that I got to lay down. I liked the fish

This is subtle and powerful work. During the guided meditation, the children were quiet and guided to a mindful state by a story about a feelings-fish. The mindful state was a present-focused one, with students noticing their own feelings and seeing them as fish in their brain-body pond. This is direct practice at integrating charged emotional energy before it gets stronger – even before thinking about why I feel that way, I can try put myself in the position of observer rather than reactor. Another key point is that children don’t have to fully understand the process to get the benefits. 

While practicing and building mindful muscles is a weekly part of program, its during the 1-on-1s that Danielle sees the tools and practice pay off. Specifically, when students have emotion/behavior challenges that fall outside of community agreements, students are given a 1-on-1 moment and a paper form that structures a process of “think and reflect” about “big emotions.” Daniel described an example of how it worked with one student this year, when… 

… one day instead of following the same behavioral pattern of not regulating emotions, taking it out on others, throwing tantrums when upset, she began to self-regulate using the outline she had worked through so many times using the think sheets. This last time she became upset she asked to come to  my office so that she could gather her thoughts to go have a constructive conversation with the other student… She then politely asked them for her things back and made it a point to ask the other student not to repeat the behavior, and instead, that if they asked nicely she would lend it to them. 

The Future of Afterschool 

The field of afterschool has evolved quickly since March 2020. Some new practices and possibilities developed in response to the pandemic yeras, from the pressing of resources at hand into the shape of urgent new needs. These new practices may spell out the future of afterschool services and the role of afterschool teachers in the post-pandemic period. As we noted in the first blog, the pandemic era saw the development of new service models that put afterschool teachers into new, externally-facing roles and relationships with children and families, depending on need.  

But the innovation also happened on the inside. New practices from the attachment aware, trauma-informed perspective also came into more frequent and intentional use. While staff at most of the sites are now using many variations on these practices (e.g., heightened warmth, mindfulness techniques, proactive check-ins, tactile and safe space options, etc.), we in afterschool lack a common standard for responding to children with post-traumatic stress symptoms and/or developmental delays. These are practices that afterschool may want to add to quality standards, particularly during and after exposure to traumatic events and particularly for communities where exposure to traumatic events is more frequent or intensive (e.g., gun violence, climate disaster). 

The State of Children in Afterschool

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email
1,581 words | 11 min

Like all managers I interviewed last fall, Keoshia, site manager for YouthQuest at Durant-Tuuri-Mott Elementary School, agreed that over 50% of this year’s students were behind academically. For that reason, the team at DTM Elementary afterschool is focused on math and literacy. However, for Keoshia and her team at DTM, the children’s socio-emotional skills are about the same as usual. This year was about picking up where last year left off in terms of socio-emotional skills – and then staying on those academics. 

Indeed, when you step into the lunchroom with DTM’s 60 students and 7 staff, it’s hard to tell the difference from prior years. The kids look happy, they typically respond to the first request for quiet, and they reciprocate the parent-like warmth and care that DTM staff provide. This is a long-standing program with connections to families, a tight staff team (an average of 6 years of experience), and by the time of our October 2023 observation, already had a working program culture in place that was right where they wanted it. 

Caption: Keoshia Cook with Durant-Tuuri-Mott Staff just Before dismissal. 

Another program that, like DTM, received one of the highest quality scores this past fall was Mt. Morris High School, where site manager Zach T. sees the same pattern with their group of almost 30 adolescents. Everyone is struggling a bit on the academics, but their tight program culture produced a near two-hour roleplaying game that was the highlight of youth leadership in our fall round of data collection. For both of these programs, the story is that the pandemic was tough, but we made it through. The program culture is strong, and the young people and families keep coming back. 

However, these two exemplary programs “picking up where we left off last year” were neither the exception nor the rule. Program cultures went in a very different direction at about 25% of the program sites we observed. Again, we’ve observed all of these sites for four years, and everything has been exemplary…. What was going on for these 25% of programs in fall 2022? 

A Post-Pandemic Shock 

For the 25% of programs in fall 2022, a much higher percentage of children entered with severe developmental and academic delays. The general wisdom was that these were children whose parents withheld them from participation in school and afterschool during the prior year – in some cases having gone two, or in some cases three, years without participating in-person in an educational setting.  

From the observations we completed in October and November 2023, two things were true for me as an observer: (a) There was a substantially increased proportion of children with socio-emotional and academic skill delays compared to prior years, and (b) for many of those struggling children, the socio-emotional and academic skill delays were more severe than I’ve ever seen in over two decades of observation.  

Even with expert staff giving it their all, a couple of unsustainable things were happening: The setting was reaching moderate chaos numerous times during the session. Staff were constantly engaged in 1-on-1s with children struggling emotionally and behaviorally… and the other students, who were also struggling to keep it together, were more frequently joining the fray than they otherwise might have. Second, staff were spending a lot of energy in response to these repeated emotional and behavioral events. In our opinion, something had to change, or children were going to get seriously injured and/or really talented staff were going to quit.  

This is an excerpt from a discussion memo that we, as a group, presented when agency staff at the state department of education asked what we were seeing in the field: 

When student behaviors require immediate adult responses, that student is likely to be in what we refer to as a “narrow and constrain” mental skill state, a socio-emotional skill set that is largely non-conscious, endures through time, and severely limits mental and behavioral options for the child.  

The pinch point for afterschool standards is that the appropriate response to these children requires one-on-one structure. Afterschool programs are not typically staffed to provide these 1-on-1 responses for more than 5% of children at any given moment. This fall (2022), the proportion of children in narrow and constrain skill states was as high as 40% at some sites. We estimate that anything over 10% is associated with light chaos, and anything over 20% brings moderate or greater chaos. Further, in the 20% or more condition, safety is difficult to assure beyond a ratio of 1 to 6. 

Specific indicators of narrow and constrain skill states (i.e., low socio-emotional skill levels) were of four types: 

  • Frequently observed post-traumatic stress symptoms (PTS; subclinical): Emotional overflow, extreme clinginess, hyperarousal (can’t keep body in activity space), hypervigilance (can’t stop worrying about others), negative self-talk, violent and/or sexual talk and pretend, verbal bullying, self-silencing. These behaviors typically lasted for most or all of the session for individual children. 
  • Expressive language delays: Children who understand adults and peers but have a hard time communicating back; using non-verbal vocalizations to communicate (growling when angry). These children are attractors for verbal bullying by peers and are often understandably frustrated when trying to communicate.
  • Lowered academic efficacy: On average, staff estimate that over 50% of children at their sites are behind where their teachers want them to be right now and likely not getting positive messages in school. 
  • Staff exposure to traumatic experience: On average, 85% of site staff report invasive thoughts related to child and family experiences this year, reflecting the severity and frequency of needs/hardships that afterschool teachers are encountering. 

Responsive Policy and Programs 

Program directors were in a dilemma at the sites where higher proportions of  children were presenting with severe socio-emotional skill delays. The “moments of moderate chaos” programs were either going to have to (a) exclude the kids who can’t self-regulate sufficiently, to assure safety for all students in the setting, or (b) relax attendance requirements and allow lower youth-to-adult ratios where necessary. In an example of evidence-driven and compassionate leadership by our state agency, requirements were relaxed with guidance to continue focusing on the most in need.  

Enlighted leadership also came from sites. As annual improvement goals, site managers (independently of each other) chose nearly identical areas of focus: First, for children in narrow and constrain skill states, the focus was on helping those children learn and practice strategies that support “return” to a calm and open state of mind and body – what we call a “broaden and build” skill state. Not consequences, not exclusion, but strategies to be more emotionally resilient and mindful in moments of reactivity when strong emotions and behavior are flowing. 

Second, and for the rest of the children, the goals were focused on helping “bystanders” intentionally exemplify and promote the program culture and norms (e.g., written community agreements, written program values). Both networks were planning more effort to communicate the desired program culture and values, i.e., the program’s purpose, what should be happening in the setting, and what the student’s role is. 

Pre-pandemic, we could count on program cultures to replicate program norms year to year with returning students. Following the pandemic, these bets may be off for a few years.  

Attachment-Aware, Trauma-Informed (AATI) in Afterschool 

This was the state of things in fall 2022. In mid-January, 2023, we asked all 25 site directors the following question: How are the narrow and constrain state kids doing? Are they quickly catching up on those socio-emotional skills for keeping bodies in place, maintaining focus, using our words etc.? Or is this going to be a long haul for this group, with the “catch up” or recovery phase lasting into subsequent years. To a person, all agreed that for many of these children it will be a multiple year effort. Several reported that things had not really changed much developmentally since late October for some children. 

What does it mean? One thing that it means is that the converging stressors during the pandemic years – amped up “social determinants of health” for many families – have produced an interruption in socio-emotional skill development for a substantial enough group of children that we may need to do some things differently, at least for the next few years. Note that as a confirmation of traumatic events happening with children, we asked staff if they were worrying about child and family hardships. About 85% said yes to this indicator of vicarious trauma. 

Another sign of interruption was the subject of technical assistance that emerged from entry interviews at the 25 sites during the fall 2022 round of observations. This is another excerpt from the memo for our state DOE:  

Finally, as part of the feedback process to individual sites this year, we’ve addressed best practices for: Verbal bullies and victim self-confidence, addressing expressive language delays, addressing negative self-talk in the moment, mindful responsiveness in the moment, coldness is the opposite of warmth and has faster effects, swearing and sexual language, violent video games, how to do on-on-ones with a child in a narrow and constrain skill state, how to do a behavioral agreement with an individual student, etc.  

These are not the same kinds of requests as in prior years – and reflect the reality of what afterschool teachers have to be able to engage with if they are to meet the goal of socio-emotional skill growth

Afterschool Excellence During Pandemic Years

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email
1,680 words | 11 min

The field of afterschool program services has evolved quickly since March 2020, from the pressing of resources at hand into responses to urgent new needs. These new practices may spell out the future of afterschool services and new roles for afterschool teachers.  

The new stuff can be broken into two chunks. First, there were new service models developed, new ways to support children and families, and these modes of service delivery could work in other times and circumstances. We’ll talk about three models below and how they could represent an evolution of the basic afterschool model – to include an external focus on connecting with children and families together, often inside of their own homes. 

The second chunk of new work is in the definition of high-quality practice. While new service models were reaching out beyond the afterschool setting, a new definition of high-quality practices has taken shape in afterschool classrooms and clubs. The new definition of quality can be explained in terms of the familiar Maslow’s pyramid metaphor – the new stuff is all about improving supports at the bottom and top of the pyramid. In other fields, these additions are called “attachment-aware and trauma-informed.” These new practices are transforming our concepts about how adults work with children in the setting, particularly the increasing proportion of stressed children presenting signs of delay in both socio-emotional and academic skill domains.  

The three blogs in this series will address this evolution of the afterschool service model in three parts: First, we’ll talk about the new externally-focused service models that emerged rapidly between March 2020 and end of summer 2022. In a second blog post, we’ll talk about the state of the children in fall 2023 and how afterschool practice is evolving to meet new needs that children are bringing to afterschool programs. In the final blog, we’ll outline some of the best practices that align with the “attachment-aware and trauma-informed” way of seeing high-quality settings.  

The Best We’ve Ever Seen 

That’s how I characterize the YouthQuest afterschool programs in fourteen Flint Community Schools and the ten Bridges to Success afterschool programs in seven different Genesee County school districts. Together, these 25 afterschool programs serve over 1,250 students each year and employ at least 150 staff. As the evaluator for these networks over the past four years, conducting evaluations and CQI cycles based on on-site observation and qualitative methods, we got up close and personal with every site. We were well positioned to describe how practices changed. 

These schools also serve children that may have been affected by the lead-in-the-water crisis – so it has been the rare bit of really good news to know that top-flight developmental programming is available to many school-age children and adolescents in the city of Flint and its near suburbs. Pediatrician Mona Hanna-Attisha knew what to do for children who may have been exposed to lead. When Rachel Maddow asked: What should be done for children after the water pipes are fixed? The answer from Dr. Mona was: Give them top-flight cognitive-developmental programs (her example was Head Start). This is exactly what YouthQuest and Bridges to Success are delivering to school-age children in and around the City of Flint. This is no joke. Although contaminants like lead can damage the brain and change the performance of brain systems, powerful developmental experiences of the loving and engaging kind heal the brain and change the performance of brain systems in ways that compensate and transcend. And this logic applies to every other “insult” to children’s bodies and brains, i.e., ACE’s. This is the core assumption embedded in the idea of “attachment-aware, trauma-informed” practices. 

The Flint region is a likely place to find professionals who know how to deliver very high-quality practice and be intentional about “positive youth development.” The city and region of Flint has a great 100 year history of community education and experiential learning – and, for the last 30 years, the Bridges and YouthQuest programs have been at the leading edge of positive youth development models for the children of Flint. Along with the Beacon’s programs in San Francisco and New York, these were networks where the positive youth development method was developed in practice in the late 1990s.  

Aimee Phillips (bottom row, third from right), Program Director of the GISD Bridges to Success Programs, accompanied by the program site managers. 

Willie Buford (center), associate program director of YouthQuest for Flint & Genesee Education & Talent LaQuanda Hammonds (right), Jimantra Grant (left), and Jeniffer Pocza (not pictured), site managers of YouthQuest for Flint & Genesee Education & Talent in Flint, Michigan. 

As an evaluator and technical assistance provider who has now completed nearly 200 observations or professional development sessions with these 25 sites, I can report that there were maybe three cancellations, never a late meeting, and staff were always prepared. The average years of experience in teaching or afterschool across site managers was at least 12, and most worked in the same program for the past 5 years or more. These are both strong organizational cultures. 

As to normative excellence: Twenty-two of 24 sites (98%) produced an instructional total score greater than 4.0 on the Youth and School Age PQA’s in the 2022-2023 school year. According to our science [Impact Evaluation for the Palm Beach County Quality Improvement System (QIS) using Fully Pattern-Centered Analytics – QTurn (qturngroup.com)), the programs scoring over 4.0 are producing equity effects in two ways: Children who enter with lower skills exit with more socio-emotional skill growth compared to their higher-skill peers; and children who enter with higher socio-emotional skills do not regress. 

I can say. without hesitation, these two organizations are among the best run afterschool programs I’ve ever encountered. We’re writing these blogs because how these folks responded to the pandemic-years crisis is worth paying attention to. It tells us something about what is possible when families are experiencing crisis, and response is necessary. 

New Service Models 

The new service models emerged in three patterns: Crisis response, blended learning, and casework. These are described in greater detail in a report [In Review]. To summarize: 

  • A crisis response model delivers connection for families in duress by prioritizing connecting with families, communicating what afterschool programs can do for children and families in the moment, checking on the students’ and families’ wellbeing, and having a plan to connect afterschool services with school districts’ and other emergency information and services. In several cases, Genesee County afterschool programs administered family outreach for their district in spring and summer 2020. 
  • A blended learning model that delivers an integrated schedule of live and on-line instruction (including materials kits to use with on-line) and makes it possible to maintain or even increase time with children when in-person attendance has to be intermittent. This capacity may be evermore important in situations where some children are in the classroom and some are at home, online, on a rotating basis. And again, in general, these afterschool programs were engaging families with programming online before many school districts. 
  • A casework school support model that delivers intensive person-centered supports through flexible calendars and content, SEL check-ins for individual students and families, active family engagement including home visits, and ongoing advocacy for the student and caregiver in connecting with the school district or other services, e.g., food, housing, transportation, mental health. More than being in a classroom, program attendance for these students and families meant continuing to innovate with school and afterschool staff to fit existing services to families’ needs, particularly around helping parents keep children connected to school day learning.  

Personal Protective Equipment (PPE) was also a new practice in the pandemic years. We worked with these two networks to develop a set of indicators for use of personal protective equipment (PPE) inside the program. These are listed in Table 1. These PPE standards were applied during the 2021-2022 and 2022-2023 school years and are found here. 

Table 1. Personally protective practices on the COVID 19 Setting Quality Assessment. 

COVID SQA 

 

 

Personal Protections 

Safe Environments 

Screening Symptoms  

Face masks 

Sanitary Setting 

Child and Teacher covid screening 

Hand washing 

Materials 

Symptomatic student/teacher protocol 

Social distance 

Ventilation 

Caregiver communication 

Cohorts 

Drinking water 

 

 

Meal setting 

 

 

Pick-up/drop-off  

 

 

Because these PPE practices were implemented at high fidelity, we had some learnings: children in grades K-grade 8 were generally willing to wear masks, travelling in youth-adult cohorts through all aspects of the program improves relationships and community, and the online part of blended programming works well for some things (during a pandemic) like individual body movement, mindfulness, and including siblings and caregivers in programming. 

Looking Forward 

In one sense, looking forward is about what the afterschool field might evolve toward. The new service models imply that afterschool programs can engage families in a different way that exceeds positive youth development experiences delivered inside of program settings. In general, these new afterschool models (that no doubt developed independently, and more or less successfully, all over the US and world during the pandemic years) open the possibility that afterschool programs could have family engagement functions, multi-generational models that might be much more effective for some children and families – and two-generation models have large effects. It may also be true that younger children are not automatically better off in a program after school is over – it may be that some kind of supported home-based service with their caregiver would be better for children under 3rd grade.  

The other angle is climate response – we’re going to have more environmental catastrophes, and more refugees from environmental catastrophes, coming to our communities in the coming years. Program models that can check in with families quickly, deliver live or online positive youth development programs flexibly, and be able to help families in crisis seek basic services and keep their children connected to school will only be more important in the future. 

In the next blog, we talk about the new needs that children were presenting with in fall of 2023… 

Child Labor – the Missing ACE that Hides Child Abusers

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email
2,346 words | 18 min

I’ve always thought there were at least two missing ACEs (adverse child experiences) due to the fundamentally bourgeois experience of the people who think these things through. One of those missing ACES was child labor, now visible for the moment in the lens of fresh New York Times reporting (https://nyti.ms/3Ld1bdV). It can be hard to define child labor, in part because the laws have kept the definitions ambiguous to protect agricultural sector interests, a few of whom represent actual smaller and middle-sized farm families who still do their own dirty work – but most of these interests are agribusiness.  

I felt as though I had to blog about this issue now because it’s near and dear to my own childhood experience – and the reason why I’ve spent my second career trying to make educational settings do less harm to children with ACES, visible and invisible. Perhaps this time our field could rise above the embarrassing non-response that happened in the child separation policy under the Trump administration. Weren’t those children in the cages “out of school?” 

Since a big part of the New York Times story was about meat packing plants – and because those were many of the violations in my own state of Michigan – let’s talk a bit about two types of work experience available to children in meat packing plants: Dangerous machines and brutality toward animals. 

Traumatic Memory Lane 

I know these issues well, as the oldest child of a single-parent dairy farmer between 1968 and 1988, I spent roughly 20 years in compulsory labor that included lots of dangerous machines and the murder and torcher of animals on a regular basis.  

The compulsory labor started at age 6 and included about 4 hours each day after school and then dawn to after dark on weekends and summers. Now, there are many kids who work to help their families, and there are conditions of family labor in which children are protected from the parts of work that are “too much, too soon.” But that’s not what we’re talking about here. These are cases where children as young as 13 are being pressed into service by bosses, not parents. 

First the machines: I can still remember standing over a late 1950’s Gehl silage wagon (google it) while blowing silage at age 13, a John Deer 3010 at full throttle right next to me running a blower fan strong enough to blow heavy chopped corn 65 feet up a tube to the top of the silo… and another tractor power take-off shaft running the wagon’s conveyor chains and beaters – all in a tight space where I had to stand to control the machines and help manage the load of silage as it met the beaters. My intrusive memory from this experience is always the same – standing on the front edge of the wagon with a long pitchfork trying to pull the top of the load down evenly so it didn’t clog the blower… and in this action I was within inches of many moving parts that were not going to stop if I got caught up in them. I was working alone. No one would have known.  

This is a traumatic memory and, while many years of well-guided work has reduced its powerful emotional charge, the memory will never go away. Even at 56 years old, it remerges every time I feel taken advantage of, every time I feel that someone is not caring about the risk they are asking me to take on for their advantage. Let’s be clear: The threats are no longer the same, now that I’m working in a collared shirt. Still, when a situation that fits comes up, I automatically project a composite of all of those “dangerous machine” days onto the present situation, including the physical and emotional experiences that the 13 year old was having then during those stressful and scary moments. If you’ve ever spent any time in a meat packing plant, it’s nothing but wicked dangerous machines with sharp edges that require cleaning with high heat and caustic chemicals. 

But that wasn’t the worst. If you think that the terms murder and torcher are too strong: 

You’ve never seen a calf’s knees buckle and it’s aimless wandering after burning its budding horns back into its skull with a hot electric iron. 

You’ve never heard the desperate lowing of a young steer as you first cut both its horns off (crushed bone and blood squirting everywhere) and then castrate it, often falling to the ground in the process. 

You’ve never had to end the life of an animal that was down – or, worse, tried to get it on the sale-truck still alive with a loader and chains. 

I could go on. This is just what it takes to make meat and milk – but children shouldn’t be involved. I specifically use the terms murder and torcher because I left my 20 years as a steward of 300+ dairy animals convinced that cows and pigs have emotions and memories. After those experiences, they never forget, just like me, and the kids at the plant. They don’t want it to happen, and the fear and pain in their eyes and voices is clear, just like for me, and it will be that way for the kids at the plant. Again, to this day, these memories are vivid, and when they come it can take days or weeks to get them suppressed again.  

Do a little reading on the subject – going back to Upton Sinclair in the 1920s – and you will see that while regulated, these are industries where the protective shields (on the machines) are often disregarded for the sake of profit. By this, I mean very specifically that the machines are in disrepair, moving parts are not shielded and are open to the operator, and unnecessary brutality to animals is acceptable to make it all go faster. Even if you kept the kids away from the most dangerous machines and most egregious experiences of animal death (which I doubt), do you want 13 year olds hanging around with non-family adults while those adults are having these experiences?  

This is what the people who want children to work think is okay for children to experience in meat packing plants. These people are child abusers and should be in jail, not receiving fines to their companies with no personal accountability. If you think that’s extreme, let’s look at the science that says that the kinds of child labor I’ve been talking about have results similar to severe cases of sexual abuse. That’s why, in the past, both child labor and child sexual abuse were both called trafficking. The people who want children to work in meat packing plants are like sexual abusers. 

What the Science Says 

Unlike the anti-science gaslighting of conservative state legislators and corporate CEOs who believe that capitalism should exploit the vulnerable – there is science to guide us here. It can help sort out the causes and effects of traumatic experiences and where they specifically might occur in children’s experience of work and the conditions around work.  

Again, children’s work in benevolent circumstances, under the protection of caregivers in conditions of family economy, is not necessarily a bad thing. I also learned a lot from my first career in childhood – but the “too much, too soon” stuff made it a bad trade. 

Here are a couple of important definitions from the United Nations Children’s Fund: 

Child Labor. Includes work by children too young to perform it and/or work that is likely to harm a child’s health, safety or morals. Excludes permitted light work by children of a specific age range.  

Hazardous Work. Work that is likely to harm a child’s health, safety or morals. Each country creates their own hazardous work list. This typically involves activities that expose children to abuse; to work underground, underwater, at dangerous heights, or in confined spaces; and/or to heavy loads or an unhealthy environment. It also includes work in difficult conditions (e.g., long hours). 

Although national statistics on prevalence (like police homicides) conveniently do not exist for the United States, the more general categories of child exploitation and child trafficking typically include both sexual and labor abuses, and many older studies put them in the same category. For some purposes that works, but here is the more nuanced story: Sex abuse survivors have more messed up family situations and more prior experience in out-of-home care, child protective services, courts, etc. – approximately four times as high (Hopper and Gonzalez, 2018). They are also a bit more frequently girls. One thing this means is that messed up families are a risk factor for sexual abuse. Family dysregulation makes children more vulnerable, less protected. 

But child labor is often an intensive form of mistreatment too and, like sexual exploitation, is a source of the more severe form of post-traumatic stress disorder (PTSD) and its endemic mental and physical symptoms. Rather than “event trauma,” we’re talking about “complex trauma” that comes from long-term repeated exposure to too much, too soon. While children who experience event trauma (e.g., a single episode of sexual abuse from a non-caregiver, a climate disaster, witnessing an act of violence) can have high rates of recovery, the same is not true for complex trauma (also called complex PTSD – see Pete Walker). Here is what the corporate CEOs and state legislators around the country are committing these children to: 

Hopper and Gonzalez reported high rates of depression and PTSD among participants with a history of labor trafficking (Hopper & Gonzalez, 2018). The majority of individuals fulfilled the DSM criteria for depression (72%) and PTSD (54%), and specific symptoms commonly assessed in a review of systems were endorsed; for example, sleep disturbance (85%), fatigue (71%), weight changes (54%), suicidal ideation (43%), and nightmares (57%). Other symptoms of emotional distress that may be assessed during medical and mental health visits include intrusive thoughts (80%), avoidance of thoughts/memories (77%), hypervigilance (63%), concentration problems (45%), and somatic dysregulation (38%). 

The story here is that even if “hazardous” child labor causes symptoms of complex PTSD, the sequence of exposure differs: In sexual abuse cases, prior family history of dysregulation makes sexual abuse more likely. In child labor, it is the initial child labor situation itself that makes the child more likely to experience additional abuses of different types, often including being under paid (no irony here for the bosses, kids are cheap labor). 

Child labor co-occurs with all of the other types of mistreatment. Making or allowing children to work makes them more vulnerable to other forms of mistreatment and atypical patterns of child development. Because child labor, often rural, can be invisible to the upper-middle class and urbane folks who do child advocacy and developmental science, most of the good science here is brand new: 

Koegler et al. (2020) documented threats, as well as physical and sexual abuse, as co-occurring exposures during the labor-trafficking experience. 

Hopper and Gonzales (2012) reported significant rates of sexual violence and physical assault co-occurring with child labor.  

Zhang (2012) reported 15% of those involved in labor trafficking endured threats to their physical integrity.  

Marquez et al. (2020) highlighted a youth involved in labor and sex trafficking that met criteria for PTSD due in part to pre- and peri-traumatic exposures during both events.  

Macias-Konstantopoulos (2017) noted in the non-scientific literature that the medical needs of labor trafficking survivors may result from violence exposure, hazardous working conditions, and lack of healthcare access (suggesting the labor-trafficking experience may include barriers to self-agency and self-protection).  

Protect Children or Be an Abuser 

Under the Trafficking Victims Protection Act (TVPA), it is considered child trafficking when force, fraud, or coercion is used to compel a minor to labor (United States Government, 2000). Doesn’t this seem a bit weak? What happens when a caregiver adult encourages a child to work? Does that make it better? The answer to this question goes back to the too much, too soon point. It may be okay if a caregiver agrees with the decision to have a child do labor, if that labor is explicitly of the non-hazardous kind, and if it happens in the context of family economy where a family labors together. Everyone learns from work. However, it’s funny how it’s okay for poor rural kids to work (in non-hazardous conditions), but the children of the people who write about ACEs, and all of their children’s friends, spend their time studying and skill building, often never experiencing real labor with back and hands in their whole lifetime. Hence a missing ACE. 

It comes down this, the New York Times and the current science says that a significant number of our children are being exposed to hazardous working conditions that will hurt them badly, often throughout their lifetime. It’s pretty simple for us in the child development field. We need to decide which side we are on. Just saying that we use a developmental method when we’ve got the children in our setting is not enough. Like the kids who were separated from their parents at the border, these children are being set up with powerful and self-destructive memories that will last a lifetime. What will the out-of-school time field do? Because we are all “developmentalists,” shouldn’t we be using our privileged knowledge of psychology to take a position? Shouldn’t we be speaking truth to the men and women in power who are abusing our children? What should we ask our philanthropic and agency leaders to do?  

Please feel free to share this blog with legislators in your state. From a non-exhaustive review, the legislatures in the states of Iowa, Minnesota, Ohio, Michigan, Wisconsin, Arkansas, Georgia, South Dakota, New Jersey, New Hampshire, and Illinois have all recently introduced or passed legislation making it easier to exploit children, see: Republicans in Some States Want to Ease Child Labor Laws to Fill Jobs (businessinsider.com); States Look to Ease Some Child-Labor Laws Amid Tight Market – WSJ

How could OST address climate change?

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email
650 words | 3 min

With the publication of the IPCC report[1], it’s not difficult to conclude that our current political leadership is not going to take us where we need to go, and we can’t wait anymore. The scientists are telling us right now, in clear language, that the time is up: Major transformations in our thinking and behavior around energy use must happen right now. We’re already on track to exceed (by the early 2030s) the 1.5C degrees threshold of global warming (above pre-industrial levels), and the weather patterns that are projected to ensue, long before 2040, are going to be catastrophic for everyone on the planet. That’s the very difficult news. However, the amazingly lucky news is that there is still time to do something. So, what should that something be for OST professionals as OST professionals? What should we be asking our leading agencies and philanthropies to step in quickly and fund?

Building on Strengths

For the children, youth, families, and communities served by OST programs, socio-emotional skills are at the top of my list, including getting up to speed on trauma-informed practices. We are likely to encounter more trauma from climate-related dislocation, intermittency of power, failing ground water systems, etc.

As important, how to protect children from talk about the ongoing climate catastrophe? How do we nurture their hopefulness rather than sharing our adult fear and unhealthy denial? How do we nurture our own hopefulness as an example? This is a profoundly important task if we are going to do anything at all as a field. The main point of this blog is that it’s our ethical obligation to the children and families that we serve to demonstrate hopefulness through our own OST practice, like setting examples in the time that we have rather than sticking our heads in the sand. It almost doesn’t matter how we go about addressing it as along as we communicate hope rather than fear while we’re doing it.

Another big one that we already know about is demographic change. We need to understand south-to-north migrations and how to be intentional about multiculturalism in practice and policy. This will be a place to use all of the equity stuff that so many of us have been working on and thinking about. We need to prepare for society with a bunch of folks from southern US states, Central America, Mexico, and the islands arriving in the next decade. Its already happening. Good news is that many of us are building our capacities to engage multi-cultural youth to embrace differences and learn about their own cultures (e.g., intersectionality) and to understand inequities.

There is a lot going on in our field in youth civic engagement. This is a small subset of our OST field, but these folks have been doing it for a long time, know how to do it, and have written a lot about it. The trick here is to re-focus what we know about how to do youth activism – e.g., community and service learning, voter registration and other issue-driven campaigns, peaceful protest – with the mental model that Greta (Thornburg) is providing about breaking carbon and consumption cycles. I can easily convince myself that learning how to break carbon and consumption cycles in everyday life is now more important than any time spent on academic testing for example (or advanced math or competitive sport etc. etc.).

 

Filling in STEM Denial

One of the huge opportunities to demonstrate hope is precisely where we are least prepared: all of the new habits of daily life that we’re going to need to learn in a zero-carbon world. We are facing a task like the early twentieth century policy of agricultural extension that sought to educate a whole country of beginner-farmers about agricultural science and how not to starve in the countryside. But what institutions are filling this void?

Elsewhere we’ve referred to these habits as the applied science of ecological stewardship[2][SP1]  – again, to push the agricultural extension analogy, think home economics for zero carbon. The new zero-carbon habits are all STEM skills even though the OST STEM field seems to be in denial. From my reading, these are the STEM issues that are directly and substantively related to reducing carbon use and living well in the future:

First, there is a lot to learn about energy use technologies and energy conservation skills. Dramatically reducing carbon use in the next three years and staying comfortable in everyday life is something we should all be learning about and about which there is almost certainly some content available for use in OST programs. This ranges from the simple stuff like when to best turn on and off the air conditioning to the basics of the new energy technologies and infrastructure that every neighborhood will need.

Another important set of STEM skills is related to food types and sources. It is an uncomfortable fact that two big ways to cut greenhouse gases and carbon use is to stop eating meat and to grow food locally. The arguments about the ecological implications of plant-based diets are clear and should at least be available to all persons. It’s also clear that growing your own local food in any city is likely going to occur on a brownfield where the underlying soil is already contaminated. This requires some horticultural design know-how and a little bit of soil science that many urban farmers are learning.

Another big one may surprise you: Almost all old trees (i.e., largest biomass retaining carbon) have been eliminated except for those in the urban forests maintained by our city and town governments and national parks. The carbon retaining potential of the urban forests is critical, and OST programs could learn a lot about trees (e.g., dendrology) and carbon retention from the several STEM disciplines involved. There are few afterschool programs actually located in non-urban areas (i.e., even in the countryside, OST programs are typically located in a small town). These cities and towns are home to the last remaining 60 year old+ trees in the United States and represent an important part of the solution for carbon retention.

Access to clean water is a major challenge of climate change, and procuring clean drinking water is going to involve all of the STEM disciplines – both building mental models for applied stewardship and then acting in those terms. Shouldn’t all OST students learn about the local clean water agenda and learn how to understand contaminants in water testing output? Shouldn’t all students learn about the science and technology involved with water filtration at home?

For an OST Social Movement

What about the power of the OST social movement – we professionals as a group? How could OST leaders help us act together as a coordinated profession to use our power? Are there a few obvious choices that we could ask leading agencies, membership organizations, and philanthropies to help us engage the field around?

The first question is: How do we quickly integrate all of this content into our professional conferences and workplaces so that we can quickly figure out how to help a next generation of ecological citizens, scientists, educators, advocates, and policy makers who are inheriting our legacy? A second question follows from the ethical charge to demonstrate hopefulness to children: How should we engage local administrators and governments on carbon-reducing practices for our buildings and program offerings? Finally: How do OST professionals engage in political activism as an interest-group to identify and advocate for breaking carbon and brownfield consumption cycles in personal lives and governance?

Although it may be uncomfortable to discuss these issues – the fear for all of us is real – it is our obligation to the children we serve to put them in the best position to deal with the situation, by building our own and their socio-emotional skills and mental models about the new zero-carbon world that’s coming. Please note also that all of the issues addressed above represent the major job categories of the future – for those still defining everything we do in terms of the economy. And finally, please also note that the OST profession is filled with rational people with progressive views. If we acted together, we could likely act as one. If we don’t act now, it will actually be too late. How can we organize ourselves and our young people and communities to amplify their voices, needs, and realities? How would we like our leaders at all levels to help?



[1] Intergovernmental Panel on Climate Change (IPCC; 2022), Climate Change 2022, Mitigation of Climate Change, Summary for Policy Makers [https://www.ipcc.ch/report/ar6/wg3/]

[2] Smith, C. (2019). SEMIS coalition for place-based ecological stewardship: Growing a movement, getting ready for growth. [https://www.qturngroup.com/wp-content/uploads/2022/04/2022-04-11_SEMIS_WP-v7.pdf]


 

How the Q-ODM impact model is a more cost-effective form of the quasi-experimental design (QED)

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email

The Quality-Outcomes Design and Methods (Q-ODM) approach to program evaluation increases the use value of all estimates produced as part of an impact analysis. Put simply: We replace the “no-treatment” counterfactual condition (i.e., children who were not exposed to an afterschool program) with low-implementation conditions (e.g., children who were exposed to lower-quality instructional practices in an afterschool program) in order to describe the impact of optimal implementation on child outcomes (e.g., socio-emotional skill change, equity effects).  Said again: The “control group” in our impact model is any quality profile, subgroup configuration, or pathway (e.g., low-quality practices profile) that is contrasted with an optimal “treatment” group (e.g., high-quality practices profile).[1]

The “Analytic Tools” section of White Paper 3 provides an introductory discussion of Q-ODM impact models for student skill and equity outcomes. Also, check out this UK impact evaluation.

Now, let’s talk about three reasons why our approach is a cost-effective choice for CEOs seeking evidence about impact and equity outcomes:

Lots of Reality-Based Estimates that Analogize to Action. Our point about cost effectiveness is this: Every estimate produced in this impact model is useful. Where coupled with QTurn measures, Q-ODM impact estimates are interpretable in terms of specific adult and child behaviors and contexts. This means that there is a direct analogy from meaning encoded in the data to meaningful teacher and student behavior that occurs in the classroom – direct analogy from data to reality. The data used to identify the lower-quality profile actually identifies the lower-quality settings! The amount of skill change that occurs in the high-quality setting actually demonstrates what’s possible in the program; that is, it sets the benchmark for other programs.

An impact estimate implies a subtraction of one magnitude from another. What use is a counterfactual estimate if there is no such thing as a counter factual condition? Doesn’t that just mean that we are subtracting an imaginary quantity from a real one?

Using Natural Groupings to Address Threats to Valid Inference. Its not just usefulness of estimates (consequential validity) but, we argue, a more valid way to rule out primary threats to validity of inference that the treatment caused an effect. Two points: The children in the low-quality group are more likely to be similar to the kids in the high-quality group for all of the right reasons (i.e., SEL histories) that are missed by most efforts at matching individuals or groups using demographic and education data.

The case that families in one group have more education-relevant resources (e.g., SEL histories) than families in the other group plays out in two ways. When families have unmeasured resources before the child attends, we are talking about selection effects. When families use those unmeasured resources during the program intervention we are talking about history effects. We argue, and present evidence, that the Q-ODM method better addresses these threats to valid inferences about impact than the pernicious and unethical use of race/ethnicity and social address variables as covariates – pretended “controls” – in linear models.

Capturing Full Information from Small Samples. Our method is designed to detect such differences in the ways things go together in the real world, in or around the average expectable environments characterizing human development and socialization (cf. Magnusson, 2003). This in-the-world structure is a constraint on the states that can and cannot occur during development. In the pattern-centered frame, small cell sizes indicate sensitivity of the approach. Relatively-low Ns are not necessarily a problem for the distribution-free statistical tests used in pattern-centered impact analyses.

 

[1] We realize that others would claim that our designs are not QED at all. We delve deeper into the rationales used to disqualify “groups that receive different dosages of a treatment” from being considered “control groups” within the context of experimental design in White Paper 4.

 

Why are Q-ODM’s Pattern-Centered Methods (PCM) More Realistic and Useful for Evaluators?

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email

Pattern-centered theory and methods (PCM) can be used to tell simple and accurate stories about how real persons grow in real school and afterschool classrooms. Stories about the quality and outcomes (i.e., causes and effects) that are modeled using PCM are particularly useful because they can address questions related to “how” programs and classrooms work and “how much” children grow skills.

Most training for education researchers and evaluators is focused on variable-centered methods (VCM), also called linear statistical methods (regression, the analysis of variance, and structural equation modeling) or the general linear model. VCM are powerful in cases where the causes and effects are similar across individuals and classrooms. In cases where that’s not true – which is most school and afterschool classrooms – VCM designs tend to provide information that means practically nothing about the actual people or contexts involved. Some of the basic issues have been summarized nicely by Todd Rose in the following TEDx presentation: https://youtu.be/4eBmyttcfU4 (“The Myth of Average”), but the critique is not new.

To better illustrate the point, let’s talk about three basic assumptions about the person-reality in afterschool classrooms and how PCM applies:

A person’s socio-emotional skills are most accurately represented as a pattern with multiple skills indicated simultaneously. This is not just about more information from more variables, although that is also a fundamental advantage of pattern-centered methods. The neuroperson is also a “multilevel system” – which is mouthful but as detailed in White Paper 1: Different parts of mental skill change for different reasons, on different timelines, and cause different types of behavior! This means different amounts and types of cause are involved in changing any mental skill or behavior. How could one variable at a time constraints of VCM ever do an adequate job of representing socio-emotional skill? PCM are uniquely fit for sorting out multilevel causal dynamics so that the full meaning encoded in the data can emerge.

Change in socio-emotional skill is always qualitative, from one pattern to a different pattern at a later time point. Given the multilevel nature of socio-emotional skills, the combination of skill parts is likely to differ at different time points and in different settings. The fact that skills turn into different skills as they change has been an Achilles heel for VCM. Check out the “Analytic Tools” section of White Paper 3 to see how PCM can be applied to (a) identify each individual’s unique pattern of skill parts at different points in time and then (b) compare across those qualitatively different patterns to detect stability, growth, or decline for each individual. When coupled with the sensitivity of optimal skill measures (see White Paper 2), PCM are ideal for describing the how (e.g., an individual child’s movement from one pattern to a subsequent pattern) and how much (e.g., how many children grew) of skills-change over short time periods, such as a semester or school year.

The same classroom causes different patterns of change for different subgroups of children. An adage from mid-20th century psychology (Kluckhohn and Murray, 1948, p. 35) is a helpful reminder: Any individual can, for different causal variables, be simultaneously like all others, like some others, or like no others. VCM work only in the first case, where every person experiences a very similar type of cause and effect. Case-study and qualitative methods are preferred in the third case, where the causes and effects may apply only to a single person. PCM are uniquely fit for the second case; that is, where different subgroups of children with different socio-emotional histories have qualitatively different types of responses to the same education settings.

In the end, VCM assumptions about the validity of single variables, the quantitative nature of skill change, and the homogeneity of causal dynamics lead to an impoverished view of reality – and likely a lot of inaccurate conclusions about what to do.

Introduction to White Paper 3

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email

Greetings friends! In this third White Paper, Realist(ic) Evaluation Tools for OST Programs: The Quality-Outcomes Design and Methods (Q-ODM) Toolbox, we extend from the neuroperson framework for socio-emotional skills to a focus on evaluation design and impact evidence. Focusing on the methods used to evaluate out-of-school time (OST) programs and to assess the impact on student skill growth is a critical issue, especially given the ambiguity about impacts from gold-standard evaluations of publicly funded afterschool programs. Are programs producing weak or no effects? Or, are gold-standard designs missing something?

We offer a sequence of evaluation questions that chart the course to realistic evidence about quality and outcomes (i.e., cause and effect, or “how” and how much”) – and is useful to managers, teachers, coaches, and evaluators. We’ve learned these questions over the past two decades by asking tens of thousands of afterschool, early childhood, and school day teachers about how data and results about their own work works best for them.

Getting the evaluation questions right calls for measurement and analytics tools that:

…reflect the assumption that children have mental skills that are causes of their behavior…. These mental skills are conceived of as several different aspects of mental functioning (i.e., schemas, beliefs, & awareness) that exist within every biologically-intact person, enable behavioral skills, and can be assessed, more or less accurately, using properly-aligned measures. When the parts and patterns of skill are reflected in theory and measures, the accuracy and meaningfulness of data about program quality and SEL skill – and all subsequent manipulations and uses of the data – are dramatically improved.

Our thinking is deeply anchored in pattern- and person-centered science. Check out a related blog here: Why are Q-ODM’s Pattern-Centered Methods (PCM) More Realistic and Useful?

Finally, we provide data visualization examples that complete an unbroken chain of encoded meaning, from the observation of students’ socio-emotional skills in an afterschool classroom, to the decoding of the data visualization by an end-user. We’re pleased to share these insights. Cheers!

P.S. For CEOs that need impact evidence: Why are gold-standard designs not as cost-effective as we might think? Elsewhere, we have argued that gold-standard designs for afterschool programs are misspecified models because they lack key moderator and mediator variables (e.g., instructional quality and socio-emotional skills). For example, the large impacts (often equity effects, as predicted by the neuroperson framework) that we typically find for students who start programs with lower socio-emotional skills but who receive high-quality instruction cannot be detected using most gold-standard designs. As a result, it is difficult (or impossible) to analogize from the results of gold-standard designs to the real actions taken by real people; thus, those designs are not very cost effective for improvement or for telling compelling stories about impact. Check out a related blog here: How the Q-ODM impact model is a more cost-effective form of the quasi-experimental design (QED).

Introduction to White Paper 2

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email

Welcome back! In this second white paper, Measuring Socio-Emotional Skill, Impact, and Equity Outcomes, we extend from the White Paper 1 skill framework to discuss implications for accurate measurement.

We are pleased to share these hard-won lessons from two decades of trying to describe the actual outcomes of “broad developmentally-focused programs” – which means trying to figure out how to measure socio-emotional skill changes of both adults (i.e., quality practices) and children over short time periods. In the paper, we work through the logic of measurement in a way that we hope non-technical readers can follow with minimal suffering. There are no Greek symbols!

We’re passionate about this subject because the potential is real. Getting measurement right will make a big difference for the oft-ignored questions about how and how much skills change during relatively short periods, such as a semester or school year. To put the conclusion up front: Maximize measurement sensitivity in applied settings by using (a) adult ratings of child behavior that (b) reference periods of not more than two weeks past,  and (c) using a scale anchored by frequency of behavior – what we call optimal skill measures.

Another message is that regardless of measure choice, items should analogize to actual mental and behavioral skill states that occur in real time, using words that all raters understand in the same way. Without this power of analogy from the raters’ concept of the verb/predicate in the written item to an observed quality in the room, external-raters can’t make clear comparisons before checking the box. The same is true for self-raters observing thoughts and feelings happening inside their own mind/body.

The kicker is that as inaccurate data are aggregated, the extent of invalidity is compounded. What if the ambivalent impact findings repeatedly demonstrated by gold-standard evaluations in publicly-funded afterschool programs were caused by leaving out accurate information about socio-emotional skills? (This is, in fact, a key argument elaborated in White Papers 3 and 4.) Thanks for checking out our work!

P.S. for the psychometrically minded. Why are many SEL skill measurement constructs likely to be inaccurate, despite psychometric evidence of reliability and validity? First, many measures of SEL skill lump things together that they shouldn’t. For example, mixing self-report items about (beliefs about emotional control in general (efficacy), the felt level of charged energy in the body (motivation), and specific behaviors (taking initiative) that follow – creates scale scores that obscure distinct parts of skill that change on different timelines and with different causes.

Second, it turns out that most measures young people encounter in school day and OST settings are self-reports of beliefs about skills. Students are rarely trained in the meaning of the words in the items that they are responding to while coming from different histories and different “untrained” perspectives on emotion-related words. We just don’t know what the words mean to the self-reporter, particularly the relative intensities offered in multi-point Likert-type response scales.

Third, items that refer to the use of skills in general (i.e., a verb without clear predicating context or time period) are much less sensitive to specific skill changes that actually occur over short periods of time. We refer to these as measures of functional skill levels that change more slowly over time.

In the new year, we’re highlighting the third white paper, Realist(ic) Evaluation Tools for OST Programs: The Quality-Outcomes Design and Methods (Q-ODM) Toolbox. In this paper, Charles Smith and Steve Peck extend the ideas introduced in White Paper 1 (socio-emotional framework) and White Paper 2 (socio-emotional measures) to program evaluation and impact evidence.

Reflections on White Paper 1

Share on linkedin
LinkedIn
Share on twitter
Twitter
Share on email
Email

In conjunction with the release of White Paper 1 this week – A Framework for Socio-Emotional Skills, Quality, and Equity – we want to mention a few of the highlights:

What are socio-emotional skills? In our view, a person’s socio-emotional skills are integrated sets of mental and behavioral parts and processes (i.e., schemas, beliefs, and awareness); these integrated systems are socio-emotional skills and produce both basic and advanced forms of agency.

Why are socio-emotional skills important? Socio-emotional skills have a compounding effect on many developmental outcomes that has been described as dynamic complementarity (Heckman, 2007); that is, socio-emotional skills beget other types of skills. Children and adults operating at high levels of SEL skill can more easily get on to the business of learning what the context has to offer. Settings that do not address SEL skills can become a further cause of educational inequity.

Why are organizations and policies struggling to implement socio-emotional skill reforms? A recent review found over 100 different frameworks describing SEL skills and supports (Berg et al., 2017). This cacophony of words and concepts undermines the shared understanding and language necessary for coordinated action, both within organizations doing the work and among evaluators producing the evidence.[i] Confusion about what constitutes SEL skill, and how “skill” may or may not differ from many other concepts – such as, competence, abilities, traits, attitudes, and mindsets – undermines scientific progress and slows policy processes that rely on at least approximate consensus around shared meanings and objects of measurement.

How can the QTurn socio-emotional skills framework help increase the effectiveness of reform? By defining, naming, and sorting out the key parts of integrated SEL skill sets, we can much more effectively measure and model both changes in socio-emotional skills and, ultimately, impacts on outcomes and equity. In White Paper 2, we extend from the socio-emotional skills framework described in White Paper 1 to corresponding guidance for measuring socio-emotional skills with increased precision, accuracy, and sensitivity.

We’ll be back with more soon…

 


[i] Given the extent of diversity across SEL frameworks, Jones et al. (2019) developed resources to help stakeholders understand the unique strengths of different frameworks as well as the alignment between core elements of these different frameworks. The general conclusions from this work are (a) there is currently no single consensus framework that is obviously more scientifically or practically valid than any or all of the others, and (b) the use of the same terms by different frameworks where presumably referring to different things (i.e., jingle fallacies), and the use of different terms by different frameworks where presumably referring to the same things (i.e., jangle fallacies), are abiding challenges faced by stakeholders charged with making funding, evaluation, training, performance, measurement, and analysis decisions. Our approach is designed to help solve these problems.