About that Mirage, Part I
Last week, an organization I admire immensely, TNTP, came out with a report on Professional Development called The Mirage. The gist of it was that PD doesn’t work very well at making people better. The study analyzed the outcomes of PD programs across three large districts and one charter management organization (CMO). (The CMO was not Uncommon, where I work; beyond that, I don’t know which organization it was). In total, researchers surveyed 10,000 teachers and 500 school leaders.
The findings in the three districts were that 3 out of 10 teachers improved as a result of PD and nothing the analysts at TNTP could identify was a reliable cause of (or even correlated with) that improvement. Improvement appeared to be unreliable at best and totally unpredictable, despite millions of dollars in direct and indirect expenses. And improvement became very rare indeed after a teacher’s first few years in the profession. Teachers corroborated these findings: PD, they said, didn’t help them very much. The first twenty-nine pages of the report document and analyze these findings.
Only on about page 30, where the report examined the CMO, did things really start to make sense. By the time I’d finished the report, I had the opposite reaction from the authors: Despite their assertion that they were unable to learn much of anything about what worked in developing teachers, the report laid out a pretty viable analysis of what probably works in helping teachers to improve. (What you might say honors them for their effort by making them better for the rest of their careers). The gist of it is this: workshops don’t develop people; organizations do. Before I tell you more about what they found, and why I think they found it, I want to be clear that in my eyes this is not a charter-versus-district schools argument even though the CMO happened to be the positive example in this study. So please hear me out.
TNTP found that 7 out of 10 teachers improved in the CMO each year. Again, the authors were unable to isolate drivers of growth, but at the bottom of page 30 they wrote: “We did find some differences on an institutional level [in comparing the successful organization to the three unsuccessful ones], specifically a more disciplined and coherent system for organizing themselves around teacher development, and a network-wide culture of high-expectations and continuous growth.” Later they summed it up this way: “Having a meaningful impact on teacher performance over time depends as much on the conditions in which development takes place as on the nature of the development itself.” I might add to that, in my opinion, the nature of the development also matters deeply, but it’s hard to tell if there isn’t a culture of excellence and growth within the organization.
What does that mean? What are the all-important “conditions in which development takes place”?
For starters, I’ve identified some of the findings about the successful organization in the bulleted list below. Again, it is more important to me that this was the one successful organization out of four studied than the fact that it was the charter organization in the bunch. In fact, I would be willing to wager that, if you disaggregated the data by school, within the three generally ineffective districts, you would also find single schools and school leaders who were able to create the sort of conditions and build a vibrant learning culture that led to reliable growth among the adults. It might be easier to do in the CMO, but leaders in districts do it as well. However, that fact that it appears to be very hard to execute that sort of culture across an entire school district is perhaps part of the point. Decisions about PD should be made at the level within an organization where decisions can be made and executed to build positive culture around it.
Here are the author’s findings about the organization (where more than twice as many teachers improved annually):
- A culture of excellence among the adults: The report noted that at the successful organization, “everyone in (the) school community was constantly working towards better instruction and pushing each other to do their best work.” “What’s unique about being at [my school] is that there is always… someone to push you,” the report quoted one teacher saying. “I don’t think I’ll ever be able to stagnate here.” At the successful organization, people pushed one another constantly. There was a clear sense across the organization that the goal was to be great at the most important work in the world.
- Development is embedded throughout the school culture: Workshops are great, but as someone who runs a lot of them I am sure they are insufficient by themselves. They require constant follow-up, conversation and support, especially at the peer-to-peer level. At the successful organization, the report notes, “Each teacher receives weekly observations from his or her coach, followed by a 30-45 minute debrief.” Further, compared to teachers in our other three organizations, teachers in the high performing organization “are far more likely to report opportunities to practice teaching outside the classroom (82 percent reporting “sometimes” or “often” practicing, compared to 17 to 38 percent elsewhere). What’s more, teachers at the successful organization “spend two to three hours every week with other teachers, reflecting on instructional practices… practicing new skills or reflecting on changes to be made next.” Professional development, in short, isn’t something you do outside your daily job; it is the daily job.
- It’s primarily an improvement culture (rather than an evaluation culture): Because of their frequency, the observations at the successful organization were formative… they were specifically tied to getting better rather than evaluation-oriented. Perhaps as a result, the report’s authors found that teachers in the successful organization–were by a wide margin–more likely to believe that observations and feedback were “effective for their improvement.” The report calls this a “culture of continuous learning” and notes that it is distinguished by its vibrancy and rigor.
- Alignment and efficiency: At the successful organization, the report notes, the central office focuses on developing leaders’ coaching skills, providing data to school leaders so they understand what’s happening in the school and supporting them with trainings. But it is school leaders and their leadership teams that do most of the teacher development. And they do lots of it—it’s perhaps their primary job. But one outcome of this is coordination. There aren’t people dropping in from all over the system to give teachers more and more advice. It’s coordinated and streamlined so one or two people give consistent advice on consistent, important and limited topics.
- Investment in Teachers: It’s also worth noting that the successful organization makes an even bigger investment in developing teachers about twice as much according to the report. The best organizations are all about investing in people development.
So for me what the report suggests is this: Developing people is a function of organizational culture, of good training in an organization that’s built to make its people better. If that last part doesn’t happen, it’s not likely that people will get better. To paraphrase my colleague Paul Bambrick, PD has to be embedded in the life of a school.
Workshops are great but most PD happens when we don’t realize it’s PD—when teachers are talking; when they are pushing one another to be rigorous about improving; when they are getting feedback and practicing constantly. And again, while it is the CMO that is doing that best in this study, there’s not much reason all schools couldn’t. In fact, as I mentioned above, I bet that within those districts, many individual schools do.
To drive this point home, tomorrow I am going to share Part II of this post, a video of powerful teacher development happening in a highly successful school. In the meantime, a final caveat: if organizational culture is a primary driver of PD results, then I think we have to consider all of this–both TNTP’s report and my interpretation of it–as conjectural since the number of organizations in the report is actually just four, which is too small of a sample size to draw firm conclusions from.