Last week, I wrote about an important new IES-funded study conducted by Mathematica, which examined the performance of secondary math teachers recruited by two selective alternate route programs: TNTP’s Teaching Fellows and Teach For America.
Now that I’ve had more time to read the full study and get some questions from folks, let’s get into the weeds. This post is going to score pretty high on the nerd-wonk index. If you don’t get excited by that disclaimer, you will be forgiven for proceeding no further.
How did Fellows do?
Collectively, the Fellows in the study were indistinguishable from comparison teachers. However, the study included Fellows of varying experience levels. They were all studied in the same year. This means that the novice Fellows were recruited and trained more recently–in some cases many years later–than the more seasoned Fellows in the study. That’s important because we have improved our training over time.
Novice Fellows outperformed comparison novice teachers by quite a lot. The math scores for novice Fellows were 0.13 standard deviations higher than the scores of the comparison group. This was statistically significant at the .01 level. This was almost double the effect size the study found for Teach For America’s novices relative to other novices (0.08 SD). Given that the study overall was excellent for our friends at TFA, this is a good finding for us.
I see it two ways. It’s great that Fellows who were trained more recently did very well. But the sample size for that sub-group wasn’t very large, and we didn’t see that same effect size across the board.
Clearly, lots of work to do. But we knew that already; we made substantial changes to our training and certification process several years ago because we felt our Fellows were not at the consistently excellent level we hoped for. Those changes are not reflected in this study–even for the novices who outperformed their peers.
Aren’t the effect sizes in the study really small?
I wouldn’t say so. Studies of groups of teachers tend to find quite small or non-existent differences between the groups. This is because even among teachers selected and trained in similar ways, there’s a wide variation in actual classroom performance. Performance differences among teachers at the secondary level tend to be even smaller, as there is less variation in the learning gains by students.
Therefore, the 0.07 effect size for Teach For America teachers across the whole study is consequential, in my view. As I said in my previous post, there is plenty of room for debate about how we build and sustain a high-quality national teacher workforce, but there really isn’t that much room for debate anymore about whether Teach For America produces effective teachers. Because TFA’s novices outperformed more experienced comparison teachers, even the question of TFA’s two-year commitment doesn’t carry the heft it did at one time, at least when it comes to student learning gains.
To me, the best way to achieve larger effects is to combine high-quality recruitment and selection work with a requirement that only effective novices earn the right to become career teachers. This would address the challenge that all pathways produce teachers of varying effectiveness. This is what we do at TNTP today. We choose wisely, support as well as we can, and then expect our teachers to meet a high standard.
Who are the comparison group teachers?
This is the most interesting question. It’s tough to be brief. When you hear “randomized assignment,” you may assume that each teacher is paired with another who is alike in every way except for the pathway he or she came through, and that the collective teacher groups are nearly identical. But that’s not how it works in the real world.
The randomized assignment applies to students. Teachers in the study are given slates of very similar students with similar starting points. The pairs of teachers are in the same school, teaching the same subject. But it would be nearly impossible to ensure that teacher pairs were also the same in all their traits unless researchers could move teachers around from school to school. That’s too disruptive.
Therefore, the comparison groups are not the same as the Fellows and Teach For America teachers to which they are being compared. Further, the group of Fellows in the study was not the same as the group of Corps Members, in terms of traits. You following all this?
For example, the typical Fellow in the study had four years of experience. The typical comparison teacher had 13 years.
There were 66 Fellows in the study, and 15 of them were first- or second-year teachers. There were 64 Teach For America teachers in the study, and 53 of them were first- or second-year teachers.
You get the idea. The researchers took many steps to draw sound conclusions despite the significant dissimilarities between the matched pairs in the study, but it’s certainly possible that future studies could follow the same protocol and come out differently. That’s just how it works. Schools aren’t labs.
What are you going to do with the results?
The main priority for TNTP is to build on the success of the novice teachers in the study. Our standard for success in our training programs is to produce the best teachers available from any route. There are places we meet that standard today and places we don’t.
We think we can do better, especially if we look carefully at the results we’re getting and use them to improve our teacher support. We’re pretty convinced that teachers who attain basic skills early in their careers are better positioned for the long run. We’ve re-tooled our pre-service training in conjunction with the fabulous team at Uncommon Schools, including Doug Lemov and Erica Woolway. We’ll publish a paper in the next six months that details our exploration of that concept and the initial results.
