Using the Rubric

The first step in creating a strong norming program is to define—and make sure everyone understands—the purpose of the observations. Will they help inform teacher development? Will they be part of formal evaluations?

After you establish your main goal, set your norming goal for observers. If, for example, you want to offer teachers high-impact professional development, your training goal might be to make sure observers can consistently identify teachers’ biggest growth opportunities.

If you are using the TNTP Core for formal evaluations, you may want to focus on inter-rater reliability across observers so that observers can use the rubric to deliver consistent, accurate ratings. Inter-rater reliability has two components:

• Accuracy: Can observers assign the correct rating?
• Consistency: Can observers regularly assign the same rating across similar circumstances?

When designing your norming program, we suggest focusing on three key components:

1. Academic Content

The TNTP Core Rubric asks observers to focus on whether the academic content students engage with is appropriately challenging and aligned to grade-level standards. A strong program will help observers understand what appropriate content looks like in each subject area and allow them to practice evaluating content. TNTP observers are trained to use tools like the Mathematics Toolkit and the Text Complexity Toolkit to evaluate content accurately.

2. Practice and Feedback

A few hours of group discussion or a close read of the rubric is not enough to successfully norm observers: they need plenty of opportunity to practice and receive feedback on their observations. Our observers watch and rate no fewer than seven full-length instructional videos during initial training and then “check in” at three to four points during the year to rate and discuss additional lesson videos or co-observe in classrooms. Overall, they get about 40 to 50 hours a year of observation practice.

3. Alignment

Facilitators should focus on helping observers score lessons as similarly to the normed ratings as possible. At TNTP, observers who conduct high-stakes evaluative observations must meet a minimum threshold of alignment to normed ratings. See below for more on our standard norming “bar” for the TNTP Core Rubric.

The Core Observation Guide provides an in-depth overview of the Core Rubric, as well as opportunities to practice assigning ratings on lesson videos and compare your ratings to TNTP’s normed ratings. If you want to develop a robust training plan, we suggest following the sequence outlined below. Most of the training time should be spent in the Assigning Ratings section, where observers can rate and discuss real lessons.

Part 1: Understanding the Rubric

Understanding the Rubric can be assigned as independent study, or you can review and discuss the content as a group and then facilitate a team discussion. When discussing each performance area, we strongly recommend asking participants to highlight the specific, concrete evidence each performance area asks observers to look for, as well as the evidence that pertains to other parts of the rubric.

We also recommend moving through the sections in the following order:

• Culture of Learning
• Essential Content
• Academic Ownership
• Demonstration of Learning
• Performance Levels
• Exemplar Lesson

After reviewing this section of the Guide, observers should have a baseline understanding of what each performance area evaluates and be able to articulate what practice at each performance level looks like.

Part 2: Using the Rubric

Once your observers understand the rubric, they can use the following sections of the Guide to learn how to collect evidence from a classroom observation and use it to assign ratings:

• Collecting Evidence and Bucketing Evidence show observers how to collect the right evidence to rate fully on the Core Rubric.
• Evaluating Content provides tools for evaluating academic content. This list is not comprehensive, so if your school or district uses other tools to evaluate content, you should highlight those here.
• Assigning Ratings provides six lesson videos with normed scores generated by TNTP staff to help you better understand where TNTP holds the bar for Proficiency on the Core Rubric.

The bulk of training time should be spent watching, taking notes on, rating, and discussing the provided instructional videos. These lessons are all generally higher-scoring on the Core Rubric.

When rating lesson videos, observers should:

• Independently watch, take notes, assign ratings, and record their evidence supporting those ratings
• Discuss their ratings and rationale in small groups or with a partner
• Compare their ratings and rationale to the normed ratings through facilitated discussion
• Ask questions and identify areas of misalignment between their ratings and the TNTP normed ratings

Keep in mind that practice rating high-performing lessons is not enough to truly understand how to use the rubric. Observers must also practice observing lessons of different grade levels, subject areas, and performance levels. Take advantage of TNTP’s large video library to develop this portion of your norming program:

Math Videos

English Language Arts Videos

Science Videos

Social Studies Videos

Part 3: Additional Practice

Practice rating lessons of varying performance levels will help observers solidify their understanding of the Core Rubric and their ability to assign accurate, consistent ratings. We’ve found that differentiating between the lower performance levels (e.g., a strong 2 vs. a weak 3) is much harder than recognizing strong performance, so we strongly recommend at least 8 to 10 hours of practice with our videos or in-classroom experiences.

These videos do not include TNTP’s normed ratings, so you’ll need to assemble a team of instructional leaders to develop normed ratings and a rationale for the videos you select. This is a great opportunity for instructional leaders to come together around a common definition of strong practice before sharing with trainees.

Before TNTP observers can conduct evaluative observations, they must meet our established norming threshold of 75 percent “within one” alignment and 50 percent “exact match” alignment, which we call the 75/50 rule. This means that when an observer generates four ratings on TNTP Core, three of those (or 75 percent) must be no more than one rating level removed from the master rating. Of those three, at least two (50 percent) must be an exact match. Keep in mind that we apply this standard across all of an observer’s practice ratings, not individual ratings.

Example: Applying the 75/50 Rule

Ana is facilitating a norming conversation with Amy and Jeff on a third-grade math lesson. Here’s how their ratings compare to the master rating:

Jeff matched the master rating three times and was one removed from the master rating on Essential Content. Amy matched the master rating on Demonstration of Learning, was one removed on Culture of Learning, and two removed on Essential Content and Academic Ownership. Their norming results:

Therefore, Jeff is normed on this video and Amy is not. This example is based on an individual rating, but for inter-rater reliability, we would normally look at their average ratings throughout initial training.

If you plan to use TNTP Core for teacher development, you may need different metrics for success, like how frequently an observer can identify a teacher’s biggest opportunities for growth. Whatever your goal, a clear bar for success is critical in ensuring your staff collect valid, reliable observation data.

Based on our years of experience norming evaluative observers at TNTP, here’s what we recommend for facilitating a good conversation:

Preparation is key. Review observers’ ratings in advance and strategically plan your conversation. Is there a particular performance area in which a large number of observers are misaligned? Look for patterns in their rationale that surface incorrect interpretations of the rubric. Those misunderstandings should be the meatiest part of your discussion.

Anticipate the devil’s advocate. Explaining why the rating is not one higher or one lower than the master rating is often just as powerful as explaining the rationale for the master rating. Be prepared to talk about both the disconfirming and the confirming evidence.

Facilitate, don’t dictate. Norming should be a discussion, not a presentation: Observers should be active participants in their own learning, sharing their rationale and pushing one another to support their thinking with objective evidence. Listen closely to observers’ justification for their ratings, especially when they don’t align, to better understand their misunderstanding.

Look for the root cause. Misalignment happens for two main reasons: 1) observers collect insufficient or biased evidence, and 2) they misinterpret the performance descriptors or essential question. Listen closely to the evidence and rationale observers share to diagnose the issue.

Keep the boat on course. Allow your observers to actively drive the conversation, but don’t hesitate to reinforce the right answer. Be prepared to fully explain and justify the master rationale.

Ensure that “right is right.” Push observers to justify their ratings and rationale with objective evidence from the lesson. Avoid assumptions, projections, or other assertions unsupported by student or teacher actions.

Summarize and generalize. The goal is to ensure that observers walk away from a norming conversation with a clear understanding of how to replicate key judgments in new situations.

Norming doesn’t end after initial training! TNTP observers have at least three to four “checkpoints” over the year to practice so their observation scores don’t begin to inflate or deflate over time. Ongoing norming can happen through independent practice on normed instructional videos, live co-observations with instructional experts, or group norming exercises. Whatever the format, the more opportunities observers have to use the rubric and discuss their rationale, the better the quality of their observations will be.

For additional information on developing a strong training/norming program, we recommend reviewing the resources linked below. These articles are based on learnings from the Measures of Effective Teaching Project, sponsored by the Bill and Melinda Gates Foundation and published in 2012.

Joe, J., Tochi, C., Holtzman, S., & Williams, J. (2013). Foundations of observations: Considerations for developing a classroom observation system that helps districts achieve consistent and accurate scores. Princeton, NJ: ETS.

Kane, T. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle: Bill & Melinda Gates Foundation.

McClellan, C., Atkinson, M., & Danielson, C. (2012). Teacher evaluator training & certification: Lessons learned from the Measures of Effective Teaching Project. Teachscape.