Do claims of Herbert’s regression have a leg to stand on? – Part 1

For my first post here I’ve decided to address some of the specific claims made in a recent PFF article by Eric Eager indicating that Herbert will regress, or is at least a prime regression candidate. The whole article is fairly questionable, but I decided to first tackle what I believe is the most misleading claim of the whole thing.

Young QBs that undergo a play-caller change fare worse

Eager writes that QBs who have undergone play-caller changes fare worse than those who do not; ergo Herbie is in trouble (below). Its a point he brought up on his podcast as well.

Let me go through this claim to show you just how bad this is. First up:

Do QBs who undergo coaching changes do worse?

Strictly speaking, yes. It is true that QBs who undergo a coaching change (blue) really do, on average, fare worse than those that don’t (orange). There’s just one big problem when inferring the casual relationship…

2. Selection bias. In short, selection bias happens when you study the effect of two conditions/interventions but groups were different from the get-go (see below; and here). A real life example would be concluding that diets make you fatter because people on diets tend to weigh more, or medicine makes you sick because only sick people take medicine. Like the decision to diet or take medicine, retaining/firing your play-caller after your QB’s rookie season is not some random event, but is instead influenced by how the rookie QB played.

Below are the EPA and PFF grades of rookies QBs (2010-2019) who have undergone play-caller changes (blue) vs. ones that did not (orange). Keep in mind this is before they underwent that coaching change. You can see a massive and statistically significant difference between those groups. The difference between these groups pre-dates the thing that Eager is attributing as the cause of the difference. In fact, the difference is bigger before the coaching change than it is after.

Also, look at how much of an outlier Herbert is! To give some context to how much Herbert stands relative to his group: his EPA (+0.164) is slightly closer to the best season in the history of the NFL (+0.409; 2004 Manning) than it is to the average season for a rookie QB who changed play-callers (-0.09). Herbert’s PFF grade of 80.1 is closer to a perfect 100 that no player has ever achieved than the 57.6, which is the average grade for rookie QBs who changed play-callers. Even if the rest of the analysis were fine (which its not), it would be a problematic to assume that he’ll act the same as a group that he stands so far apart from.

3. Correcting for selection bias further exposes how bad this point is. We can try to get around selection bias issue by further separating based on whether they were good or bad as rookies (here I chose EPA above or below 0.0). This allows us to do something closer to an applies-to-apples comparison (i.e. good rookie with a coaching change vs good rookie without one). You can see that the good rookies tend to do well whether or not they have a coaching change (green vs. blue). Bad rookies tend to go on to do poorly whether or not they had a coaching change (orange vs. purple), but close the gap a bit by taking big steps not quite seen with the good rookies.

4. Bad logic is a two-way street. If we are to claim that the dataset doesn’t suffer from selection bias (it does), then we can say these two groups are the same before a play-caller change (they aren’t), and we can compare the effect of play-caller changes (we shouldn’t).

You can see that both groups improve by both metrics, but on average, QBs who change player callers improve more. This is especially true of EPA, where the improvement in rookies with a QB change is more than twice as big (+0.099) as the ones with stable coaching (+0.036). The improvement in the first group is statistically significant, the improvement in the 2^nd group is not. These data suggest that a play-caller change boosts productivity above the modest improvement expected based on continuity. Again, I think that this bad science and that this effect has more to do with pre-existing differences but Eager can’t make that claim without undermining his initial point.

You seem to be telling me that how well a QB does as a rookie tells me how the his career will go regardless of what happens to him next. Is that right?

Not quite, but sort of. Implicit in what I told you above is the notion that bad rookie season = bad career and that the coaching change is just noise: an effect of that rookie season, not a cause of what follows. To show you how much that rookie season can tell you I’ve plotted rookie season EPA vs career (excluding rookie season) EPA. Now that’s a damn tight correlation. That R² indicates that 59.1% of the career EPA variance can be explained by how they did their rookie season. I think we kind of already knew that given how often bad rookies fail to turn things around and good rookies tend to continue to excel.

tl;dr – Bad QB play appears the cause, not the effect of coaching changes. The difference between these groups started as rookies pre-dates the coaching changes. Rookie seasons are very predictive of career EPA, and bad rookies tend to have bad careers irrespective of whether their team changes play-callers. Good rookies tend to have good careers. Going back to the examples from earlier, what this article is doing is akin to saying diets make you fat and medicine makes you sick.

I’d also point out there is some ,mild cherry picking used exaggerate the difference between QBs who do and do not change teams. The article gives every example of a guy that underwent a play-caller change, yet when it comes to the other guys he says, “the group of guys who didn’t change coordinators includes Cam Newton, Patrick Mahomes, Deshaun Watson, Carson Wentz, Jameis Winston, Ryan Tannehill, and Josh Allen”, omitting Locker, Ponder, RG3, Kyler, Bridgewater, and EJ Manuel even though they did not undergo a coaching change. The difference between the groups is obvious whether or not the full list is included, but the difference is reaches statistical significance when using the cherry picked subset. So it wouldn’t be incorrect to say the premise is wrong from the get go, though I think that’s being a bit of a stickler.

I want to be clear though, these results also don’t mean Herbert is immune from regression and that he’s set to have a good career. Only time will tell. 59.1% of the career EPA variance can be explained by how well they did their rookie season, but that also means another 40.1% remains unexplained. That 40.1% can be the difference between an all-time great and a guy who flamed out. The person selling you faulty logic above believes they explain that 40.1%, but call me incredulous.

*Side note: I looked at the list prior to 2010 and its interesting to note how much rarer it was that play callers were fired after a rookie QB was drafted. Here’s the list of 1st round QBs since 2002 who kept their play callers after their rookie year: Mark Sanchez, Matt Ryan, Matthew Stafford, Josh Freeman, Joe Flacco, Vince Young, Big Ben, Rivers, Eli, Brady Quinn, Jamarcus Russell, Matt Leinart, Jay Cutler, Jason Campbell, Carson Palmer, Byron Leftwich, Kyle Boller, and David Carr.

Here’s the much shorter list of guys that changed play callers after their rookie seasons: Aaron Rodgers, Alex Smith, JP Losman, Rex Grossman, Joey Harrington, and Patrick Ramsey.

Now the split is nearly even. My guess is that with the rookie wage scale starting in 2011 it became more important to capitalize on a rookie QBs depressed contract and teams started cycling through OCs and HCs much quicker.