Analytics Do Have Limitations

The Steelers have expanded into the analytics field, but Marty Flaherty looks back to remind them not to forget the nuance that's involved.

Sideline choices benefit from long view

Coming off two 8-8 seasons, the Steelers’ leadership has had its acumen questioned in the draft war room, in contract negotiations (or lack thereof), and perhaps most vehemently on the sidelines. Coach Mike Tomlin and General Manager Kevin Colbert have undoubtedly remade the roster so that only a few vestiges of the pre-Tomlin era remain, and most fans expect that the roster decisions will take time to prove their worth.

Few seem to be willing to accord the same patience to the in-game decisions and their role in crafting a young group of players into a cohesive unit.

The jury was in swiftly, for example, on perhaps last year’s most criticized choice – Tomlin’s decision to punch the ball in for a go-ahead touchdown with just more than 1:30 left in the game against the Packers with a chance at the playoffs on the line.

“There are coaching decisions that require more nuance and contextual analysis than strictly appealing to common sense. This is not one of them. A mediocre Madden player would have figured this one out.”

That’s Bill Barnwell of, though many people made the same argument.

The crux of the argument was that kneeling and running the clock down to kick the field goal provided Green Bay a 1 percent chance of winning, while running the ball in for a TD provided the Packers a 9 percent chance to win. And the math is not in dispute. At least not from me.

But the problem with the math is that it’s thinking in a vacuum. Which is to say, it’s like when, in middle-school physics class, we learned how to calculate the speed of a falling object. It’s a simple formula whose chief flaw is that it’s only accurate if you use a vacuum remove the effect of the innumerable air molecules providing resistance against the fall.

In this case, we are pretending that the only three outcomes of the decision are win, lose or draw. We’ve removed all the long-term effects of the decision to allow for a math problem to be effective, making the outcome of one game more important than everything that comes after.

Which is not to say that Tomlin made the right call. There are long-term risks accompanying the potential long-term benefits that we’re conveniently ignoring.

For my part, I don’t see the point of making the playoffs with a defense that can’t stop Matt Flynn from leading a TD drive in the final two minutes of the game, with clock issues negating the benefit of the Packers’ No. 7-ranked rushing attack. The Packers almost scored, and detractors of Tomlin’s decision seized on that as evidence of how dumb the call was. But really, it proved the opposite point, namely that the defense - and the special teams unit that allowed a huge return – needed to be honed under pressure.

If you're a Navy fighter pilot, eventually you have to land your jet on an aircraft carrier. You can spend all the time you want in a simulator, but that only prepares you for the technical aspects of the task.

Same thing with a do-or-die defensive scenario. There is no simulation for crunch-time NFL play. You can't hone “clutch” play in practice. And with Aaron Smith, Casey Hampton, James Farrior, and James Harrison all gone, Brett Keisel, Ryan Clark and LaMarr Woodley on their way out the door, and Ike Taylor and Troy Polamalu not far behind them, the young guys need the opportunity to grow their games. For this reason alone, I'd always score the touchdown in that situation.

But that’s just my take. Like I said, there are long-term risks. An athlete who cracks under pressure might never recover. A loss that could’ve been avoided by the safe play could erode the team’s confidence in their leader. A quarterback convinced his defense is made up of choke artists might be motivated to attempt riskier passes in a bid to avoid punts. Others might weigh these potential negatives and find they outweigh the opportunity for long-term benefits. That’s valid. What’s not valid is to operate as though there are not consequences to a football decision other than wins and losses.

The important takeaway is not that Tomlin’s move was the correct one; it’s that statistical analysis is only truly useful if you acknowledge its limitations. A scientist’s conclusion if posed with the question of what was the proper decision for Tomlin to make would be that the data were insufficient. Barnwell’s conclusion is fit for a mediocre Madden player, for whom the long-term consequences of such a decision don’t exist.

A football coach will tell you that he’s “taking it one game at a time,” and “not looking past” the next team on the schedule. But that’s just a platitude. Every decision has long-term consequences. And the coach’s most important job isn’t to win one particular football game, but to build a stronger unit that produces a better product in terms of teamwork. And there’s very little, if anything, of value in terms of statistical data on how to hone men. A coach is left with little more than his “gut” to determine which is the best course of action.

Whether Tomlin judged correctly in this regard is up for debate. But the debate Barnwell (and many, many others) sought is one without nuance. They achieved this by relegating to the scrap heap a host of other concerns that are pertinent to football decisions. I’m not much of a scientist, but I’m enough of one to know that, in the confines of this vacuous argument, a brick and a feather fall at the same rate.

On the topic of confusing bricks with feathers, the Packers’ win probability after the Steelers scored the go-ahead touchdown was 9 percent. And if Aaron Rodgers (who has as many Pro Bowls as Flynn has wins) had miraculously healed and replaced Flynn for the final 1:30 of the game, the Packers’ win probability would have been 9 percent.

So while there’s no cause to dispute the math that went into calculating the probability, it is, just like g = 9.8 m/s/s, really just a starting point that contributes to a larger understanding. Barnwell treats it like the final, complete answer.

It’s likely that Tomlin will make another call that doesn’t play the probabilities this season. In such occasions, it’s good to keep in mind that the goal is more than winning just one game, and that statistical analysis, while a valuable part of good decision making, shouldn’t be the only element considered.

The NFL is increasingly adopting advanced data analytics, and I understand the Steelers have hired someone to keep pace with that. But it’s important to avoid the pitfalls, most notably the tendency of those who love numbers-crunching to devalue or dismiss anything that can’t be measured or calculated.

Steel City Insider Top Stories