NHTSAball: How the Agency Plays the Game

Moneyball has opened in the movie theaters, starring Brad Pitt as the redoubtable Billy Beane, general manager of the Oakland Athletics. Beane had been a future baseball superstar who washed out after three seasons of bench-warming and bouncing between Triple-A and major league teams. He was, however, that rare individual who resisted his own PR, which was certain and glowing, and the perfect foil for the movie’s real subject: baseball statistics and how numbers could guide a poor team like the A’s to the playoffs more regularly than the conventional wisdom said was possible.

The A’s, as Michael Lewis, author of the book on which it is based describes the team, was populated with Major League’s castoffs – catchers and pitchers who looked more like beer-swilling bowlers, college prospects that the talent scouts had deemed too cold to bother with, and a computer nerd for an assistant GM.  But Beane knew from his own field experience just how wrong Major League Baseball could be in assessing players. He chose, instead, to exploit a different set of metrics developed by a passionate fan who happened to be statistically apt, but shunned by the rest of Beane’s peers. Using this recipe, he cobbles together victory after victory.  The book roughly follows the A’s during the 2002 season, from the June amateur player’s draft to the historic September 4th game, in which the A’s won their 20th consecutive game – breaking a 102-year old American League record.

Among the book’s heroes is Bill James. James is the quirky inventor of Sabermetrics, the statistical analysis of baseball data, used to better understand success and failure in the game. James had a restless, numerically inclined mind and, it would be fair to say, an obsessive interest in baseball. The marriage of his aptitude and his interests produced a series of abstracts and books on baseball metrics, beginning in 1977. At first, James was ignored by all but a handful of fans. But, eventually, he gained a following. And in 1997 one of them became the manager of the Oakland A’s.

If the narrative has a villain, it’s Major League Baseball itself. Wholly in the thrall of its own myths, this billion dollar enterprise, when it bothered with data at all, used it incorrectly. Rich teams could skim the obvious cream of talent, but beyond buying wins, hadn’t a clue how it was really done.

Lewis’s point was that even with huge sums of money at stake, those most vested in the outcome can be utterly blinded by their own subjective biases. Once biases are institutionalized, they become facts, stubborn in the face of new or competing or different information. Those who can’t even identify the data that should be informing critical decisions – let alone using it properly – are bound to make expensive mistakes.

So what’s The Safety Record’s point? We saw a number of striking parallels to the way NHTSA operates, when it comes to numbers.  The agency is, ostensibly, data-driven and science-based. Hell, it slaps that phrase on everything from Powerpoint presentations to appropriations requests. But there’s plenty of evidence that the agency dismisses – out-of-hand – outside information; that long-held beliefs hamper its data analyses; that it pays way too much attention to data that is not useful and fails to collect data that is.

Moneyball: For example, here Lewis characterizes James’ assessment of MLB’s metrics:

“Worse, baseball teams didn’t have the sense to know what to collect, and so an awful lot of critical data simply went unrecorded.”

NHTSAball: There’s lots of data NHTSA doesn’t collect. For example, the agency has promulgated a recall system for tires that is predicated solely on the Tire Identification Number. Years of rulemaking were devoted to debating the format of the TIN, popularly known as the DOT code, an 11-character, alpha-numeric code that represents the plant, size, date of manufacture molded into the sidewall of a tire. The aim of the TIN was to identify a recalled tire and it is the only definitive measure to do so.  Does NHTSA require tire makers who recall their products to list the range of TINs in the recall population? No, it does not. Some manufacturers report it; others don’t. NHTSA’s good, either way. The DOT code would be a nifty little data point in any tire-related fatality in the Fatality Analysis Reporting System. The agency could use such data to identify tire failure trends, but it remains information uncollected. And while we’re on tires, NHTSA doesn’t require manufacturers to report tire claims if the tire is older than five years – so no data, no problem for the aged tire issues that took center stage following Firestone recalls in 2000 and 2001 and remain a constant source of catastrophic crashes. Continue reading