February 8, 2019
On May 7, 2016, Joshua Brown, a Tesla enthusiast, died, when his 2015 Model S in Autopilot mode, collided with a tractor trailer crossing a highway near Williston, Florida. A month later, the agency opened an investigation to throw open the hood of Tesla’s technology and probe its Automatic Emergency Braking (AEB) system design and performance, the human-machine interface issues, modifications Tesla had made to its Autopilot and AEB systems and Tesla crash data.
Six months later, the Office of Defects Investigation closed the Preliminary Evaluation saying they could find nothing wrong – in fact, the agency’s examination of the crash data showed that Tesla’s Autopilot system, beefed up with Autosteer, was a god-damned miracle! By the agency’s calculations, airbag deployments in Tesla vehicles with Autosteer dropped by 40 percent after the installation of the technology – either as original equipment or through an over-the-air software update.
But, a new analysis of the original data by Randy Whitfield, of Quality Control Systems (QCS) Corp., actually shows the opposite. For the subset of vehicles (those that had mileage before and after the installation of Autosteer) in which all of the relevant data are known – the exact mileage at the installation of the technology, Whitfield found that the airbag deployment crash rate increased by 59 percent after Autosteer technology was added. (NHTSA’s Implausible Safety Claim for Tesla’s Autosteer Driver Assistance System)
He concluded: “Our replication of NHTSA’s analysis of the underlying data shows that the Agency’s conclusion is not well-founded.”
Ahem. That understatement doesn’t begin to capture all of the ridiculous, but troubling elements of this story. Sean Kane, president and founder of Safety Research & Strategies, and a frequent collaborator with Whitfield, says the agency’s bad math coupled with its resistance to transparency bodes ill for public safety and push for unregulated autonomous vehicles.
“NHTSA has shown its unwillingness to regulate the safety of the electronics that control modern vehicles and to properly assess potential safety defects in increasingly complex vehicles. This once-storied public health agency, built on epidemiological principles, now resorts to hiding data and promoting the business interests of companies they were entrusted with regulating as it promotes autonomous vehicles and surrenders its oversight role in favor of industry ‘guidance.’” Kane says.
NHSTA Exponent-izes the Tesla Data
In the report NHTSA submitted along with the Closing Resume of its 2016 Tesla investigation, the agency claimed that Tesla’s addition of Autosteer had a significant and measurable effect on safety:
ODI analyzed mileage and airbag deployment data supplied by Tesla for all MY 2014 through 2016 Model S and 2016 Model X vehicles equipped with the Autopilot Technology Package, either installed in the vehicle when sold or through an OTA update, to calculate crash rates by miles travelled prior to [fn. 21] and after Autopilot installation. [fn. 22] Figure 11 shows the rates calculated by ODI for airbag deployment crashes in the subject Tesla vehicles before and after Autosteer installation. The data show that the Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation.
That NHTSA decided to feature airbag deployments as a determinant of the efficacy of autonomous technology was odd, because in its Information Request to Tesla, the agency never asked Tesla for airbag deployment numbers. Nonetheless, for the ensuing year and a half, Tesla dined out on this claim, trotting out the 40 percent reduced crash rates whenever another one of its vehicles crashed. By May 2018, the agency was forced by the journalistic clamor for the basis of this startling statistic, to walk it back. But only Whitfield persisted and succeeded in obtaining the underlying data to reveal how statistically weak the figure was.
Crash rates are born of numerators and denominators. In this case, the numerator was the number of airbag deployments. The denominator was comprised of vehicle miles travelled – which represented the vehicle’s exposure to the risk of a crash, and therefore a scenario in which the airbag might deploy. In order to pin down the denominator, NHTSA needed to know the vehicle mileage at the time Autosteer was installed, but for most of the vehicles in the study, Tesla didn’t provide the exact data.
Whitfield found that “the actual mileage at the time the Autosteer software was installed appears to have been reported for fewer than half the vehicles NHTSA studied.” Of the 43,781 vehicles studied, only 5,714 vehicles – or 13 percent – had complete mileage data and driving experience before and after installing Autosteer.
For the data missing the exact Autosteer installation mileage number, “NHTSA treated the exposure mileage that could not be classified as either before or after the installation of Autosteer as if it were zero mileage,” Whitfield says. “This results in an undercount of the denominators. The problem is the under-count affected the “before” category much more than it did the “after” category.”
The Nearly Two-Year Battle for the Data
It took Whitfield, plus a lawyer, and 641 days to get the data.
Whitfield was suspicious of NHTSA findings, and the lack of back-up data from “one of the most incessantly self-professed data-driven government agencies” and sought to replicate its analysis. On February 24, 2017, QCS filed a Freedom of Information Act (FOIA) request for “all of the mileage and airbag deployment data supplied by Tesla analyzed by ODI to calculate the crash rates shown in Figure 11…” He also asked for any “statistical formulas, models, adjustments, sample weights, and/or any other data or methods relied upon to calculate the crash rates.”
At the end of March, the agency promised to respond by mid-April. Three months later, when no response seemed forthcoming, Whitfield filed a FOIA lawsuit for the data in U.S. District Court in Washington D.C. On July 21, 2017, NHTSA notified Whitfield that it had denied his request, based on two exemptions to the FOIA – Exemption 4, which shields information that could cause competitive harm, and Exemption 5 – which shields an agency’s “deliberative process” from public view. NHTSA tends to hand these out like after-dinner mints. In the past, the agency has misused Exemption 5 to deem any piece of information – a photograph, a number – as a critical part of its deliberative process, and deny a FOIA request. (DOT Settles Lawsuit over Toyota UA Documents, New Congressional Inquiry Raises More Questions)
On September 30, 2018, U.S. District Judge Dabney L. Friedrich denied motions by both QCS and DOT for a summary judgement (a favorable ruling). However, in her 13-page ruling ordering the parties to prepare for further proceedings, Judge Friedrich made it abundantly clear that she found both Tesla’s claims of competitive harm and NHTSA claims of deliberative process to be less than persuasive.
On the matter of the competitive harm, Judge Friedrich scratched her head over Tesla Director of Field Performance Engineering, Matthew L. Schwall’s lengthy December 20, 2017 declaration describing the various ways the data could reveal “proprietary secrets.” She methodically eviscerated Schwall’s five arguments – a shorter version might be: You wrote many, many words. None support your position.
Then, she swept aside NHTSA’s arguments that Office of Defect Investigation’s Jeffrey Quandt used some super-secret deliberative methods that could not be exposed to the light of day:
It thus appears that Quandt performed a straightforward mathematical calculation involving categories of data clearly identified in Figure 11. Based on Figure 11 and his declaration, it appears that Quandt simply divided the total number of airbag activations by the total number of miles driven to determine the average crash rate (per million miles) for select Tesla vehicle models (both before and after Autosteer installation).
Following this judicial beat-down, NHTSA told Tesla that it was rescinding its grant of Confidential Treatment for the data QCS requested and turned it over in late November. (The data provided to QCS by NHTSA is available here.)
The Moral of the Story
Last May, an American Automobile Association (AAA) released the results of its latest survey tracking consumer trust in automotive autonomous technology, and found that it has “quickly eroded. Today, three-quarters (73 percent) of American drivers report they would be too afraid to ride in a fully self-driving vehicle, up significantly from 63 percent in late 2017. Additionally, two-thirds (63 percent) of U.S. adults report they would actually feel less safe sharing the road with a self-driving vehicle while walking or riding a bicycle.”
And stories like this aren’t going to move the numbers upward.
NHTSA owes its public health mission and the driving public its due care and transparency in guiding the transition. Instead, the agency has let the industry auto-steer us towards their next big business model, while cheering from the sidelines.
We can only hope the burns sustained from the exploding 40-percent claim will discourage NHTSA in the future from throwing out statistical spitballs, providing automakers with marketing copy and trying to hide from independent investigators.