Tesla Autosteer ability to reduce accidents challenged by report
EV pioneer’s Autopilot system might not have reduced serious accidents by as much as previously stated by US road safety agency, according to an investigation into the claims
The regime for assessing the safety of self-driving car systems is under scrutiny after consultants discovered errors in an analysis of Tesla crashes by a US government agency, which then delayed release of the data for two years in an apparent attempt to divert attention from the issue.
Quality Control Systems Corp (QCSC) made a deep-dive into data that purported to back up an impressive claim by NHTSA, the powerful US road safety body, and Tesla that the Autosteer system in Autopilot had reduced serious accidents by 40% — a game-changing improvement.
Tesla boss Elon Musk tweeted the results as NHTSA’s findings were released in January 2017.
But QCSC found that the NHTSA analysis failed to take into account all the mileage driven by the 43,781 vehicles studied. In fact NHTSA only used mileage data for 14,791 vehicles. As a result, the crash rate before Autosteer was inflated, leading to the wrong conclusion, says QCSC.
“The importance of this research goes well beyond the specific issues addressed in our statistical analyses,” said QCSC. “The larger question is whether the field experience of autonomous vehicles and advanced driver-assistance systems will be fairly and transparently assessed by public officials.”
When QCSC, which specialises in data analysis as a business tool, approached NHTSA for access to the publically-funded research, they were refused. The log-jam took two years to break, only after QCSC threatened court procedings to obtain the data via a freedom of information request.
In response to a request for a comment from Autocar, the Washington-based NHTSA released a limited statement: “The agency is reviewing the report released by Quality Control Systems Corp. with interest and will provide comment as appropriate.”
QCSC’s report is titled “NHTSA’s Implausible Safety Claim For Tesla’s Autosteer Driver Assistance System” and looked at data for airbag deployments of MY2014 – 2016 Model S and MY2016 Model X models equipped with Tesla’s Autopilot.
QCSC’s analysis of the data, contained in a 24-page report, found that airbag deployments actually increased from 0.76 million to 1.21 million, a 59% rise, rather than decreased.
“Before and after comparisons of the resulting crash rates are unbiased by missing data for exposure mileage because there are no missing data in this subset of the data,” said QCSC in its report.
When NHTSA analysed airbag deployment and mileage travelled data supplied by Tesla, NHTSA came to the conclusion that there were 1.3 deployments per million miles before Autosteer and 0.8 million after – a 40% reduction.
NHTSA’s findings were widely reported in January 2017, partly because of the fatal accident in Florida six months earlier when a Model S operating with Autopilot ploughed into a truck, having failed to detect the vehicle making a turn across the car’s path.
Autopilot is a Level 2 advanced cruise-control self-driving system that allows the driver to temporarily relax their grip on the steering wheel. Autosteer is the lane-keeping part of the Autopilot system, which also facilitates lane changes when the indicator is operated.
Introduced in 2014, Autosteer was enhanced in 2016 with an upgrade. Significantly, it means a lane change can be executed with less oversight by the driver. Ultimately Tesla aims for automatic lane-changing manoeuvres in Level 3 or 4 autonomy.
In response to the report, a Tesla statement read: “QCS’ analysis dismissed the data from all but 5714 vehicles of the total 43,781 vehicles in the data set we provided to NHTSA back in 2016. And given the dramatic increase in the number of Tesla vehicles on the road, their analysis today represents about 0.5% of the total mileage that Tesla vehicles have traveled to date, and about 1% of the total mileage that Tesla vehicles have traveled to date with Autopilot engaged.
“NHTSA’s original report not only indicated that “Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation”, the agency also concluded that it “did not identify any defects in the design or performance of the AEB or Autopilot systems,” nor did it find “any incidents in which the systems did not perform as designed”. They also found that “the potential for driver misuse was evaluated as part of Tesla’s design process and solutions were tested, validated, and incorporated into the wide release of the product.”
“Our own vehicle safety data for Q3 and Q4, which includes data from roughly two billion miles driven in Tesla vehicles, shows that drivers using Autopilot were significantly less likely to be involved in an accident than those driving without using Autopilot.”
Source: Autocar Online