Page 34 - EETimes Europe June 2021

34 EE|Times EUROPE

AV Safety-Report Scorecard Reveals Gaps in Information

(Source: Safety Self-Assessment Report)

EE Times examined every VSSA filed by AV developers on the NHTSA what companies are doing about safety.” He calls it “#ISOwashing.”
website. Each company talks about “saving lives.” They state that “win- Waymo, for example, makes no reference to standards in its Septem-
ning the public trust” is important. Each company says “transparency” ber 2020 Safety Report but does promote its System Safety Program,
is its goal in issuing the safety report. stating, “As the first company to complete a fully self-driving trip on
Because this sort of self-congratulation lacks rigor, we decided public roads in 2015, we have written our own playbook at Waymo.”
to create a scorecard (bit.ly/34rDiIA) to discern, if possible, what In an accompanying document, “Waymo’s Safety Methodologies
each company that has filed a VSSA really means by “taking safety and Safety Readiness Determinations,” published in October 2020, the
seriously.” Our examination placed heavy emphasis on companies’ company cites a host of standards and explains why it is not entirely
approaches to industry standards such as ISO 26262, ISO 21448, and relying on them. On ISO 26262, for example, it states, “ISO 26262 has
ANSI/UL 4600, because each standard has a role to play, and all are provided significant insights for Waymo’s hazard analysis processes.
applicable to any autonomous vehicle. To derive the VSSA Score for However, Waymo does not rely strictly or exclusively on ISO 26262’s
each company listed, we calculated its Glossy Pix Ratio (GPR) — the principles, which are not a perfect fit for a Level 4 ADS (Automated
ratio of the number of pages in the report devoted to promotional pho- Driving System).”
tography against the number of pages devoted to text — and counted Asked to unpack this, Koopman said, “It’s important when interpret-
the number of times the company’s report includes the word “stan- ing statements like this to consider what it says and what it doesn’t
dards” or mentions a particular standard. We found that a report with a say. Waymo appears to say they are mindful of the section of ISO 26262
higher number of pages and a high GPR might contain less information Part 3 that covers Hazard and Risk Analysis. However, because that’s
than a report with fewer pages but a lower GPR. not a perfect fit, they do not rely on it. Unsaid is whether they have
Three of the 24 safety reports made zero mention of the industry considered other parts of ISO 26262. Also unsaid are the principles they
standards. The filing companies were Zoox (now a part of Amazon), actually rely on, since they don’t seem happy with those in the well-
Local Motors, and Apple. At the other end of the spectrum, European established ISO 26262 functional safety standard.”
carmakers BMW, Mercedes-Benz, and Bosch are heavy promoters of He summed up, “The reality is that ISO 26262 should apply to many
industry standards. aspects of their vehicle, as it does to any other vehicle. Since ISO
Among the many variations between those extremes, numerous 26262 is tailorable, it’s hard to understand why they don’t adjust it to
VSSA filers mentioned ISO 26262 but avoided claiming that their vehi- fit their needs.”
cles conform to the standard. Then there’s Argo AI, which wrote in its recently issued Safety
Report, “Our approach to systems engineering is built around two key
#ISOWASHING ISO standards: ISO 26262 and ISO 21448.”
Philip Koopman, co-founder and CTO of Edge Case Research and Koopman acknowledged that “the sentiment seems well-
professor at Carnegie Mellon University, noted that many of the filed intentioned” but added that a reader “has no way of knowing what this
VSSAs “do a sort of standards roll call without really committing to means in concrete terms. It certainly gives the impression that they
do anything specific in a mentioned standard. Vague statements that conform to both standards, but why wouldn’t they say ‘conform’ to it
amount to picking and choosing pieces of standards don’t really tell us if that were true? I’d think they would make the strongest true claim

JUNE 2021 | www.eetimes.eu