An Overview of Draft UL 4600: “Standard for Safety for the Evaluation of Autonomous Products”

By: Dr. Phil Koopman, Co-Founder & CTO, Edge Case Research

This article gives an overview of the current status of the Draft UL 4600 standard that describes a safety case approach to ensuring autonomous product safety in general, and self-driving cars in particular. Information is current as of June 2019.

NOTE: The author is a principal technical contributor to the current draft standard text. This article is the personal viewpoint of the author. While intended to generally reflect the current trajectory of draft standard UL 4600, that draft is subject to change as it undergoes the UL consensus process. This article is presented in the spirit of transparency and inclusion of the greater stakeholder community during the initial drafting process. While the author acknowledges very helpful contributions from other STP members, this document does not necessarily reflect the views of UL or the STP members.

Why Do We Need Another Standard?

Current safety standards provide essential guidance for designing safe vehicles. However, existing standards such as ISO 26262 and ISO/PAS 21448 were envisioned for vehicles that ultimately have a human driver responsible for safe operation of the vehicle. With existing standards, safety is typically achieved via following a specified design process, together with the imposition of specific technical requirements and validation methods. Higher degrees of risk result in more rigorous engineering requirements to ensure appropriate risk mitigation.

The technology in self-driving cars and other autonomous systems exceeds the scope of these and other traditional safety standards. Those standards are necessary, but not sufficient. Something more is required to handle:

· Autonomous vehicle technologies, including machine learning and sensor fusion, that exhibit complex, non-deterministic, and potentially unpredictable behaviors

· The pervasive implications of vehicles not having a responsible human driver

· Changes to the environment that will require continual updates to remain safe

· Fast-paced technology change that will quickly invalidate any inflexible standard approaches

· Ensuring the safety of novel technology for which accepted practices are still emerging

(A revision to ISO/PAS 21448 is said to be expanding its scope to handle some of these areas. That new scope is being taken into account with an aim of avoiding unnecessary overlap in the development of UL 4600. Other potentially overlapping standards efforts are similarly being considered based on available information.)

How Is UL 4600 Different?

Rather than require a particular technical approach, UL 4600 concentrates on ensuring that a valid safety case is created. A safety case includes three elements: goals, argumentation, and evidence. Goals describe what it means to be safe in a specific context, such as generic system-level safety goals (e.g., don’t hit pedestrians) and element safety requirements (e.g., ensure a computing chip produces correct computational results despite potential transient hardware faults). Arguments are a written explanation as to why a goal is achieved (e.g., vehicle-level argumentation that the system can detect and avoid pedestrians, including ones that are unusual or appear in the roadway from behind obstacles, within the limits of physics and subject to the vehicle displaying appropriate defensive driving behavior). Evidence supports that the arguments are valid, typically based on analysis, simulations, and test results (e.g., for a computing chip mathematical analysis of error correction codes combined with the results of fault injection experiments).

The key to the UL 4600 approach is that it is goal based and technology-agnostic. That means UL 4600 requires explaining why the self-driving car is safe without requiring the use of any specific design approach or specific technology use. For example, using LIDAR is not required. Rather, the safety case has to credibly argue that relevant objects will be successfully detected and classified with whatever sensors are installed within the limits of the intended operational design domain. Similarly, there is no fixed limit on the number of road testing miles that must be accumulated before deployment. Rather, the safety case must argue that an acceptably robust combination of analysis, simulation, closed course testing, and safe public road testing have been performed to ensure an appropriate level of system safety for the initial vehicle and each software update.

UL 4600 does not try to invent yet another V-cycle-based engineering process for computer-based system safety. Rather, it serves to take the information produced by existing and potentially new design and validation processes and standardizes how to organize the results into a coherent safety case. That safety case provides credible evidence that a highly autonomous vehicle is indeed appropriately safe for deployment. UL 4600 is specifically designed to work well with existing automotive safety standards such as ISO 26262 and ISO/PAS 21448. However, it is generic enough that it can also play well with other standards as autonomy becomes adopted into other domains. The overarching safety case for the whole system can take inputs from both traditional safety activities and new approaches required to validate novel technical approaches such as machine learning.

Put another way, UL 4600 helps ensure that the safety engineering activities that might be conducted using processes and techniques defined in existing standards actually cover all the bases for autonomous vehicle safety. Much of the standard consists of extensive lists of items that need to be considered in deciding if the system is appropriately safe. (Some list elements can be waived if inapplicable, but the point is to have a way to make sure that the design team didn’t miss something that was reasonably foreseeable.)

What Value Does UL 4600 Provide?

The key reasons for self-driving car design teams to use UL 4600 are:

· UL 4600 supports state-of-the-art safety case approaches, which permit standardizing an approach to safety while at the same time enabling the use of rapidly evolving technology, tools, and methods. It is both technology neutral and development process agnostic.

· UL 4600 provides a uniform set of rules to help ensure that essential aspects of safety have been thoroughly considered before deployment. This includes strong guidelines as to completeness, classes of hazards that must be considered, and level of detail necessary.

· UL 4600 is specifically designed from the ground up for highly autonomous vehicles that can operate with no human driver and no human safety supervisor. (The implications of not having a responsible human present are profound, and include issues beyond the driving task such as failure management, load/unload operations, and social interaction with other road users.)

· UL 4600 is designed to evolve quickly (for a standard) over time. Within a particular project or company the extensive lists in the standard can be expanded and tailored in response to experience and the needs of the project. At the industry level, UL plans to put the standard under “continuous maintenance” and has a mechanism to permit urgent updates in a few months and periodic updates perhaps yearly. (The actual update timeframe is at the discretion of UL and the STP.) Over time, the standard will incorporate both accepted practices and lessons learned in the normal course of its ongoing maintenance process. Among other things, UL 4600 can serve as a nucleus for sharing safety information across the industry. UL 4600 has impact analysis and safety case configuration management provisions to minimize the burden imposed by standard updates within a flexible grace period.

· UL 4600 uses feedback loops to permit managing the risk of “unknowns.” Let’s face it — autonomous vehicle technology will deploy with significant unknown unknowns. Rather than pretending that the engineers have thought of everything (they haven’t, and they won’t), the standard accommodates the responsible management of uncertainty. It specifically requires mechanisms for collecting and processing field feedback data as well as managing uncertainties, assumptions, and potential gaps in the safety case after deployment. This includes addressing issues related to software updates and product evolution. Such an open process continuously grows the domain knowledge of the community, much like disclosure processes used in the field of aviation.

· UL 4600 provides a well-defined interface for component safety cases. A specific goal is to enable a uniform approach to independent component safety case assessment while protecting component proprietary information. The result is a top-to-bottom vehicle safety case that contains plug-in safety assessment modules for third party components.

· UL 4600 is designed for transparent assessment. The entire standard is written from an assessment point of view to help ensure completeness and clarity. If the designers have produced a valid safety case, there should be no surprises during assessment.

· UL 4600 provides flexibility in assessment independence. Use of credentialed external assessors is recommended, but not required so long as assessor independence and capabilities are credible and documented.

· UL 4600 plays well with existing safety standards (e.g., ISO 26262 and ISO/PAS 21448) while filling potential gaps. (Example: any credit for “controllability” taken in ISO 26262 conformance must be reconciled in some manner against the absence of a human driver to exercise control.)

· UL 4600 enables the use of components developed to other standards and even other domains while providing a proper overarching safety case.

How Does UL 4600 Work?

The current draft standard is more than 200 pages long, so by necessity this is an incomplete summary. At the highest level, developers start with various safety activities being performed as they would be for conventional cars (e.g., in conformance with ISO 26262). Additional analysis and design will be required to handle autonomous driving capabilities (e.g., in conformance with ISO/PAS 21448). Other autonomy safety approaches will likely be required (e.g., to validate the effectiveness and brittleness of machine learning based approaches to perception and prediction). No particular standard is required, but key activities such as identifying hazards must be performed somehow to create the necessary safety case elements of hazard identification, risk analysis, and sufficient risk mitigation.

All of these design and validation activities produce a set of safety goals, fragments of argumentation, and evidence. This information is combined into a safety case. (ISO 26262 already requires a safety case, but UL 4600 provides the details of how to assess a safety case for completeness and validity in the context of autonomous vehicles.) The various clauses of UL 4600 place requirements on the minimum acceptable levels of goal coverage, argumentation validity, and evidence credibility.

Among other things, building a credible safety case includes the following aspects specified by UL 4600:

· accounting for important real-world issues (e.g., visually impaired pedestrians, deferred maintenance, vehicle component failures),

· ensuring internal consistency of the safety argumentation (e.g., can you trace from every hazard via argumentation to sufficient evidence that the hazard has been appropriately mitigated?),

· safely responding to unusual operational conditions given the constraints of the system’s operational design domain (e.g., fire in a tunnel, rain in the desert), and

· ensuring that the entire vehicle lifecycle and supply chain has been considered (e.g., if a claim is made that a camera will be clean due to the use of a spray wash, have you accounted for a faulty low-fluid sensor, or the possibility of fluid that has insufficient anti-freeze for winter?).

While no standard can think of everything, the provided starter lists are robust and provide a framework for each developer and the entire industry to accumulate lessons learned as the technology matures.

To support the automotive supply chain, component suppliers can create safety case fragments with a defined safety case interface. This is not an electronic component interface, but rather a set of capabilities and assumptions that can be independently assessed along the lines of a hardware or software Safety Element out of Context. The vehicle-level safety case can then take credit for the component’s capabilities as evidence so long as the assumptions are argued to be valid. Support for component assessment will be especially important when integrating third party sensors and actuators as well as adding autonomy kits onto automotive platforms.

Development teams are expected to conduct a preliminary safety case assessment on their own. (The assessment criteria are in the standard itself.) Then, an independent assessor checks to make sure the the internal assessment didn’t miss anything in ensuring that the safety case conforms to the standard. In principle there should be no surprises from the independent assessment, but having such an independent check and balance is a crucial aspect of ensuring safety over the long haul. Independent assessors can be external specialists, but can also be internal if sufficient independence and capability are argued and documented as part of the assessment report. Part of the feedback loop process is that assessors have a responsibility to feed-back suitably scrubbed lessons learned for the improvement of the standard over time.

The June 2019 draft of UL 4600 includes the following chapter topics. Note that some essential topics are included as sections within these chapters (for example, component qualification, map data, and relationship to cybersecurity are included as sections within these chapters):

1. Scope
2. References
3. Terms and Definitions
4. Safety Case and Argumentation
5. Risk Assessment
6. Interacting with Non-Driver Humans
7. Autonomy Pipeline and Machine Learning
8. Software and System Development Processes
9. Dependability and Redundancy Management
10. Data and Networking
11. Verification, Validation, and Testing
12. Tool Qualification and Off-the-Shelf Components
13. Lifecycle and Operational Concerns
14. Maintenance
15. Safety Metrics
16. Assessment

In keeping with UL 4600’s high level approach, these chapters do not specify phases of an engineering process. Rather, they place requirements on the minimum acceptable contents and relationships of work products from engineering processes to result in a well formed, credible safety case for an autonomous vehicle.

While human factors involved in the dynamic driving task are out of scope for UL 4600, the safety of the vehicle side of a human+autonomy system is squarely within scope. That means that while the sweet spot of this standard is SAE Levels 4 and 5 (fully autonomous driving operation), the standard can provide partial support for lower autonomy levels if used in conjunction with other materials covering human driver safety and driver/machine interface safety. In particular, UL 4600 addresses the machine side of the human/machine interface for safety supervision, human driver takeover, and teleoperation.

What Happens Next?

A comprehensive draft (200+ pages in length) was provided to the Standards Technical Panel (STP) in May 2019 for review and discussion. The STP is the balanced body of industry representatives that ultimately votes on whether to accept a final version of the standard. STP members currently include categories such as: traditional OEM, autonomy full stack provider, first tier supplier, integrated circuit manufacturer, insurance, government research, university research, government regulation, legal, tools/simulation, international standard liaison, consumers, and third-party assessors. STP members come from around the world including North America, Europe, and Asia.

The STP physically met to discuss the initial draft and any high level issues on June 12–13, 2019.

A revision provisionally addressing initial STP comments will be reviewed by the STP members during Summer 2019. Additional revision and review cycles lasting 30–60 days each might be necessary.

Once STP comments have been substantially addressed, a version will be made available to additional stakeholders (in UL terms, a preliminary review version). Any responsible party can request to be a stakeholder. For practical purposes once distributed to stakeholders that version of draft UL 4600 will be a public draft standard (with distribution limited to registered stakeholders and those who as a result of the release document request to review and comment). We hope this will happen by the end of summer 2019, although that is an aggressive goal.

Registered stakeholders have the right to comment, but do not have voting rights. After one or more rounds of considering STP and stakeholder comments, the STP will enter a balloting phase that, if all goes well, will result in a public standard in late 2019 or early 2020. (Timing depends upon the number and types of stakeholder comments received.)

UL 4600 is being developed to follow an intended path of first an ANSI/UL standard and then potentially an international standard. The Underwriters Laboratories is open to and actively entering discussions of potential collaboration and coordination with other standards bodies.

The initial version of UL 4600 concentrates on autonomous passenger vehicles (e.g., SAE Level 4 and 5 self-driving cars and similarly capable cargo vehicles). It can be adapted for use with conventional vehicles and driver assistance approaches via appropriate tailoring, especially for assessment of component safety cases. As written, it will provide significant content that will be useful to other types of autonomous vehicles such as last-mile delivery robots and non-automotive application domains. Over time, additional domain-specific standards can be created that tailor UL 4600 to specific application domains.

Apply To Be a Stakeholder:

Contact Ms. Deborah Prince, the program manager at UL: Deborah.Prince@ul.org to request a Stakeholder application form. If approved, you will have access to the draft standard when it is released for stakeholder review and comments.

More resources:

Visit http://UL4600.com
(That web site is provided as a convenience and is not an official UL site.)