How Do You Write Testable Requirements for an AI/ML Component in a Safety-Critical System?
This is one of the hardest open problems in modern systems engineering. Traditional requirements—“the brake actuator shall respond within 50 milliseconds”—are precise because the behavior is deterministic and bounded. A learned model does not work that way. Its outputs are probabilistic. Its failure modes are not enumerable at design time. Its performance degrades in ways that depend on the statistical distribution of inputs it actually receives versus the distribution it was trained on. Writing a requirement that is specific enough to verify, without being so restrictive that no real model could pass it, is genuinely difficult.
This article walks through what the emerging regulatory frameworks say, what the technical community has converged on, and how to structure AI/ML requirements so that they can actually drive a verification program.
Why Standard Requirements Language Fails for ML
Consider a perception system for an autonomous ground vehicle. You might initially write:
The object detection subsystem shall correctly identify pedestrians in the field of view.
That sentence contains no verifiable information. “Correctly” is undefined. “Identify” is undefined. “Field of view” is a geometric claim but tells you nothing about lighting conditions, occlusion, sensor noise, or object distance. You cannot build a test suite against that requirement, and a regulator reviewing your evidence package will immediately recognize that you haven’t constrained anything meaningful.
The problem runs deeper than prose quality. ML components have three structural properties that conventional requirements language was not designed to handle:
- Stochastic outputs. The same input can produce different outputs across inference runs, depending on hardware, floating-point behavior, or model quantization.
- Distribution dependency. Performance on a test set is not performance on the operational domain unless you can prove the test set is representative—which requires characterizing the operational domain in the first place.
- Epistemic opacity. You cannot exhaustively enumerate the failure modes of a neural network the way you can enumerate the state space of a finite state machine.
Acknowledging these properties is the starting point for writing better requirements.
What the Regulatory Frameworks Are Saying
The DO-178C ML Supplement
DO-178C itself does not address ML directly—it was designed around software with verifiable structure. RTCA SC-205, working alongside EUROCAE WG-114, has been developing a dedicated supplement specifically for machine learning components in airborne systems. As of mid-2026, the document is in advanced draft review.
The supplement’s key move is to reframe the verification unit. Rather than verifying a model against a functional specification, it directs teams to define operational performance requirements tied to specific operational scenarios and then verify that the model meets those bounds within each scenario class. This sounds incremental, but it forces teams to do something concrete: enumerate the scenarios that matter for safety, assign measurable performance targets to each, and collect scenario-stratified test data that can serve as evidence.
The supplement also introduces the concept of a performance monitoring requirement—a requirement that the deployed system must detect when its inputs fall outside the operational design domain and respond with a defined mitigation behavior (hand off to human, flag for review, invoke a backup system). This is the regulatory formalization of out-of-distribution detection.
The FAA Machine Learning Roadmap
The FAA published its ML Roadmap in 2021 and has updated it through subsequent policy statements. The roadmap does not prescribe a single compliance path but identifies the properties that any acceptable ML-based system must demonstrate. The properties relevant to requirements writers are:
- Bounded performance: the system must have quantified performance bounds, not just “good” performance.
- Explainability sufficient for oversight: a human responsible for the system must be able to understand why a particular output was produced well enough to judge whether the output was reasonable.
- Monitored deployment: the operational performance must be tracked against the approved bounds, and a process must exist to respond when it degrades.
For requirements writers, “bounded performance” translates directly into the need for numeric accuracy floors, confidence interval requirements, and scenario-stratified metrics.
ISO/TR 4804
ISO/TR 4804 addresses safety and cybersecurity for automated driving systems and is more immediately actionable than the airborne frameworks for many ground vehicle programs. It explicitly requires that perception and decision-making systems have defined operational design domain (ODD) boundaries, and that the behavior of the system at ODD boundaries be specified and verified.
The TR’s approach to ML components is to treat ODD boundary detection as a functional requirement: the system shall detect when it is approaching or has exceeded its ODD and invoke defined safe state behavior. This is a testable requirement. You can inject ODD-boundary scenarios, verify that the detection fires, and verify that the safe state behavior executes correctly.
Four Techniques for Writing Testable ML Requirements
1. Accuracy Floor Requirements
An accuracy floor is the minimum acceptable performance metric on a defined evaluation dataset for a defined scenario class. A correctly formed accuracy floor requirement looks like this:
The pedestrian detection function shall achieve a true positive rate of ≥ 0.95 and a false positive rate of ≤ 0.02, evaluated on the program-defined Scenario Class A dataset (urban, daylight, ≤ 30m range, occlusion < 30%), using the measurement protocol defined in [Verification Plan reference].
Every noun in that sentence is defined or referenced. You can build a test against it. You can fail it. You cannot game it by changing the dataset without also changing the requirement.
The parenthetical scenario description is not decoration—it is the requirement. The same model might achieve 0.95 TPR in daylight and 0.61 TPR in low-light rain. If your safety analysis shows that the system operates in low-light rain, you need a separate accuracy floor requirement for that scenario class, or you need a requirement that the system must not operate without a specified confidence threshold in that condition.
2. Uncertainty Quantification Requirements
Uncertainty quantification (UQ) requirements constrain what the model must do with its own confidence. A model that is wrong but confident is more dangerous in a safety-critical system than one that is wrong but flags its uncertainty. A UQ requirement looks like this:
For any output from the object classification function, the system shall produce a calibrated confidence score. Calibration error (measured as Expected Calibration Error on the program-defined evaluation sets) shall be ≤ 0.05. The system shall flag any output with confidence below 0.80 for supervisory review or escalation to the defined fallback behavior.
This requirement has two parts: a calibration quality requirement (you can measure ECE) and an operational threshold requirement (you can test that low-confidence outputs trigger the right behavior). Both are verifiable.
3. Out-of-Distribution Detection Requirements
OOD detection requirements specify what the system must do when its inputs are statistically unlike its training distribution. These requirements are technically challenging to write because “out of distribution” is not a binary property—it is a continuum—but practical formulations exist:
The OOD detection function shall flag as anomalous any input for which the Mahalanobis distance from the nearest training distribution cluster centroid exceeds [threshold value], as computed by the approved OOD detection module. Flagged inputs shall trigger [defined safe behavior] within [latency bound]. The false anomaly rate on the in-distribution evaluation set shall be ≤ 0.01.
The threshold value must be determined through analysis and documented in the design record. The false anomaly rate requirement is what keeps the OOD detector from being so sensitive that it disrupts normal operation.
4. Scenario-Based Performance Bounds
Scenario-based performance bounds are the synthesis of the above techniques into a structured requirements architecture. Instead of one monolithic accuracy requirement, you decompose the operational design domain into scenario classes, each with its own performance bounds.
The scenario classes should be derived from the system-level safety analysis. If your hazard analysis shows that misclassification of a specific object type in wet conditions is a contributing factor to a top-level hazard, that combination—object type × wet conditions—is a mandatory scenario class. Every scenario class with a safety connection must have explicit accuracy floor, UQ, and OOD behavior requirements.
This decomposition is also what makes the verification program tractable. You collect test data stratified by scenario class, run verification against each class independently, and produce a scenario-stratified evidence package. Regulators and certification authorities can then evaluate coverage directly.
Traceability Is Not Optional
Writing good individual requirements is necessary but not sufficient. The requirements must be connected to each other and to the verification evidence in a way that supports audit and ongoing configuration management.
The traceability chain for an ML component in a safety-critical system runs at minimum from:
Safety goal → System-level requirement → ML function requirement (accuracy floor / UQ / OOD) → Scenario class definition → Test dataset specification → Verification results
Each link in that chain must be explicit. If your test dataset changes—because you collected more data, or removed a corrupted batch—the change must propagate through the chain so that affected verification evidence is flagged for re-review. If a safety goal changes during design refinement, all downstream ML requirements must be re-evaluated.
In practice, teams that manage this traceability in document-based systems—spreadsheets, Word documents, manually maintained RTMs—consistently encounter the same failure mode: the chain breaks under configuration pressure, and no one knows it until the certification review. The chain looks correct at a snapshot in time but has been silently invalidated by a sequence of individual changes, each of which seemed locally reasonable.
How Modern Tools Support This Structure
Flow Engineering was built around exactly this traceability problem. Its graph-based model represents requirements, test scenarios, verification evidence, and their relationships as nodes and edges in a live model rather than as rows in disconnected documents. When you modify a performance bound—say, tightening an accuracy floor because a safety analysis identified a previously uncharacterized hazard—the tool immediately surfaces all downstream scenario definitions and test plans that are potentially affected. The team can see the blast radius of the change before executing it.
For AI/ML requirements specifically, Flow Engineering supports the layered structure described above: system-level safety requirements linked to ML function requirements, linked to scenario class definitions, linked to test dataset references and verification evidence. Teams working on DO-178C supplement compliance or ISO/TR 4804 programs have used this structure to produce the kind of scenario-stratified evidence packages that regulators are starting to expect.
The platform’s deliberate focus is on requirements and traceability. It does not provide ML training infrastructure, dataset management, or model evaluation tooling. Those capabilities live in the ML development environment (MLflow, DVC, Weights & Biases, or similar). Flow Engineering connects to that environment at the requirements and evidence level—pulling in test results as verification artifacts linked to specific requirements—rather than replacing it.
This is an intentional architectural choice: maintain separation between the ML development workflow and the systems engineering artifact record, while keeping them formally connected through the traceability model.
Practical Starting Points
If you are beginning to write AI/ML requirements for a safety-critical program and have not done this before, start here:
Step 1: Define your scenario taxonomy before writing any ML requirements. Derive scenario classes from your system-level hazard analysis. Every scenario class that appears in the hazard analysis must appear in your ML requirements structure.
Step 2: Write accuracy floor requirements for every scenario class with a safety connection. Use the format described above—metric, threshold, dataset reference, measurement protocol reference. Do not allow “to be determined” placeholders past the system requirements review.
Step 3: Add a UQ requirement to every ML function in a decision-critical path. Specify both the calibration quality requirement and the operational threshold behavior.
Step 4: Write OOD detection requirements explicitly. Do not assume OOD handling is an implementation detail. It is a safety requirement. Specify the detection mechanism, the threshold, the triggered behavior, and the false anomaly rate constraint.
Step 5: Build the traceability chain from day one. The longer you wait to formalize the links between safety goals, ML requirements, scenario definitions, and verification evidence, the more expensive it becomes to reconstruct them later.
Step 6: Treat dataset changes as configuration changes. Any modification to a test or training dataset that affects a verification result must be controlled and traced the same way a requirements change is controlled and traced.
The Honest Assessment
There is no fully mature, universally adopted framework for AI/ML requirements in safety-critical systems today. The DO-178C supplement is not yet published. ISO/TR 4804 gives direction without complete prescriptive detail. The FAA Roadmap is a policy document, not a compliance handbook. Teams are doing real certification work in this space right now, and they are doing it by applying the best available guidance and building the case as methodically as they can.
What is clear is the direction. Requirements must be scenario-stratified. Performance must be bounded numerically. Uncertainty must be specified and constrained. OOD behavior must be a formal requirement, not an assumption. And the traceability connecting those requirements to verification evidence must be maintained as a live, auditable artifact—not reconstructed after the fact.
That is harder than writing traditional software requirements. It is also the only honest path to claiming that an AI/ML component in a safety-critical system has been verified.