How to Structure Requirements for ML-Enabled Safety-Critical Systems
Machine learning components are being integrated into safety-critical systems—flight management, autonomous driving, medical imaging, industrial control—faster than the standards bodies can write guidance. The result is that engineering teams are making consequential architectural decisions about ML safety requirements without a settled playbook to follow.
This is not a reason to wait. Several serious frameworks have emerged from regulators and standards bodies that, taken together, provide enough structure to proceed rigorously. This article explains what those frameworks say, where they conflict or leave gaps, and how to translate them into a practical requirements structure that a systems engineering team can actually implement.
The Honest State of Standards
Three frameworks are doing the most work in this space right now.
EASA’s AI Trustworthiness Framework (the Concept Paper on EASA’s Artificial Intelligence Roadmap, with ongoing updates) addresses AI/ML components in certified aviation products. It introduces the concept of Learning Assurance and maps it to existing certification frameworks like DO-178C and ARP4754A. EASA’s approach is the most mature for aviation and has been influential on thinking in adjacent safety domains.
The FAA’s ML Risk Mitigation document (FAA-HSIA-ML, updated in 2025) is less prescriptive than EASA’s materials but provides a useful risk categorization framework. It distinguishes between ML components that are decision-making versus advisory, and between those that operate in bounded versus open-world conditions—distinctions that directly shape how you write requirements.
ISO/IEC 42001:2023 is an AI management system standard, not a product-level safety standard. It addresses organizational governance of AI development rather than specifying what requirements an ML component must meet. It is useful for establishing the process infrastructure around ML development but should not be mistaken for a substitute for domain-specific safety standards.
None of these three frameworks fully aligns with the others. A team building a certified aviation system will work primarily from EASA’s framework while using FAA materials as a check. A team building an automotive system will work from ISO 26262 and SOTIF (ISO 21448), which have their own evolving ML guidance. The practical starting point is: identify your domain’s lead regulatory framework, then treat the other documents as supplementary.
The Four Hard Problems in ML Requirements
1. Writing Performance Requirements for Probabilistic Systems
Traditional safety requirements are written as deterministic assertions: The braking system shall bring the vehicle to a complete stop within 4.2 meters when traveling at 30 km/h on dry asphalt. You can verify this with a test. Pass or fail.
ML components do not behave deterministically. A perception model that classifies road obstacles will be correct 99.3% of the time across a test distribution—but “99.3% accuracy” is not a safety requirement. It is a metric. The requirement must specify:
- The performance metric being bounded (precision, recall, F1, AUC—chosen because of what failure modes matter)
- The statistical confidence interval around that bound
- The input distribution over which that bound holds
- The consequence model that justifies why that bound is sufficient
A well-formed ML performance requirement might read: The pedestrian detection model shall achieve a false negative rate of no greater than 0.2% at 95% confidence when evaluated on inputs drawn from the operational design domain defined in [ODD-001], as measured by the test protocol specified in [EVAL-007].
That requirement is traceable, testable, and honest about its scope. It is also significantly harder to write than a deterministic assertion, because it forces the team to define the ODD and the evaluation protocol as explicit artifacts—which is exactly the discipline that makes ML systems more defensible.
2. Specifying Training Data Quality
Training data is not an implementation artifact that requirements engineers hand off to the ML team. It is a source of system properties—capabilities, biases, and failure modes—that must be specified and controlled at the requirements level.
A training data quality specification should cover at minimum:
Coverage requirements: Which conditions, object classes, lighting environments, sensor configurations, and edge cases must be represented in the training set, and at what frequency? This connects directly to ODD definition.
Labeling quality requirements: What inter-annotator agreement threshold is required? What is the allowable label error rate? How are ambiguous cases handled?
Provenance and lineage requirements: Where did the data come from? What are the licensing and privacy constraints? Can it be traced to a specific collection protocol?
Balance and representation requirements: Are there subpopulations that must be represented above a minimum threshold to prevent systematic underperformance?
These are requirements. They should be written as such, allocated to the data engineering function, and verified with evidence—data sheets, collection logs, annotation audit results—that is traceable back to the requirement.
3. Establishing Operational Monitoring Requirements
A ML component that passes all pre-deployment tests can still fail in deployment because the world changes. This is not hypothetical. Weather patterns shift, sensor hardware ages, user populations evolve, adversarial conditions emerge. Systems that pass certification testing at T=0 may degrade silently over time.
Operational monitoring requirements define what must be observed, how often, and what triggers a review or operational limitation. These requirements sit at the intersection of systems engineering and MLOps, and most traditional requirements management processes have no place for them.
Monitoring requirements should specify:
- Which performance indicators are tracked in operation (not just metrics from test—metrics that can be computed from operational data, which often means proxy metrics because ground truth labels are not available in real-time)
- Threshold values that trigger alerts, operational restrictions, or mandatory retraining
- Logging requirements: what data must be captured, at what fidelity, for how long
- Review cadence: how often is operational performance formally assessed against requirements
These are not afterthoughts. They should appear in the requirements baseline before the system enters service.
4. Handling Distributional Shift
Distributional shift—the divergence between the distribution the model was trained and tested on and the distribution it encounters in deployment—is the central operational risk for ML safety components. It must be treated as a requirements problem, not just an engineering problem.
The Operational Design Domain (ODD) concept, borrowed from autonomous vehicle development, is the practical tool here. The ODD is a formal specification of the conditions under which the ML component’s performance requirements are valid. Anything outside the ODD is explicitly out of scope for the component, which means the system architecture must include a mechanism to detect ODD exceedance and respond to it.
ODD monitoring itself becomes a requirement: The system shall detect when sensor inputs are inconsistent with the operational design domain defined in [ODD-001] and shall transition to a degraded-mode behavior specified in [SAFE-012] within 200ms of detection.
Learning Assurance and the Extended V-Model
Learning Assurance, as articulated in EASA’s framework and in the emerging EUROCAE ED-324 / RTCA DO-328 guidance, extends the V-model to account for the ML development lifecycle. The key insight is that ML development has verification activities that have no analog in traditional software development: data collection, data preprocessing, feature engineering, model training, and model evaluation are all development activities that produce artifacts that must be traced and verified.
The extended V-model adds a parallel development track for data and model artifacts that runs alongside the traditional software development track. On the left side of the V, you define not only system requirements and software requirements, but also:
- AI/ML Constituent Requirements: the performance, robustness, and explainability requirements specific to the ML component
- Learning Data Requirements: the data coverage, quality, and provenance specifications described above
- Model Development Constraints: architectural constraints (e.g., interpretability requirements that rule out certain model families), training procedure constraints, and hyperparameter governance
On the right side of the V, verification activities include:
- Data Verification: evidence that the training and test datasets satisfy the Learning Data Requirements
- Model Verification: statistical evaluation against the performance requirements across the defined ODD
- Integration Verification: testing the ML component in the full system context, including ODD monitoring and degraded-mode transitions
- Operational Monitoring: the ongoing verification activity that continues post-deployment
Each of these activities produces artifacts—data sheets, model cards, evaluation reports, monitoring logs—that must be traceable to the requirements they verify. This is where traditional requirements management tooling breaks down. Most requirements tools were built to manage text-based requirements and link them to test cases. They were not designed to handle the artifact types and traceability relationships that Learning Assurance demands.
How Modern Tools Support ML Requirements Traceability
Structuring ML safety requirements is difficult enough conceptually. Making them manageable in practice—traceable, reviewable, and maintainable over the system lifecycle—requires tooling that treats the requirements and their evidence as a connected graph, not a document hierarchy.
This is where graph-based requirements management tools have a genuine advantage over document-based tools like IBM DOORS or Jama Connect (which are excellent at managing text requirements but were not designed for the artifact-rich, relationship-dense assurance cases that ML systems generate).
Flow Engineering is one example of a tool built on this model. Its graph-based structure allows teams to connect an AI/ML Constituent Requirement directly to the ODD specification it depends on, the training data requirement it delegates to, the evaluation protocol that verifies it, the test results that constitute evidence, and the operational monitoring threshold that governs it in service. That entire chain is navigable, not reconstructed from cross-references in documents.
For Learning Assurance specifically, this matters because the assurance case for an ML component is not a linear chain—it is a graph. A performance requirement is verified by multiple evaluation reports run across multiple data subsets. A training data requirement has evidence from multiple annotation audits. An ODD monitoring requirement links both to the system safety requirements that motivate it and to the operational monitoring infrastructure that implements it. Managing these relationships in a flat document structure or a traditional hierarchical requirements database produces assurance cases that are technically present but practically unusable—too cumbersome to update when anything changes, too opaque to audit efficiently.
Flow Engineering also supports the definition of custom requirement attributes, which matters for ML requirements. Standard attributes like rationale, verification method, and status need to be extended with ML-specific attributes: confidence level, applicable ODD, training data version, model version. These attributes are not cosmetic—they are part of the traceability record that tells a regulator or safety reviewer what the requirement means and what evidence supports it.
This is not a claim that tooling solves the hard problems. The hard problems—defining a valid ODD, choosing appropriate performance metrics, designing a monitoring strategy that detects distributional shift without generating constant false alarms—require engineering judgment that no tool provides. What tooling does is make it possible to manage the resulting artifact structure at the scale and complexity that ML safety systems generate.
Practical Starting Points
If your team is beginning to structure requirements for an ML-enabled safety-critical system, here is a sequence that works:
Start with the system safety analysis. Before writing any ML-specific requirements, the hazard analysis and risk assessment must identify which failure modes of the ML component contribute to system-level hazards, and at what severity. This analysis drives the rigor required in the ML requirements.
Define the ODD first, in writing. The ODD is the foundation for all ML performance requirements. Without it, performance requirements are unverifiable. The ODD should be a formal, controlled artifact with its own identifier, traceable to the hazard analysis.
Write ML performance requirements as statistical bounds over the ODD. Specify the metric, the bound, the confidence level, and the evaluation protocol. Reference the ODD explicitly.
Write training data requirements as first-class requirements. Assign them identifiers. Allocate them to the data engineering function. Define verification methods and evidence types.
Add operational monitoring requirements to the initial requirements baseline. Do not treat monitoring as a post-deployment addition. The monitoring strategy is part of the safety argument and must be defined before the system enters service.
Choose tooling that can represent the graph, not just the list. If your requirements tool cannot represent the relationship between a performance requirement, the ODD it depends on, and the test results that verify it without reducing everything to a flat cross-reference table, you will spend more time managing the traceability structure than engineering the system.
Honest Assessment
ML safety requirements are genuinely harder to write than traditional software requirements. The standards are incomplete. The tooling is catching up. Regulators are learning alongside the industry they regulate.
What is not acceptable is treating this difficulty as a reason to write vague requirements, defer data quality specifications to the ML team, or skip operational monitoring requirements because the standard does not mandate them yet.
The teams doing this well are treating ML-specific artifacts—ODD definitions, training data specifications, evaluation protocols, monitoring thresholds—as first-class engineering artifacts with the same rigor and traceability discipline they apply to hardware and software requirements. The frameworks to support that approach exist, even if they are incomplete. The tools to manage the resulting artifact structure are available. The discipline to apply them is a choice.