The Question That Started This

A test engineer at a mid-size aerospace supplier recently posed this to their program manager: “We’re at 90% test execution. Why is the safety review board saying we have coverage gaps?”

The manager’s assumption — shared by a lot of people outside test and systems engineering — was that if 90% of your test cases have run, you’re 90% covered. The test engineer knew this was wrong but struggled to explain exactly why in terms the manager could act on.

Here is the precise answer, and everything that follows from it.

What Requirements Coverage Actually Means

Requirements coverage is the proportion of system requirements that have at least one verification method assigned and at least one verification event planned or completed.

That definition has two distinct conditions. Both must be true for a requirement to count as covered:

Condition 1: A verification method is assigned. The requirement has been linked to a specific approach — test, analysis, inspection, or demonstration. Not a vague category. A specific method traceable to that requirement.

Condition 2: A verification event is planned or completed. There is a concrete activity — a test case, an analysis task, an inspection checklist item — that either has a scheduled execution date or has already been executed and has a recorded result.

A test case that exists but is not linked to any requirement does not contribute to requirements coverage. A requirement with a verification method assigned but no test case created is not covered. A test case that ran successfully but traces only to a derived requirement, not to the parent stakeholder requirement, leaves the parent requirement uncovered.

This is the gap the test engineer was trying to explain. The 90% test execution figure was measuring something real — execution progress against a test suite — but the test suite itself may not have mapped back to the full requirements set. The 10% of untested cases might have included requirements with zero traceability. Or the test suite may have had excellent internal execution rates while 15% of requirements had no test case assigned at all.

Count Coverage vs. Criticality Coverage

Once you accept the correct definition, the next critical distinction is between two types of coverage measurement:

Requirements Count Coverage

This is the simple ratio: number of requirements with at least one planned or completed verification event divided by total number of requirements. If you have 400 requirements and 360 have linked verification events, your count coverage is 90%.

Count coverage is a useful operational metric. It tells you whether your verification program is complete as a whole. Project managers, systems leads, and program offices use it to track progress toward milestone gates.

Requirements Criticality Coverage

This metric weights requirements by their safety, regulatory, or mission-critical designation. It asks: of the requirements that must not be missed — the ones tagged as safety-critical, safety-relevant, or regulatory-mandatory — what percentage have verification assigned and planned?

These are not the same number. A project can have 90% count coverage while having 70% criticality coverage if the uncovered requirements are disproportionately drawn from the critical tier.

Why this matters in practice: In DO-178C, ISO 26262, IEC 61508, and most defense acquisition frameworks, safety-critical requirements carry an absolute coverage obligation. There is no acceptable percentage below 100% for a Criticality A or ASIL D requirement. A program that reports 90% overall coverage but has three unverified safety-critical requirements does not have a 90% coverage posture — it has a compliance failure regardless of what the aggregate number says.

Any coverage report that does not separate these two metrics is hiding information that will eventually surface at the worst possible time: a certification review, an accident board, or a customer audit.

What Gaps in Coverage Look Like

Coverage gaps fall into four recognizable patterns:

Orphaned requirements. A requirement exists in the specification with no verification method assigned and no test case linked. It was written, baselined, and then never picked up during V&V planning. This happens most often with late-added requirements, requirements inherited from a parent program, or requirements added during a requirement change request after the initial test plan was written.

Stranded test cases. A test case exists and may have even run, but it traces to nothing — or traces to a derived software requirement that itself has no upward trace to a system-level requirement. The test ran. The requirement was never verified.

Coverage by proxy. A high-level requirement has a verification method assigned, but the only test case linked to it is a test of a child or derived requirement that doesn’t fully address the parent. The parent appears covered but is not exercised.

Planned-only coverage with no execution. A requirement has a verification event planned, which satisfies the first threshold of many coverage tools, but the event has never been executed and is chronically deferred. At milestone review, it shows as “coverage planned” rather than “coverage confirmed.” This is the most dangerous gap because it looks green in most reporting tools.

The Traceability Matrix as a Coverage Instrument

Most engineers encounter the Requirements Traceability Matrix (RTM) as a compliance artifact — something generated before a review to demonstrate that requirements link to tests. That framing understates what the RTM actually is.

A properly maintained RTM is a coverage instrument. Every blank cell in the requirement-to-verification column is a coverage gap. Every test case row with no requirement in the upstream column is a stranded test. Every requirement row with a verification method assigned but no test case ID populated is a planned-only gap.

Reading an RTM for coverage means scanning for structure, not just counting populated cells. The questions are:

  • Are there requirements with no verification method at all?
  • Are there requirements where the assigned method is “TBD”?
  • Are there test cases that link downward to subsystem specs but have no upward trace to system requirements?
  • Are there safety-critical requirements marked with any status other than “complete with result”?

An RTM generated once before a CDR or PDR and then frozen tells you almost nothing about current coverage posture. Requirements change. Test cases get deferred. Verification methods get revised during design evolution. A static RTM reflects one moment, and that moment is usually several months behind the current state of the program.

This is the fundamental limitation of document-centric coverage reporting: it requires someone to regenerate the report, reconcile it against the current requirements baseline, and distribute it manually. By the time it reaches anyone who can act on it, it may already be stale.

How Modern Tools Implement Coverage Measurement

Traditional requirements tools — IBM DOORS, DOORS Next, Jama Connect, Polarion, Codebeamer — all support traceability matrices and can be configured to generate coverage reports. These tools are mature, widely deployed, and well-understood by certification bodies. Their coverage reporting works when it’s used.

The operational problem is that coverage in these tools is typically a report you run, not a state you observe. Someone has to know to query it, configure the view correctly, interpret the output, and distribute the findings to the people who can act on them. In large programs with thousands of requirements across multiple tools and multiple contractors, that process creates latency. Gaps that exist today may not surface in a coverage report until the next scheduled review.

Flow Engineering takes a structurally different approach. Built as an AI-native, graph-based requirements management platform for hardware and systems engineering teams, it treats coverage as a continuously computed property of the requirements graph rather than a report generated on demand.

In Flow Engineering, every requirement node carries a live coverage state derived from its linked verification artifacts. The system tracks not just whether a verification method is assigned, but whether a concrete verification event exists and what its execution status is. Coverage is not a number you calculate before a review — it’s a visible property of the graph at any point in time.

For safety-critical requirements, Flow Engineering maintains a separate coverage lane that cannot be rolled into aggregate statistics. A requirement flagged as safety-critical shows its individual coverage state independently of the overall project percentage. If it’s uncovered, it’s visible as uncovered — not averaged away into a 94% headline figure.

The practical implication for teams using Flow Engineering is that coverage gaps surface during normal engineering work, not during milestone preparation. When an engineer adds a new requirement without linking a verification method, the gap is immediately visible in the requirement’s status. When a test case is deferred, the affected requirements immediately reflect a degraded coverage state. This changes when teams respond to gaps — earlier, when the cost of fixing them is lower.

Flow Engineering’s focus is hardware and systems-level requirements. It is not designed to replace software-level test management tools like JIRA or Xray, and teams working across the full stack will still need integration strategies to connect software test results back to system-level requirements. That integration is achievable through its API layer, but it requires deliberate setup. For organizations whose primary coverage challenges are at the system and subsystem level — which describes most hardware programs — this tradeoff is well-suited to what they actually need.

Practical Starting Points

If you’re trying to establish or improve coverage measurement on an active program, three things will have the highest immediate impact:

1. Separate your criticality tiers before measuring anything else. If you don’t have a clear tagging scheme for safety-critical, safety-relevant, and standard requirements, your coverage numbers will be misleading regardless of what tool you use. This is a requirements management hygiene problem, not a tool problem.

2. Audit your RTM for stranded test cases and orphaned requirements. Run both directions: requirements with no test case, and test cases with no requirement. The second category is frequently larger than teams expect.

3. Establish a coverage review cadence that isn’t tied to milestones. Coverage reviewed only at CDR and PDR is coverage managed reactively. Weekly or sprint-level coverage visibility — even informally — catches gaps while they’re inexpensive to close.

The test engineer who started this conversation was right. Test execution percentage and requirements coverage percentage are measuring different things, and the gap between them is exactly where verification programs fail. Understanding the difference precisely is not a semantic exercise — it’s the foundation of any V&V program that will survive a certification audit.