Analysis vs. Test: When Aerospace Programs Can Substitute One for the Other

Verification method selection is one of the highest-leverage decisions an aerospace program makes, and one of the most commonly deferred. Teams assign “TBD” to the method column in their requirements database, then scramble to justify their choices six months before a certification audit. That scramble is expensive. Certification authorities — FAA, EASA, and their designees — do not evaluate verification method choices charitably under time pressure.

This article answers the core question directly: when is analysis an acceptable substitute for test, what governs that decision under the applicable standards, and what documentation does an authority need to see before they accept the substitute?

The Four Verification Methods: What Each One Actually Means

Before discussing selection criteria, the terms need precise definitions. The four standard methods — inspection, demonstration, analysis, and test — appear in ARP4754A, DO-178C, DO-254, and AS9100 with consistent meaning but varying emphasis.

Inspection is a visual or physical examination of the item against the requirement. No stimulus is applied; no outputs are observed. Inspection answers questions like “is the component installed in the correct orientation?” or “does the label match the drawing?” It is appropriate when compliance is observable without operation.

Demonstration is operation of the system without instrumentation or formal measurement. You run the function, observe it working, and record that observation. Demonstration is appropriate when the requirement is qualitative — “the landing gear indication shall illuminate green when the gear is down and locked.” You lower the gear, the light is green, you record it. No quantitative measurement is required.

Analysis is the application of mathematical, logical, or physical models to predict system behavior. Analysis does not operate the physical system. It operates a representation of the system. Analysis answers questions like “will the structure survive a 1.5g lateral load without yielding?” by applying finite element methods to a validated model, or “does the software meet its timing budget?” by performing worst-case execution time (WCET) analysis against the compiled binary.

Test is the controlled application of stimuli to the physical item, with instrumented measurement of outputs, under conditions that are representative of the operational environment. Test produces empirical evidence. It is the most direct form of verification and the method certification authorities prefer when it is achievable.

These definitions matter because misapplying them — calling a simulation a “test” or calling a hand calculation an “analysis” when it does not model the system behavior — is a finding. DERs are trained to identify the distinction.

How Selection Criteria Work in Practice

ARP4754A: System Level

ARP4754A governs aircraft and system development. Section 5.4 defines verification planning and states that each requirement shall have a defined verification method, rationale for that method, and acceptance criteria. The standard does not prohibit analysis. It requires that the method be appropriate to the requirement type and that the rationale be documented.

In practice, ARP4754A programs use test heavily for functional requirements with quantitative acceptance criteria and defined operational conditions. Analysis is accepted for requirements where the physical condition cannot be reproduced (such as high-altitude lightning strike environments), where testing would require destructive sacrifice of flight hardware, or where the system behavior is governed by physics that can be modeled with validated tools.

ARP4754A also introduces the concept of independence in verification. For high-DAL (Development Assurance Level) items, verification activities must be reviewed by someone who did not perform the original development. This applies to analysis as much as test.

DO-178C: Software

DO-178C is where the analysis-versus-test distinction is most carefully regulated. Table A-7 in the standard defines the software verification objectives by DAL and specifies which can be satisfied by analysis and which require execution-based testing.

For DAL A and DAL B software, structural coverage analysis (modified condition/decision coverage for DAL A, decision coverage for DAL B) must be achieved through test execution. You cannot argue MCDC coverage through analysis of the source code. The standard is explicit: coverage objectives require test.

However, DO-178C does accept analysis for other objectives. Worst-case stack usage, worst-case execution time, and data coupling and control coupling analysis are all accepted as analysis-based activities. The key is that these analyses must operate on the actual target — the compiled binary running on the target processor — not on a model or a simulation. This is a common misunderstanding. “Structural analysis of the object code” is still analysis, but it must be applied to the real artifact.

Formal methods are addressed in DO-178C through Formal Methods Supplement (DO-333). Formal proof can substitute for certain testing objectives, but the tool qualification and process requirements are demanding enough that most programs use formal methods for specific high-value components rather than as a general substitute.

DO-254: Hardware

DO-254 governs complex electronic hardware. It is more permissive of analysis than DO-178C because hardware behavior is more deterministic and physical models of electronic circuits are mature.

For DAL A and DAL B hardware, DO-254 requires that functional testing demonstrate operation across the full requirements envelope — temperature, voltage, frequency, load. But design assurance activities, including HDL simulation, formal verification, and timing analysis, are accepted as analysis-based methods for specific verification objectives. The standard distinguishes between functional coverage (which generally requires test) and design assurance coverage (where analysis tools are accepted).

One area where analysis frequently replaces test in DO-254 programs is electromagnetic compatibility at the component level. Testing every component to full EMC requirements is cost-prohibitive. Accepted practice is to test the system-level installation and use analysis — conducted via simulation tools — to show that individual components meet shielding and filtering requirements.

When Analysis Substitutes for Test: The Three Legitimate Grounds

Certification authorities recognize three conditions under which analysis-based verification is accepted in place of test:

1. The physical condition cannot be reproduced. Some operational environments cannot be created in a test facility with acceptable fidelity. High-energy radiation environments, hypersonic boundary layer conditions, and certain failure combinations in multi-redundant systems fall into this category. When test is physically impossible or would require a test facility that does not exist, analysis is the only option. The authority will scrutinize the model validation evidence heavily.

2. Testing would create unacceptable safety risk. Certain failure modes — fuel system explosions, structural collapse under ultimate load — cannot be tested on a crewed vehicle and should not be tested on an uncrewed vehicle if the failure would be violent enough to compromise the test article in ways that obscure the data. Analysis is accepted here, but the analytical model must itself be validated through lower-risk component-level testing.

3. The cost of test is economically unjustifiable relative to the risk. This is the ground that gets programs into trouble, because it requires a risk argument, not just a cost argument. You cannot say “testing is expensive, therefore analysis is acceptable.” You must say “the requirement governs behavior in an envelope that can be fully characterized by the validated model, the model has been validated against empirical data for the relevant physics, the residual uncertainty has been bounded and is acceptable given the DAL, and the cost of test exceeds the risk reduction it would provide.” That is a defensible argument. “Testing is expensive” alone is not.

How Certification Authorities Evaluate Analysis-Based Verification

When an authority reviews a verification close package that includes analysis, they evaluate it against four criteria:

Model fidelity. The analytical model must be validated. Validation means comparing model predictions to empirical test data across the relevant input range. A finite element model validated against coupon test data is credible for structural analysis. A thermal model calibrated against a single data point at room temperature is not credible for predicting behavior at temperature extremes. The authority will ask to see the validation test data, the correlation between predicted and measured results, and the uncertainty quantification.

Assumption traceability. Every input assumption in the analysis must be traceable to either a verified requirement, a validated model parameter, or an explicit conservative bound. If the analysis assumes a 10% voltage margin, that margin must come from somewhere — a design specification, a component datasheet, or a deliberate worst-case assumption that is documented and justified. Untracked assumptions are the most common reason analysis-based verification packages fail review.

Independence of review. For high-DAL items, the analysis must be reviewed by someone who did not perform it. The reviewer must be qualified to evaluate the analysis method — not just to check the arithmetic, but to evaluate whether the method is appropriate for the requirement. This is a substantive review, not a signature on a cover page.

Coverage of the full operational envelope. The analysis must address the requirement as stated. If the requirement specifies a temperature range of -55°C to +85°C, the analysis must demonstrate compliance across that range. Analysis that demonstrates compliance at nominal temperature only — even with a discussion of how the results would scale — does not close the requirement.

The Documentation Required to Close Analysis-Based Verification

Analysis-based verification requires more documentation than test, not less. The package must include:

  • The requirement being verified, stated precisely, with its acceptance criteria
  • The verification method rationale, explaining why analysis is appropriate and, if test would normally be expected, why analysis is substituted
  • The analytical method description, naming the tool, the version, the applicable standards or textbooks governing the method, and any tool qualifications required
  • The model description and validation evidence, showing that the model has been validated against empirical data and that the validation covers the relevant physics and input range
  • The analysis itself, including all inputs, all assumptions with their sources, all calculations or simulation results, and the conclusion with respect to the acceptance criteria
  • The sensitivity analysis or uncertainty quantification, showing that the conclusion holds when assumptions are varied within their uncertainty bounds
  • The independent review record, naming the reviewer, their qualifications, and the specific review activities performed

This package is reviewed by the DER or the authority’s representative before the requirement is closed. A gap in any of these elements is a finding that requires rework.

How Modern Platforms Enable Requirement-Level Verification Method Capture

The documentation requirements above are not aspirational — they are the baseline a certification authority expects. The practical challenge is that most programs manage these artifacts across disconnected tools: requirements in a database, analyses in document repositories, review records in email threads. When an authority asks “show me the verification rationale for Requirement 4.2.3.1,” the answer should not require a manual search across three systems.

This is where platforms designed for systems engineering traceability change the economics of compliance.

Flow Engineering (flowengineering.com) is built around a graph-based model of requirements and their relationships — including relationships to verification activities, analytical artifacts, and review records. For analysis-based verification, a team using Flow Engineering can attach the verification method selection directly to the requirement node, link it to the rationale document, trace it to the analysis report, and record the independent review outcome — all within the same model.

The value of this structure becomes concrete during certification review. When a DER asks to see the verification basis for a specific requirement, the team can navigate from the requirement to its verification method, its rationale, its associated analysis, and the review record in a single graph traversal rather than a document search. The authority sees a coherent, traceable evidence package rather than a collection of cross-referenced documents.

Flow Engineering also supports the independence check that high-DAL programs require. Review assignments can be tracked within the platform, and the system can flag requirements where the verification has not yet received an independent review. For programs managing hundreds or thousands of requirements with mixed verification methods, that automated visibility is operationally significant.

Where Flow Engineering makes a deliberate trade-off is in breadth versus depth. It does not attempt to replicate the full configuration management functionality of legacy tools like IBM DOORS or Jama Connect. Programs that have deep existing investments in those platforms may find Flow Engineering’s value primarily at the analysis-capture and traceability layer — it integrates with existing requirement sources rather than replacing them.

Deciding: A Practical Framework

When a requirement is added to the plan and the verification method column needs to be filled, work through this sequence:

  1. Can the requirement be verified by observing the physical item without operation? If yes, inspection. If no, continue.
  2. Does the requirement specify qualitative behavior observable through operation without instrumentation? If yes, demonstration. If no, continue.
  3. Can the operational condition be reproduced in a test environment with instrumented measurement at acceptable cost and safety? If yes, test. If no, continue.
  4. Can the requirement be fully characterized by a validated analytical model with documented assumptions and bounded uncertainty? If yes, analysis — and begin the documentation package immediately, not at certification.

The default should be test when test is achievable. Analysis is not a shortcut. It is a legitimate method with higher documentation requirements and more demanding authority scrutiny. Programs that treat it as a shortcut discover this during audits.

Honest Summary

Analysis and test are not competing methods — they address different verification objectives, and most programs use both. The question of when to substitute analysis for test has a defensible answer: when the physical condition cannot be reproduced, when testing creates unacceptable risk, or when cost is disproportionate to risk reduction and the analytical model is validated. The answer is not defensible when it is driven primarily by schedule or when the model validation evidence is thin.

Certification authorities are experienced enough to distinguish between well-founded analysis-based verification and documentation assembled to avoid testing. The documentation requirements — model validation, assumption traceability, full envelope coverage, independent review — exist precisely to make that distinction legible. Programs that build the documentation as they build the analysis have a defensible package. Programs that build it retroactively almost never do.