AI is shifting the economic bottleneck in software engineering from production to acceptance.
For Consequential Software, local functional success is not enough: acceptance depends on lifecycle evidence and accountable technical authority.
Agentic tools can create a maturity illusion: visible functionality rises quickly while integration, verification, operational readiness, and evidence converge later.
The response is neither rejection of AI nor a return to indiscriminate manual control: it is to redesign the working conditions around agentic engineering.
For most of software history, useful code was expensive to produce. AI-assisted development changes that at a different scale: it can translate natural-language objectives into plans, navigate a repository, generate alternatives, run commands, inspect failures, and revise its own output. The result is a sharp reduction in the cost of reaching something that looks like a working solution.
Yet organizations do not ultimately need code. They need reliable behavior in a defined context — a system that can be operated, changed, audited, secured, supported, and, in some sectors, certified. The most useful question is not "How much code can an agent write?" It is "How much additional engineering work can an organization responsibly accept?"
Production capacity is the ability to generate designs, code, tests, documentation, analyses, and changes.
Acceptance capacity is the ability to determine that a particular outcome is suitable for a stated use, supported by relevant evidence, and accepted by an accountable authority.
The capacities are related but not interchangeable. An organization can increase production capacity and reduce delivery performance if review queues, integration burden, evidence gaps, or unresolved decisions grow faster than output.
The most expensive AI output is not always obviously wrong. An obvious failure is cheap to reject. The difficult output is coherent, plausible, and almost right.
It may satisfy the prompt while violating an unstated architectural rule. It may generate tests that confirm the same assumptions embedded in its implementation. It may use a valid library in an unsupported way. It may handle the normal case while missing a timing, concurrency, rollback, or degraded-mode requirement.
The cost of converting "almost right" into accepted engineering work is paid through senior review, architectural reconstruction, integration debugging, test expansion, compliance mapping, and operational caution. This is the hidden tax that can absorb the gains from faster generation.
Consequential Software is software whose failure, degradation, manipulation, unavailability, or untraceable change can materially affect human safety, health, financial integrity, essential services, legal obligations, or the operation of a complex system, and whose acceptance therefore requires lifecycle evidence and accountable technical authority beyond local functional correctness.
Agentic tools can produce visible functionality quickly, so the project appears to advance sharply. Yet system maturity depends on integration, non-functional behavior, evidence, operational constraints, and convergence across disciplines. If those obligations are deferred, the gap is discovered later, when architectural flexibility is lower and change is more expensive.
The paper illustrates the pattern with three Consequential Software scenarios — automotive, medical devices, and financial infrastructure — where locally successful software creates a false impression of system maturity.
A useful classification does not begin with a binary label. It asks how consequence is produced. Five dimensions are particularly relevant — and a sixth, lifecycle duration, is often useful: complex products can remain in operation for decades, and evidence must survive organizational and technological change.
Severity — What could happen if the software is wrong, unavailable, manipulated, or misunderstood?: Determines the depth of evidence and authority required.
Coupling — How many physical, software, data, supplier, and organizational systems depend on the behavior?: Local correctness may not predict system behavior.
Reversibility — Can the change be safely rolled back, and can its effects be fully undone?: Irreversible or slowly reversible changes require stronger pre-action assurance.
Observability — Will failure be detected quickly and unambiguously?: Silent or delayed failures weaken post-action correction.
Accountability — Must a named person or institution defend the decision to customers, regulators, operators, or the public?: Creates requirements for traceability, evidence, and retained decision rights.
Three illustrative composites — not descriptions of specific incidents — show how an implementation can be locally successful while remaining unready for system acceptance.
Automotive: An energy-management feature passes its component-level tests. At vehicle level it must still hold timing assumptions under bus congestion, handle stale sensor data consistently with the vehicle architecture, preserve the safety concept in diagnostic fallbacks, and stay compatible with supplier components delivered on different schedules.
Medical devices: An alarm-logic change passes tests generated from the written requirement. The organization must still determine whether the change remains consistent with the risk-management file, software safety classification, human-factors assumptions, hazard controls, and post-market obligations.
Financial infrastructure: A retry improvement in a payment service passes unit and integration tests. A retry that is safe before authorization may be unsafe after a partially completed settlement step — the operational dashboard can show successful retries while a downstream ledger records duplicate economic events.
The unresolved risk exists in relationships outside the visible feedback surface. A qualified human or independent system must reconstruct those relationships before acceptance. This is the difference between generated code and accepted systems.
The emerging requirement is an independent Assurance Plane that preserves five conditions: independent verification, validation against authoritative intent, named human authority, transferable evidence, and evidence freshness.
The companion Kealu paper defines this proposed category as the Agentic Engineering Assurance Layer and specifies the technical obligations in detail.
Kealu is working on an early-access implementation of this concept within Kealu Vector, currently designated Vector Assurance Assist. The product is not the subject of this paper. The purpose is to establish the problem, define Consequential Software, and invite a wider engineering and research community to test whether the proposed category is necessary and sufficient.