AI is shifting the economic bottleneck in software engineering from production to acceptance.
For Consequential Software, local functional success is not enough: acceptance depends on lifecycle evidence and accountable technical authority.
Agentic tools can create a maturity illusion: visible functionality rises quickly while integration, verification, operational readiness, and evidence converge later.
The response is neither rejection of AI nor a return to indiscriminate manual control: it is to redesign the working conditions around agentic engineering.
Throughout software history, useful code has been expensive to produce. AI-assisted development changes that at a different scale: it can translate natural-language objectives into plans, navigate a repository, generate alternatives, run commands, inspect failures, and revise its own output. The result is a sharp reduction in the cost of reaching a solution that looks like a working solution.
Yet organizations do not ultimately need code. They need reliable behavior in a defined context — a system that can be operated, changed, audited, secured, supported, and, in some sectors, certified. The most useful question is not "How much code can an agent write?" It is "How much additional engineering work can an organization responsibly accept?"
Production capacity refers to the ability to generate designs, code, tests, documentation, analyses, and changes.
Acceptance capacity is the ability to determine whether a particular outcome is suitable for a stated use, supported by relevant evidence, and accepted by an accountable authority.
Although related, these capacities are not interchangeable. An organization can increase its production capacity and reduce delivery performance if review queues, integration burdens, evidence gaps, or unresolved decisions grow faster than the output.
The most expensive AI output is not always obviously wrong. An obvious failure is cheap to reject. Difficult output is coherent, plausible, and almost correct.
It may satisfy the prompt while violating an unstated rule. It may generate tests that confirm the assumptions embedded in its implementation. It may use a valid library in an unsupported manner. It may handle the normal case while missing the timing, concurrency, rollback, or degraded-mode requirements.
The cost of converting "almost right" into accepted engineering work is paid through senior review, architectural reconstruction, integration debugging, test expansion, compliance mapping, and operational caution. This is the hidden tax that can absorb the gains from faster generation.
Consequential Software is software whose failure, degradation, manipulation, unavailability, or untraceable change can materially affect human safety, health, financial integrity, essential services, legal obligations, or the operation of a complex system, and whose acceptance therefore requires lifecycle evidence and an accountable technical authority beyond local functional correctness.
Agentic tools can produce visible functionality quickly; therefore, projects appear to advance sharply. However, system maturity depends on integration, non-functional behavior, evidence, operational constraints, and convergence across disciplines. If these obligations are deferred, the gap is discovered later when architectural flexibility is lower and change is more expensive.
The paper illustrates the pattern with three Consequential Software scenarios — automotive, medical devices, and financial infrastructure — where locally successful software creates a false impression of system maturity.
A useful classification does not begin with binary labels. It asks how consequence is produced. Five dimensions are particularly relevant — and a sixth, lifecycle duration, is often useful: complex products can remain in operation for decades, and evidence must survive organizational and technological change.
Severity — What could happen if the software is incorrect, unavailable, manipulated, or misunderstood?: Determines the depth of evidence and authority required.
Coupling — How many physical, software, data, supplier, and organizational systems depend on behavior?: Local correctness may not predict the system behavior.
Reversibility — Can the change be safely rolled back, and can its effects be completely undone?: Irreversible or slowly reversible changes require stronger pre-action assurance.
Observability — Will failure be detected quickly and unambiguously?: Silent or delayed failures weaken the post-action correction.
Accountability — Must a named person or institution defend the decision to customers, regulators, operators, or the public?: It creates requirements for traceability, evidence, and retained decision rights.
Three illustrative composites — not descriptions of specific incidents — show how an implementation can be locally successful while remaining unready for system acceptance.
Automotive: An energy-management feature passes its component-level tests. At vehicle level it must still hold timing assumptions under bus congestion, handle stale sensor data consistently with the vehicle architecture, preserve the safety concept in diagnostic fallbacks, and stay compatible with supplier components delivered on different schedules.
Medical devices: An alarm-logic change passes tests generated from the written requirement. The organization must still determine whether the change remains consistent with the risk-management file, software safety classification, human-factors assumptions, hazard controls, and post-market obligations.
Financial infrastructure: A retry improvement in a payment service passes unit and integration tests. A retry that is safe before authorization may be unsafe after a partially completed settlement step — the operational dashboard can show successful retries while a downstream ledger records duplicate economic events.
The unresolved risk exists in relationships outside the visible feedback surface. A qualified human or independent system must reconstruct these relationships before acceptance. This is the difference between the generated code and the accepted systems.
The emerging requirement is an independent Assurance Plane that preserves five conditions: independent verification, validation against authoritative intent, named human authority, transferable evidence, and evidence freshness.
The companion Kealu white paper defines this proposed category as the Agentic Engineering Assurance Layer and specifies the technical obligations in detail.
Kealu is working on an early-access implementation of this concept within Kealu Vector, currently designated Vector Assurance Assist. This product is not the subject of this white paper. The purpose is to establish the problem, define Consequential Software, and invite a wider engineering and research community to test whether the proposed category is necessary and sufficient.