Change Failure Rate and Lead Time When Agents Write the Code

We recomputed DORA change failure rate and lead time on 4,545 pull requests in one public repository. Each figure moves with a choice the metric leaves open.

Change Failure Rate and Lead Time When Agents Write the Code

In short

Change lead time and change fail rate are DORA metrics defined from commit to production and per deployment. We recomputed both on MLflow, a public repository, for 4,545 pull requests merged in fifteen months, 1,225 of them labeled agent-authored by a research dataset. Agent-labeled changes merged a median 1.6 hours after opening, against 23.5 hours, and both groups took about twenty days from first commit to a release. Of at least 25 reverted changes, six had shipped, in 3 of 32 releases. Each figure moves with a choice the metric leaves open: where the clock starts, what counts as a deployment and a failure, and which changes are the agent's.

What DORA measures

DORA's guide defines change lead time as "The amount of time it takes for a change to go from committed to version control to deployed in production" and change fail rate as "The ratio of deployments that require immediate intervention following a deployment. Likely resulting in a rollback of the changes or a "hotfix" to quickly remediate any issues" (DORA, software delivery performance metrics, last updated 5 January 2026). It adds that the metrics "are best suited for measuring one application or service at a time" and warns that "Comparing metrics between vastly different applications (for example, a mobile app and a mainframe system) can be misleading."

DORA collects both through its annual survey. The 2025 questionnaire asks "how long does it take to go from code committed to code successfully running in production" and offers ranges from "Less than one hour" to "More than six months". For failure it asks what share of "changes to production or releases to users result in degraded service (for example, leads to service impairment or service outage) and subsequently requires remediation (for example, requires a hotfix, rollback, fix forward or patch)" (DORA, 2025 survey questions). The answers are self-reported, so a team that wants a measured number decides for itself where a change starts and which deployments failed.

The definitions have moved before. DORA's history records that in 2014 change fail rate "did not significantly correlate with the other variables", and that in 2024 it "acted as a proxy for the amount of rework a team must perform", which led to a fifth metric, deployment rework rate (DORA, A history of DORA's software delivery metrics). Our article on developer productivity metrics explains why end-of-chain measures hold up better than output counts.

One repository, recomputed

MLflow is an open-source machine learning library that ships versioned releases from release branches. We took every pull request merged into its main branch from 1 January 2025 to 31 March 2026 and labeled as agent-authored the ones the AIDev dataset attributes to a coding agent (Li, Zhang and Hassan, arXiv 2507.15003; AIDev, release with a 31 March 2026 cutoff). Its labels for MLflow begin in May 2025, so the first months of the window hold none. First commit, opening and merge came from the GitHub API, and the first version tag holding each change from the repository's history. A library has no production deployment, so a release stands in for one, as DORA's own "releases to users" allows. Google's reference implementation, Four Keys, warns that projects "with releases and no deployments, for example, libraries, do not work well" with it, and its repository says it "is not currently maintained" (dora-team/fourkeys). Its definition of the failure metric shows the unit: "The number of failures per the number of deployments. For example, if there are four deployments in a day and one causes a failure, that is a 25% change failure rate."

The same 4,545 merged changes in MLflow, January 2025 to March 2026. Left, median times from pull request opened to merge, from first commit to first release and from merge to first release, for agent-labeled changes and the rest. Right, how many changes or releases count as failed depending on what is counted.
The same 4,545 merged changes in MLflow, January 2025 to March 2026. Left, median times from pull request opened to merge, from first commit to first release and from merge to first release, for agent-labeled changes and the rest. Right, how many changes or releases count as failed depending on what is counted.
MeasureAgent-labeled (1,225)Other (3,320)All (4,545)
Pull request opened to merge, median1.6 hours23.5 hours14.0 hours
Merge to first release, median19.9 days15.4 days17.1 days
First commit to first release, median20.3 days19.9 days20.0 days
Merged changes later reverted on the main branch, at least52025

Lead time follows the release train

Half the agent-labeled changes merged within 1.6 hours of opening. From commit to release, as DORA defines lead time, both groups took about twenty days, because a merged change waits for the next release whoever wrote it. All but 3 of the changes reached users through 32 releases, 17 of them patch releases.

The start of the clock is a choice too, and the time an agent's task waits before its first commit appears in neither git nor the pull request. Four Keys starts at each commit and advises against squash merging, which "breaks the link between the commit into trunk and the commits from the branch you developed on". MLflow squash-merges.

Change failure depends on what is counted

Reverts, the closest signal a public repository offers, give a different count at each step. Revert commits that cite a pull request number point to 7 changes. Quoted titles add 9, and reading each revert and its description adds 9 more: at least 25, four of them partial, or 0.55 percent of merged changes. Other commits undo earlier ones without the word revert, so the count is a floor. Nineteen were reverted before any release carried them. Six shipped, two from release branches that did not yet hold the revert and four reverted after their release, in 3 of the 32 releases. Per release, as DORA counts, the first two methods find 1 failed release in 32 and the full reading finds 3.

Patch releases give another scale, closer to the rework question in DORA's survey: "Approximately what percentage of deployments in the last 6 months were not planned but were performed to address a user-facing bug in the application?" Of the 14 minor and major releases tagged in the window, 4 were followed by a patch of the same line within seven calendar days and 8 within fourteen. Patch releases also bundle planned fixes, so those counts are an upper bound.

A revert is not always a failure in DORA's sense. One says the issue its original fixed "no longer exists on latest proto update". Another removes a database setting because "WAL mode uses shared memory files (.shm) that break when SQLite databases are accessed across Docker container boundaries via volume mounts", a defect caught by the test suite before release. Most revert descriptions give no reason, fixes made forward leave no revert at all, and Four Keys ties a failure to a deployment only when someone types "root cause:" and the change's identifier into an incident.

Which changes are the agent's

The agent label follows the dataset's mining rules, built on "recognizable signals such as bot author names", "branch prefixes" and "explicit messages in the PR body". In MLflow, 455 changes outside the labeled set name an AI assistant as co-author in their merged commit message, which the rules do not search, so both groups hold agent work in proportions nobody recorded. At least five reverted changes carry the label, too few to compare the groups.

What to record so the metrics hold

These choices are hard to reconstruct afterward and easy to settle with a record kept alongside each change: when the task was assigned and by whom, which agent, model and settings produced which commits, which release first carried the change, and, for a revert or a fix forward, the reason and the deployment it belongs to. Each evidence item should also state how it relates to the producer: whether whoever issued it is the producer, shares a model or context with it, is technically separated from it or is organizationally independent.

Where Kealu Vector fits

The Agentic Engineering Assurance Layer is the proposed category for the layer that governs how agentic engineering work is performed and evidenced, so that humans can trust and accept the resulting work. Our white paper AI Has Moved the Bottleneck sets out the problem it answers, and the questions raised here hold whatever tools a team uses to answer them. Kealu Vector, currently designated Vector Assurance Assist, is our early-access implementation of that category.

FAQ

What is change failure rate?

Change failure rate, which DORA calls change fail rate, is the share of deployments that need immediate intervention after they go out, usually a rollback, a hotfix or a fix forward. The denominator is deployments, so several failed changes in one release count as one failure, while bundling many changes into each release makes each release more likely to carry a failed change. A team also has to decide what counts as a deployment for software that ships as releases.

What is lead time for changes?

Lead time for changes is the time a change takes from being committed to version control to running in production, which DORA collects in ranges from less than one hour to more than six months. Measuring it means choosing where the clock starts, at the first commit, the pull request or the request for the work.

How do you calculate change failure rate for agent-written code?

Divide the deployments that needed remediation by all deployments, and record which changes each deployment carried and who or what produced them. Reverts are an incomplete signal: in one public repository most reverted changes never reached a release, and some reverts were not failures. The agent share needs a label recorded when the change is made, since detection afterward misses agent work.

Do coding agents improve DORA metrics?

In the repository we recomputed, agent-labeled pull requests merged a median 1.6 hours after opening, and their lead time from commit to release was about twenty days, close to other changes, because releases set the pace. One repository cannot settle the question, and it suggests that on a release schedule a faster merge moves lead time little unless the cadence moves too.

On Google

Add Kealu as a preferred source: our articles get highlighted in Top Stories, AI Mode and AI Overviews.

Add as a preferred source on Google

Talk to our team.

Questions about this article, or about using Kealu Vector in your organization? Send us a message.

Talk to our team.