Why We Can't Prove AI Increased Engineering Capacity

AI has created a strange executive problem. Companies can count licenses, adoption, prompts, accepted code, and minutes saved. Yet when planning season returns, the question that matters is brutally simple: did the investment actually buy more engineering capacity? That is what the business case promised—not faster boilerplate, but more products, shorter roadmaps, and less pressure to add headcount. Software engineering has no broadly accepted way to prove it delivered any of that.

The research only sharpens the problem. Google and Microsoft have reported meaningful gains in developer productivity. METR found experienced open-source developers became slower on realistic maintenance work even while believing they were faster. DORA has found strong perceptions of productivity improvement alongside a far less tidy picture at the organizational level. These findings are often treated as competing verdicts. More likely, they are measuring different parts of the same system.

Engineering capacity, in this context, is an organization's ability to repeatedly turn business ideas into production software. That includes far more than typing speed. Architecture, coordination, requirements, testing, security, deployment, and operations all shape what an engineering organization can actually absorb and deliver. Once capacity is framed that way, the current argument over AI starts to resemble a much older argument about software measurement itself.

By the late 1960s and early 1970s, software organizations often treated lines of code as a practical proxy for output. The logic was straightforward: software was code, so more code appeared to mean more work. The measure eventually collapsed under its own contradictions. Strong engineers often created more value by deleting code, simplifying systems, or avoiding unnecessary features. Code mattered, but volume was a poor substitute for value.

In 1979, IBM's Allan Albrecht introduced Function Point Analysis as an attempt to measure delivered business functionality rather than implementation volume. The method had its own limits, but the shift was important. Software measurement was beginning to move away from counting what developers produced and toward estimating what users received.

Fred Brooks had already pushed the profession toward a deeper break with factory thinking. Published in 1975, The Mythical Man-Month argued that software work could not be reduced to interchangeable units of labor. Brooks's Law became the famous line, but the larger point was more consequential: communication, coordination, learning, and integration are part of the work. An engineering organization behaves like a system, and its capacity cannot be calculated by simply adding up the apparent productivity of the people inside it.

The next several decades expanded that systems view. Tom DeMarco's Controlling Software Projects, published in 1982, helped establish measurement as a serious management discipline. Martin Fowler's Refactoring, first published in 1999, made the long-term cost of poor design harder to ignore; architecture affects not only today's delivery but tomorrow's room to move. In the mid-2010s, Daniel Vacanti's work on flow metrics pushed teams to study how work actually moves rather than endlessly refining estimates. DORA, launched in the 2010s and popularized further through Accelerate in 2018, gave engineering leaders a practical language for delivery speed and operational stability.

None of these developments produced a single measure of capacity. They produced better views into different parts of the engineering system. The profession learned to measure functionality, flow, deployment performance, reliability, and the drag created by technical debt. It became much better at describing the condition of an engineering organization without ever settling on a direct measure of how much that organization could reliably accomplish.

That history explains why the current AI studies can all be credible and still leave executives unsatisfied. Researchers can measure whether a developer completes a bounded task faster. They can measure acceptance rates, cycle time, or perceived productivity. Executives are asking whether the organization can take on another product, shorten a roadmap, reduce hiring pressure, or respond to a new market demand. Those are related questions, but they are not the same question.

A developer completing a task thirty percent faster does not automatically create thirty percent more customer value. Requirements may still be unclear. Architecture may still resist change. Teams still wait on one another. Testing, security, deployment, and operational support do not disappear because code arrived sooner. Brooks's systems argument remains relevant fifty years later because local acceleration can be absorbed by constraints elsewhere.

So the original question remains: did the investment buy more engineering capacity? It is the right question, but the profession lacks a settled baseline against which to answer it. Software engineering has become remarkably good at measuring the components surrounding capacity while leaving capacity itself largely inferred.

AI may increase engineering capacity. In many organizations, it probably already has. Proving it requires more than a faster coding benchmark or a developer survey. It requires a clear definition of capacity, a credible baseline, and evidence that the organization can repeatedly deliver more meaningful work without simply moving the bottleneck. Software engineering is not struggling to measure AI's effect on capacity. It never learned how to measure capacity in the first place.

Continue Exploring

• Fred Brooks — The Mythical Man-Month (1975)
  Brooks's lasting contribution was not simply Brooks's Law. It was the idea that software organizations behave as systems rather than factories, fundamentally changing how productivity and coordination are understood.

• Tom DeMarco — Controlling Software Projects (1982)
  DeMarco helped establish disciplined measurement as a core management practice, laying much of the foundation for how engineering organizations think about metrics today.

• Martin Fowler — Refactoring (1999) and writings on Technical Debt
  Fowler's work illustrates that architecture influences an organization's future ability to deliver software, making technical debt a long-term capacity concern rather than simply a quality issue.

• Daniel Vacanti — Actionable Agile Metrics for Predictability (2015)
  Vacanti argues that organizations understand themselves better by observing flow than by refining estimates, extending the profession's shift toward systems thinking.

• DORA and Accelerate (2014–2018)
  DORA transformed how engineering leaders evaluate software delivery through deployment frequency, lead time, change failure rate, and recovery time. These measures reveal the health of delivery systems, even if they stop short of measuring engineering capacity itself.

• Recent AI Productivity Research
  The Google, Microsoft, METR, and DORA studies are all valuable. Their results are not necessarily wrong; they answer different questions than the one executives ultimately care about.