AI in Software Development: Expectations vs Reality
It’s a basic truth that using AI tools help increase productivity in software development. Right?
Well, not always. Paradoxically, there are cases where using AI tools may even hinder development. Read on to find out more.
When Using AI in Software Development is a Bad Idea: Lessons from the Front Lines
Over the past couple of years, many companies have invested heavily in AI-powered developer tools, driven by the promise of faster coding, higher productivity, and leaner teams. But what happens when those promises are tested in the real world?
A new randomized controlled trial (RCT) conducted by METR in early 2025 offers a sobering finding: experienced open-source developers actually performed 19% slower on real-world tasks when assisted by state-of-the-art AI tools compared to when they worked without them.
This result is particularly striking given that the participants — highly skilled contributors to large open-source codebases — expected to be 24% faster with AI and felt they had improved by 20% after using it. The contrast between perception and measurable performance raises important questions for business and technology leaders considering enterprise-wide AI rollouts.
Why Didn’t AI Improve Productivity?
The developers in METR’s study worked on mature, complex open-source projects — some with over a million lines of code. In such environments, the AI tools (Cursor Pro equipped with Claude 3.5 and 3.7 Sonnet models) faced several key challenges:
- High codebase familiarity: Senior developers already had deep knowledge of the architecture and conventions, leaving little room for AI to “shortcut” their work.
- Large and complex repositories: Today’s LLMs struggle to effectively process and reason over massive codebases. Relevant context often gets lost or misinterpreted.
- Tacit knowledge gaps: AI tools could not fully grasp unspoken project norms, naming conventions, or the nuanced expectations that seasoned developers navigate intuitively.
- Prompting friction: Using AI added interaction overhead — developers spent time crafting prompts, reviewing AI output, and often waiting for completions — time that didn’t always pay off.
- Low acceptance rate: Less than 44% of the AI-generated code suggestions were accepted without significant modification, limiting actual efficiency gains.
Implications for Business and Engineering Leaders
For executives exploring AI-assisted development, this study is a critical reminder that context matters deeply. AI tooling may not deliver a universal productivity boost — especially not in environments where developers are already experts in their codebases.
This doesn’t mean AI is a bad investment. But it does mean:
- Careful targeting is essential: You may see better returns in greenfield projects, less mature codebases, or where onboarding new engineers is costly.
- Don’t rely solely on developer perception: While the subjective experience was positive, actual performance lagged. Ongoing measurement and validation are crucial.
- Adoption strategies need depth: Business outcomes will depend on more than just access to AI tools — developer workflows, project complexity, and organizational knowledge must all be considered.
Perhaps most importantly, the study reinforces that AI productivity benefits are not plug-and-play — they require finetuning, monitoring, and a clear-eyed understanding of when and where the tools are likely to help (or hinder).
Final Thought
AI is rapidly evolving. What underperforms today may outperform tomorrow. But early experiments like METR’s offer a vital dose of realism in an environment filled with hype. For companies betting big on AI in software development, evidence-backed strategy will matter more than ever.
What about the effect of AI on developer productivity in your company (if you use AI tools to support development)? Is your experience overwhelmingly positive, or do you see drawbacks like the one outlined in this article? Let me know in the comments!
