AI in Software Engineering: What's Real, What's Hype

The Extremes: What Could Go Right — and Wrong

Worst-Case Scenarios That Have Already Happened

• Amazon (US), 2025: An AI coding agent autonomously deleted a production environment, causing a 13-hour AWS (Amazon Web Services) outage — one of the year’s most visible infrastructure failures
• Klarna (Sweden), 2025: After cutting workforce from 5,527 to ~3,000 partly using AI, customer satisfaction collapsed — CEO admitted “we went too far” and began rehiring humans
• Lovable (Sweden): 170 out of 1,645 AI-created web applications had vulnerabilities exposing personal data to anyone who looked
• Samsung (South Korea), 2023: Engineers accidentally leaked proprietary semiconductor source code by pasting it into ChatGPT prompts — the company subsequently banned generative AI tools internally
• Supply chain attacks (global): AI models hallucinate non-existent software packages in ~20% of code samples. Attackers register those package names with malware. At least one confirmed malicious package was found on npm (Node Package Manager), pulling 233 downloads/week

Best-Case Scenarios With Evidence Behind Them

• Amazon Q Developer (US): Migrated 30,000 applications from Java 8/11 to Java 17, saving an estimated 4,500 developer-years and ~€240M ($260M) — a specific, repeatable, well-defined migration task where AI excels
• Spotify (Sweden), internal “Honk” system: Over 1,500 merged AI-generated pull requests (code changes submitted for review), with 60-90% time savings on fleet-wide code migrations
• Booking.com (Netherlands): Measured a 16% productivity lift across its engineering organisation using structured measurement frameworks
• Google (US) internal trial: 21% speed improvement in a 96-engineer randomised controlled trial on real development tasks
• Alibaba Cloud (China), 2025: Reported that its Tongyi Lingma coding assistant surpassed 7 million users, with developers accepting AI suggestions on roughly 30% of code completions — scale adoption in a non-Western ecosystem

The Productivity Evidence: What Studies Actually Show

The number you’ll hear most often is GitHub’s claim that Copilot makes developers 55.8% faster. That figure comes from a single 2023 study where 95 freelance programmers built one specific task in a lab setting. The confidence interval ranged from 21% to 89%, making it almost meaninglessly wide. No large-scale independent study has replicated anything close to this number.

What independent research actually shows is a spectrum of conflicting results — and the highest-quality study produced the most sobering finding.

The METR (Model Evaluation & Threat Research) study, published July 2025, is the most methodologically rigorous investigation to date. Sixteen experienced open-source developers tackled 246 real-world tasks on codebases they knew intimately, averaging over a million lines of code each. The result: developers using AI were 19% slower . The most striking finding wasn’t the slowdown but the perception gap — before the study, developers predicted a 24% speedup. After completing tasks more slowly, they still believed they’d been 20% faster. They were wrong about their own productivity by nearly 40 percentage points.

On the positive side, Google’s internal randomised controlled trial with 96 engineers found a 21% speed improvement . A multi-company field experiment across 4,867 developers at Microsoft, Accenture, and a Fortune 100 firm found participants completed 26% more tasks — but junior developers benefited far more (35-39% speedup) than seniors (8-16%). McKinsey’s controlled studies showed routine tasks completed up to twice as fast, but less than 10% time savings on complex tasks.

The DX research group, tracking 400 companies over time, found pull request throughput increased just ~10% . The 2024 DORA (DevOps Research and Assessment) report, based on 39,000 professionals, found that higher AI adoption correlated with slightly worse delivery outcomes — suggesting that without workflow changes, more AI doesn’t mean better results.

NOISE vs SIGNAL: Productivity Claims

Why the Numbers Contradict Each Other

If most studies show 10-25% productivity gains but the most rigorous study found a 19% slowdown , something needs explaining. This isn’t a data error — it’s a pattern that reveals what AI coding tools actually do well and where they fail.

The studies showing gains (Google’s 21% , the multi-company 26% more tasks ) measured isolated coding tasks — writing a function, completing a ticket, generating a test. Within the narrow act of producing code, AI tools genuinely accelerate output. Nobody seriously disputes this.

The METR study measured something different: end-to-end task completion on large, familiar codebases. This includes reading existing code, understanding context, planning an approach, writing code, debugging, and verifying. The experienced developers in the study spent significant time prompting, reviewing AI output, correcting hallucinations, and re-doing work the AI got wrong — time that exceeded the time saved on raw code generation. Their familiarity with the codebase meant they could already work fast without AI; the overhead of managing the tool outweighed the acceleration it provided.

The reconciliation is straightforward: AI tools accelerate code production but can slow down software engineering. Code production is perhaps 30-40% of what a developer does. The rest — reading, designing, debugging, reviewing, communicating — is either unaffected or made harder by AI (particularly the review burden). When studies measure the narrow task, gains appear. When they measure the full workflow, gains shrink or reverse.

This matters for decision-making because it predicts where gains will materialise. Greenfield projects with well-defined specs, boilerplate-heavy work, and junior developers doing structured tasks will see real gains. Complex maintenance work on mature codebases by experienced developers — the majority of real engineering — may not.

The Adoption Map: Where Companies Actually Stand

Adoption among developers is genuinely widespread but shallow. The JetBrains 2025 survey of 24,500+ developers found 85% regularly use AI tools, but the Stack Overflow 2025 survey (49,000+ respondents) revealed that positive sentiment has declined to 60% from over 70% the previous year. Only 43% of developers trust AI output accuracy, and more than half report not using AI agents at all.

Enterprise adoption tells a different story depending on geography. Eurostat data from December 2025 shows 20% of EU enterprises (with 10+ employees) use AI technologies, compared to roughly 78% of US organisations. Within Europe, the Nordic countries lead (Denmark 42%, Finland 38%, Sweden 35%) while Eastern Europe trails significantly (Romania 5%, Poland 8%). China’s adoption picture is harder to quantify, but Alibaba’s Tongyi Lingma alone reports over 7 million developer users, and Baidu’s Comate claims similar scale. India’s adoption is accelerating through the outsourcing sector, with firms like Infosys and Wipro rolling out AI-assisted coding across tens of thousands of engineers to maintain competitive positioning.

What separates leaders from the rest

The top companies getting real value share specific traits that go far beyond tool selection. Shopify (Canada) provides the most aggressive adoption model: CEO Tobi Lütke’s April 2025 internal memo declared AI usage a baseline expectation, integrated into performance reviews and headcount decisions. Spotify (Sweden) built a custom internal agent system. Booking.com (Netherlands) invested in structured measurement frameworks. Mercado Libre (Latin America) reported a 25% reduction in development time after deploying AI coding tools across its engineering teams. These companies share one commonality: they restructured processes, not just installed software.

Only 22.5% of organisations globally have published formal AI coding policies. Only 1% of organisations have reached what McKinsey considers mature deployment.

NOISE vs SIGNAL: What Leaders Do Differently

The Tooling Reality: What Works, What Costs, Where It Fails

The market has consolidated around a handful of dominant tools. Here’s what matters for procurement decisions.

GitHub Copilot remains the most widely adopted, with over 50,000 organisations and 1.8 million paying subscribers. Business tier pricing runs €17.50/user/month (~$19). It excels at inline code suggestions for popular frameworks but is increasingly seen as weaker than competitors for complex multi-file tasks. The acceptance rate for its suggestions hovers around 25-35% — meaning two-thirds of what it generates gets discarded.

Cursor has become the preferred tool among individual developers and small teams, approaching €920M ($1B) in annual revenue. Its Pro plan costs €18.40/month (~$20), but a mid-2025 switch to usage-credit billing caused backlash — heavy users discovered the plan could effectively cost €18.40/week with premium model usage. Budget predictability is poor for intensive use.

Claude Code (Anthropic, US) achieves the highest scores on SWE-bench Verified (a benchmark measuring AI’s ability to resolve real software issues) at 80.9% and is widely regarded as the most capable for deep reasoning and architectural changes. But costs scale aggressively: average API (Application Programming Interface) costs run €92-170/developer/month (~$100-185), and one tracked heavy user generated ~€12,700 ($13,800) in eight months.

Amazon Q Developer at €17.50/user/month (~$19) delivered the most impressive migration case study — 30,000 applications across Java versions, saving an estimated ~€240M ($260M) in annualised efficiency. This was a specific, well-defined, repetitive task — exactly the sweet spot for AI tools. The numbers come from the vendor selling the tool.

Tabnine (Israel) occupies a unique niche for regulated industries: fully air-gapped deployment (no data leaves your infrastructure), training only on permissively licensed open-source code, at €35.90/user/month (~$39). For organisations with strict data sovereignty requirements — defence, healthcare, finance, or companies under GDPR (General Data Protection Regulation) constraints — it remains the primary option despite declining market share.

Devin (Cognition, US), marketed as an autonomous AI developer at €425/month ($500), illustrates the gap between vendor claims and reality most starkly. Cognition claims a 67% PR (Pull Request) merge rate, but independent testing by Answer.AI found only 3 out of 20 tasks completed successfully — a 15% success rate. It works for contained migration tasks but fails on anything requiring architectural judgment.

The Money Question: ROI, Headcount, and What Happened at Klarna

The Simple ROI Arithmetic

A Copilot Business licence costs €210/developer/year (~$228). A fully loaded developer in a Western economy costs €75,000-130,000/year depending on location (higher in Nordics and Switzerland, lower in Southern and Eastern Europe, roughly €110,000-150,000/$120,000-165,000 in the US). The tool pays for itself if it saves roughly four hours per year — about five minutes per week. Self-reported data from a survey of 135,000 developers shows average savings of 3.6 hours/week, which at median Western rates translates to roughly €8,000-10,000/developer/year in value. Even at conservative estimates, the return on subscription costs alone ranges from 250% to over 800%.

This calculation is real — but it only covers the licence. It ignores the hidden cost stack above.

The Klarna Case Study: The Cautionary Tale

The Junior Developer Crisis

The impact on junior hiring is the most consequential trend for long-term workforce strategy, visible globally. A Harvard study tracking 62 million workers across 285,000 firms found that junior employment in AI-adopting firms dropped 7.7-10% within eighteen months. Entry-level tech postings fell 67% in the US and 46% in the UK during 2024. India’s NASSCOM reported similar trends, with entry-level IT services hiring falling sharply as firms deploy AI-assisted coding to boost output from existing senior staff. More than half of engineering leaders surveyed globally plan to hire fewer juniors due to AI tools.

The counter-argument is sharp: the multi-company controlled trial showed juniors benefit most from AI (35-39% speedup), suggesting AI-equipped juniors could be more valuable, not less. AWS CEO Matt Garman called the junior hiring decline “one of the dumbest things I’ve ever heard,” asking who will have learned anything in ten years. The US Bureau of Labor Statistics still projects 15% growth in software developer jobs through 2034.

The tension is between short-term cost optimisation and long-term talent pipeline sustainability — a calculation that plays out differently depending on local labour markets, employment protections, and hiring culture.

NOISE vs SIGNAL: The Headcount Question

Security and Technical Debt: The Unresolved Risks

Security: The Numbers Are Alarming

AI-generated code introduces critical web security vulnerabilities (OWASP Top 10 — the industry-standard list of most dangerous software weaknesses) in 45% of cases according to Veracode’s 2025 report. Cross-site scripting vulnerabilities appear in 86% of AI code samples; log injection in 88%. Research across Fortune 50 enterprises found AI-generated code introduced over 10,000 new security findings per month, with privilege escalation paths jumping 322%.

Perhaps more dangerous than the vulnerabilities themselves: a Stanford study demonstrated that developers using AI assistants wrote significantly less secure code and were more likely to believe their insecure code was secure. The false confidence effect is a systematic risk multiplier — the same perception gap that distorts productivity estimates also distorts security confidence.

Samsung’s 2023 incident, where engineers leaked proprietary semiconductor code through ChatGPT prompts, triggered a wave of internal AI bans across major corporations. It remains the most cited case for data leakage risk — and it happened not through a sophisticated attack, but through routine developer behaviour.

A new attack vector called “slopsquatting” has emerged globally — AI models hallucinate non-existent software package names in roughly 20% of code samples. Attackers register those names with malicious code. At least one confirmed malicious package was found on npm. The attack surface scales with AI adoption.

Technical Debt: Accelerating Measurably

GitClear’s analysis of 211 million changed lines of code found an 8x increase in duplicated code blocks during 2024. AI tools generate substantially more code than humans for identical requirements — one hands-on comparison found 186 lines versus 29 for a REST API endpoint. Code discarded within two weeks of being written (a proxy for wasted effort) is projected to double. MIT’s Armando Solar-Lezama described AI as a “brand new credit card” for accumulating technical debt.

The Review Bottleneck

This is the structural challenge that most organisations are missing. AI tools generate code faster, but the downstream review burden grows disproportionately. Telemetry across 10,000 developers showed the pattern clearly: 98% more pull requests generated, 91% more review time consumed , and flat delivery metrics. Only 3.8% of developers report both low hallucination rates and high confidence shipping AI code without human review. Senior engineers spend 4.3 minutes reviewing each AI-generated suggestion versus 1.2 minutes for human-written code.

Review capacity — not code generation speed — is now the binding constraint on delivery velocity.

Regulatory and Legal Reality

The EU AI Act

AI coding tools like Copilot or Cursor fall into the limited or minimal risk categories under the EU AI Act — they are not classified as high-risk. The primary obligations are transparency (disclosing when content is AI-generated) and AI literacy. Article 4, effective since February 2025, requires organisations to ensure staff has sufficient training on AI operation and risks. Penalties for non-compliance reach up to €35M or 7% of global annual turnover . The GPAI (General-Purpose AI) Code of Practice, published July 2025 and signed by major providers including Amazon, Anthropic, Google, Microsoft, and OpenAI, requires transparency about training data origins.

GDPR and Data Sovereignty

Developers routinely send code containing personal data — credentials, PII (Personally Identifiable Information) in variable names, database connection strings — to external AI services. Under GDPR, this constitutes data processing requiring a legal basis and potentially a DPIA (Data Protection Impact Assessment). GitHub launched EU data residency for Enterprise Cloud in October 2024, but Copilot’s AI inference is not restricted to EU data centres — requests route to the nearest available server. For organisations under strict data sovereignty requirements, this is a significant gap. Samsung’s incident shows this risk is not theoretical.

Copyright: Globally Unresolved

AI-generated code without meaningful human creative input likely does not qualify for copyright protection under current EU interpretation. In the US, the Copyright Office has taken a similar stance — purely AI-generated output is not copyrightable. A EU Copyright Directive review scheduled for June 2026 may address AI-specific provisions. In the meantime, the IP status of AI-generated code remains uncertain in every major jurisdiction.

What to Do Monday Morning

If you’re a leader (structured AI adoption in place)

• Shift measurement from coding speed to end-to-end delivery metrics — cycle time, deployment frequency, change failure rate
• Invest in review capacity scaling — this is now your binding constraint, not code generation
• Run security audits specifically on AI-generated code — vulnerability rates are 45%+
• Develop a deliberate junior developer strategy that balances short-term efficiency with long-term talent pipeline
• Track hidden costs quarterly — not just licence fees

If you’re mid-pack (tools deployed, workflows unchanged)

• Publish a formal AI coding policy — only 22.5% of organisations have one; this is table stakes
• Restructure review processes before expecting productivity gains — without this, more AI means more bottlenecks
• Deploy security scanning as a hard gate on all AI-generated code
• Start measuring actual delivery outcomes, not developer self-reports — the perception gap is 20-40 percentage points
• Budget for hidden costs: €1,750-3,700/developer/year (~$1,900-4,000) beyond licences

If you haven’t started

• Begin with GitHub Copilot Business at €17.50/dev/month (~$19) — lowest risk, broadest evidence base
• Pilot with a single team on well-defined tasks: test generation, documentation, boilerplate code
• Complete a data protection impact assessment and establish privacy-compliant data handling before scaling
• Set up baseline delivery metrics now — you cannot measure improvement without a starting point
• Don’t skip governance: learn from Amazon’s outages, Klarna’s reversal, and Samsung’s data leak before making your own mistakes