Skip to content
Founderz: Responsible Use of AI home

in collaboration with Microsoft

All case studies

Case study · Research

METR: perceived and measured productivity with AI

Experienced developers felt 20% faster with AI, but their tasks took 19% longer. A 2026 follow-up pointed the other way, and METR calls its results unreliable.

Principles involved

When and where

2025 to 2026

Worldwide (METR studies)

What happened

In early 2025, the research organization METR ran a randomized study with 16 experienced open-source developers working on 246 real tasks in projects they knew well. Before starting, the developers expected AI tools to make them 24% faster. Afterward, they believed AI had made them 20% faster. Measured, tasks with AI took 19% longer.

In February 2026 METR published an update on a larger follow-up, with 57 developers and more than 800 tasks. Its results pointed the other way, but METR called them unreliable, because many developers avoided tasks they didn’t want to do without AI. METR now thinks developers are probably faster with current tools, says the evidence is weak and is redesigning its study.

What the sources establish

The 2025 study measured a specific group: experienced developers in familiar code, with early-2025 tools. It’s a preprint, not peer reviewed, and METR has marked it as out of date. The follow-up changes the picture on speed. The finding most useful to everyone else still stands: in 2025, those 16 developers misjudged their own speed by a wide margin.

Implications

How productive AI feels is weak evidence of how productive it is. Tools change fast, so today’s measurement may not hold next year either. When a decision depends on the benefit, measure it on your own work.

Recommended practice

  1. 01Time a few real tasks with and without AI before you change how a process works.
  2. 02Decide what would convince you it isn’t working before you look.
  3. 03Measure again when the tools change.

Sources

What each source establishes, and its limits. The practices and recommendations on this page are ours, and the facts come from the sources. See every source we use.

  1. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity METR · July 10, 2025 · Research (preprint) Sixteen experienced developers working on 246 tasks expected AI to make them 24% faster and felt 20% faster afterward, but tasks with AI took 19% longer. Limits: Not peer reviewed, with early-2025 tools. METR has since marked the result as out of date.
  2. We are Changing our Developer Productivity Experiment Design METR · February 24, 2026 · Research update A follow-up with 57 developers and more than 800 tasks pointed toward speed-ups, but METR calls the results unreliable because many developers avoided tasks they didn’t want to do without AI; it thinks developers are probably faster now and is redesigning its study. Limits: METR describes the evidence as weak.

Cite this page

Founderz (2026). METR: perceived and measured productivity with AI. The RUAI Standard, 2026 edition. Developed by Founderz in collaboration with Microsoft. https://responsibleai.founderz.com/toolkit/case-studies/metr-perceived-and-measured-productivity

Licensed under CC BY 4.0: share and adapt with attribution.