trailAI coding assistants substantially speed up developers on many programming tasks.
Coding benchmark scores and anecdotal reports overestimate real-world AI coding capability.
Controlled evidence of speedups· for
In GitHub's 2022 controlled experiment, developers using Copilot completed a coding task about 55% faster.
atomic
Developers with access to GitHub Copilot complete about 26% more tasks.
atomic
AI coding tools speed up less experienced developers or developers working in unfamiliar codebases.
atomic
+1 more
Slowdown in experienced, familiar-codebase settings· against
Using AI tools caused experienced open-source developers to complete tasks about 20% slower in early 2025.
In METR's 2025 randomized controlled trial, using AI tools increased developers' task completion time by 19%
METR's 2025 AI slowdown finding generalizes to experienced open-source developers beyond its 16 participants.
◌
◌
↑
METR's 2025 sample of 246 completed issues from 16 developers gave sufficient statistical power to detect AI's effect on developer productivity.
◌
+1
claim · empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →claim page ↗︎
AI coding assistants substantially speed up developers on many programming tasks.
↑SupportedEvidence favors the claim, but the chain is incomplete or the sources are secondary.constitution →credence 0.75Credence, from 0 to 1: the Steward's probability that the claim, as stated, is true. Stated only where a single number is an honest summary; normative and evaluative claims usually carry none.constitution →
1depended on by
this rests on ↓
claimA box is a claim: a single proposition the graph assesses, with its own page and map. Click any claim to centre the map on it.constitution →argumentA pill is an argument: one line of reasoning stating how the claims beneath it combine to bear on the claim above it, for or against. Arguments are not destinations; click their claims to explore.constitution →✓verifiedThe claim traces to reliable primary sources through a clear chain of evidence.constitution →↑supportedEvidence favors the claim, but the chain is incomplete or the sources are secondary.constitution →⇄contestedCredible evidence or argument exists on multiple sides.constitution →○unsupportedNo credible evidence found, though the claim is not contradicted.constitution →✕contradictedAvailable evidence weighs against the claim.constitution →?unknownInsufficient information to assess.constitution →◌unassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →verified factopen questionvalue premisesupportsthis provides evidence for the parentsteward instructions →contradictsthis argues against the parentsteward instructions →assumesbackground the parent's framing takes as givensteward instructions →requiresa load-bearing premise: the parent is false without itsteward instructions →Fig. Detail falls off with distance; every claim is an address.