On that NBER working paper, I think it over states the productivity effect meaningfully. Their sample is weighted towards small open source projects and does not modulate for quality.
The below is the best I've been able to find on the impacts of LLM use in large commercial organizations. It's an observational study covering 22000 developers across 4000 teams over 2 years comparing their lowest AI adoption quarter to their highest within their org.
Your post here is the mechanism and I think the Faros data is the downstream outcome. And it makes sense. As your codebase becomes mature, changes have to be made such that the product is stable. If you're filtering human work through an unreliable mechanism, you're going to introduce changes that start to subtract value.
I wrote this in Talk is Cheap:
"Here’s the difference I see - in every case where a radical new technology has revolutionized industry - manufacturing machines, plumbing, planes, electricity, computers, the internet, etc - there is an increasing trend of reliability. Each one of those foundational technologies started from a place of unreliability and moved to a place of very high reliability. They became foundational because humans learned to trust them implicitly."
Assuming the reliability problem remains sticky, I'm really curious about what the world looks like in 2,5,10 years. Surely there's value in even unreliable AI. We won't be automating away all the jobs, but hopefully we'll have found better uses than the spray-and-pray workslop approach we're doing now. Software development will obviously be accelerated, but I think there will also be stranger applications that we figure out.
Thank you! And same here, don't doubt there will be a lot of value even if reliability issues remain. Even just modestly increasing productivity across most knowledge work would be a massive deal in aggregate.
Really appreciate this post. As you very concisely outline, there is too much focus on (top) capability, rather than reliability. METR is quite explicit about evaluating 50%/80% success rates, but this gets lost in translation. I do wonder about who the piece is mostly targeting. A few thoughts.
- Yes, we have not seen the productivity boom, but to be fair one of the first Brooking links (by Baily and Brynjolfsson) explicitly stated that we would be facing a j-curce and should not expect an immediate productivity increase.
- If OpenAI and/or Anthropic were just about to automate large parts of white-collar work, their valuations should be much higher than (almost) a trillion? My laymen interpretation is that the market does not expect the full automation, but GenAI to be helpful and be able to fully automate some tasks, and augment other task (bundles).
- Reliability belongs to systems, not models, and these systems require scaffolding, in particular for tasks that are not exceptionally well-defined and somewhat static. Many companies are working on how to create such systems, and many are probably realizing that it is more difficult than they had hoped or even anticipated. Thus, more botsitting and botshitting, than real agents doing somewhat automated work.
Overall: True, we have not seen a productivity boom. But I don't think the markets nor the companies are expecting it. OpenAI should be worth much more, and companies are not - at all - beginning to lay off people.
Agree with all of this, markets are definitely not pricing in the super-optimistic scenarios. And like I say in the article, I do except some measurable productivity gains at some point, though not dramatic ones. But the degree of AI utopianism / dystopianism out there, including in very high places, is shocking, and I'm partially targeting this group. Although, I do hope the article is clarifying for those with more realist perspectives too.
As someone who studies medical error, where reliably getting stuff right is still incredibly hard, I appreciate this. I do suspect that some hybrid LLM-symbolic systems with verification will improve reliability. I also think as the reliability issue comes into clearer focus (thanks to articles like this), more research will focus on it, and we should see greater improvements.
I've followed your takes on AI development with interest for a while, but this piece is just another level. Easily the best analytical piece within the broad "AI future" field I've read in a long time. This is the kind of healthy AI skepticism we need more of - and there are still many more avenues to explore. Kudos!
If “The unavoidable price of reliability is simplicity.” [C.A.R. (Tony) Hoare’s Turing Award lecture, The Emperor’s Old Clothes (1980)] then the proverbial 'unforeseen and unwanted consequences' of extreme complexity (trillion+ parameter Gen AI that is not understandable) + agency (Agentic AI) may be unfathomable system fragility.
The reliability gap maps cleanly to clinical AI because unreliable output is not just lower quality; it changes workflow ownership. For an OR-adjacent agent, I would want the eval to show what evidence it saw, what uncertainty stayed visible, and what condition forced it to stop before the room inherits a confident but brittle recommendation.
It can be and will prove increasingly useful but it's current unreliability renders the current valuations to completely unjustifiable. Chickens will roost.
I suspect the answer to AI unreliability is more AI. LLMs mimic humans, and humans are unreliable. How do we manage human unreliability? More humans.
The software industry is (maybe for the first time ever) a model here. Almost always when software fails it is because a human made a mistake. The software industry is built around processes of testing and review that screen errors down to an acceptable (but far from zero) level, and then react to address further errors as they appear. Almost the entire software lifecycle, except the tiny fun part at the start, is about addressing human cognitive failure.
Within that process there are multiple different human roles, but each of those roles can be formalized as encoding informal language into code or decoding code into informal language, so each of those roles can, potentially, be played by an LLM. Will they be as good as humans? Definitely not as good as the best humans. But will they be acceptable? Probably. And they're much faster, so to some extent you can replace ability with iteration speed. And if you have different models with only limited shared context writing specs, writing code, writing and running tests, doing hands on testing, filing bugs and reviewing code, and you do it faster than humans would be able to do it, is the net result as reliable as a human result? I don't see why it shouldn't be.
I have a suspicion though that the cost of inference is still too high though. Where we stand right now we could easily double the fully loaded cost of a developer if we let them use as many tokens as they might want to automate their whole workflow.
I actually thought about preempting this point in the article, but decided it'd take too long of a digression.
The basic issue is that, in practice, LLMs don't do a very good job of checking each other's work. There's a small, but interesting literature on this already. Here are a few papers to check out:
The main issue with your argument at a conceptual level is that you're conflating LLMs with a specific sub-category of high-competence humans who coordinate well relative to their context. But an arbitrarily chosen group of humans will often coordinate poorly, and if you read the papers above, you'll find that LLMs reproduce a lot of human-like coordination failures, such as group-think and propagating false beliefs.
Thank you for the references. They are interesting, but I actually think they point to different and more tractable problems than the one you suggest. No-one who ever worked in a software engineering organization would suggest that the participants are particularly competent or that they coordinate well! Essentially nothing ever works on the first attempt and communication between the different engineering teams is notoriously unreliable and contentious. And yet we do in fact succeed in producing useful software, at least sometimes. This is why I think this particular case is fruitful to think about in terms of LLM reliability - why does the human engineering process work in spite of poor one-shot reliability and noisy communication?
Start with a simple voting model like the ones described in the “Consensus is Not Verification” paper. As the authors correctly say, in order for consensus to improve reliability the errors have to be uncorrelated. LLM answers are notoriously correlated, so this does not work. But this is also true for humans! Ask two people to write the same code and they will make very similar mistakes. So we don’t do that (of course, there are other reasons).
Instead we set up adversarial-collaborative relationships, for instance between testers and developers. To pass, all the tests and the code have to be correct. With enough tests, the probability of all the tests and the code having the same conceptual error drop. Now LLMs are more correlated than humans, but they also don’t dislike writing tests nearly so much. I don’t know whether academic work has been done on this, but anecdotally people seem to have had good results having separate Claude instances write and test their code.
At a very handwavy level, most of the papers you provided seem to point to architectural naivety in currently proposed multi-agent solutions. To get good results from this approach, you would have to both combine different results by “and-ing” them to eliminate errors and polling them to establish consensus. You would have to construct scaffolding and roles that pushed agents as far as possible into uncorrelated outputs. And you need the right framework of iteration to get convergence over time. You need to work with the nature of LLMs - they are faster than humans and tireless, but they are less reliable, less flexible and do not learn reliably, but nonetheless I think there’s some value in looking at human organizational structures as a starting point.
>I suspect the answer to AI unreliability is more AI.
The data presented in this post argues against this point. let's take a simple model. Say the LLMs you're working with have pass@1 of 50%.
If you just take the implementer LLM output and ship what it produces into production then you have a 50% chance on one run of getting a working implementation and 50% you ship something buggy.
If your verifier's check is uncorrelated to the correctness of what implementation is doing, you get no gain in reliability. You've just spent more tokens.
I'd also note, in only one case do you get the right code shipped into production - the top one so 25% chance of that.
And if you retry indefinitely on failure, the math converges to exactly your original 50% pass@1. Infinite retries with an uncorrelated verifier buys you nothing but token spend.
>LLMs mimic humans
Not in many important ways. One of the most important is reliability. When you do something regularly, you routinely get more and more reliable at that thing. LLMs do not.
>Almost always when software fails it is because a human made a mistake.
Right, this only works if implementer and tester agents make correlated correct decisions and uncorrelated errors, and if they can iterate and self-correct multiple times. The risk is that they make uncorrelated decisions, or make correlated errors. This happens with humans too. It happens more with LLMs, but LLMs can iterate more times and write more tests since they don't get tired or bored. The key is, can you construct the framework through which agents interact to eliminate the uncorrelated errors efficiently, and can you write the prompts to reduce the rate of correlated errors as far as possible. And once you've built that is it cost efficient?
If coding models don't learn at all (as opposed to just learning within their context window) and if their decisions are completely uncorrelated (as your math assumes) or too correlated, then any idea of using multiple agents to improve reliability will not work. But I don't think the situation is quite that bad. If it were that bad people would not find these things useful at all, and they clearly do.
People also find slot machines "useful" and junk food and cigarettes and opioid pills. Just because a lot of people do something or like something does not mean it's good for us.
There have been many times humanity has charged head long into fads that ended up being net negative for us. The question is - what are LLMs?
On that NBER working paper, I think it over states the productivity effect meaningfully. Their sample is weighted towards small open source projects and does not modulate for quality.
The below is the best I've been able to find on the impacts of LLM use in large commercial organizations. It's an observational study covering 22000 developers across 4000 teams over 2 years comparing their lowest AI adoption quarter to their highest within their org.
https://www.faros.ai/blog/ai-acceleration-whiplash-takeaways
Highlights:
1. 16% PR merge rate improvement
2. -11% release frequency (that's a negative sign)
3. +860% code churn increase (not a typo)
4. +480% feature lead time (also not a typo)
5. +250% incidents per pull request.
I analyzed this data here: https://unessays.substack.com/p/talk-is-cheap
Your post here is the mechanism and I think the Faros data is the downstream outcome. And it makes sense. As your codebase becomes mature, changes have to be made such that the product is stable. If you're filtering human work through an unreliable mechanism, you're going to introduce changes that start to subtract value.
I wrote this in Talk is Cheap:
"Here’s the difference I see - in every case where a radical new technology has revolutionized industry - manufacturing machines, plumbing, planes, electricity, computers, the internet, etc - there is an increasing trend of reliability. Each one of those foundational technologies started from a place of unreliability and moved to a place of very high reliability. They became foundational because humans learned to trust them implicitly."
Wow, this study is fantastic, bummed I didn't see it earlier and tempted to just add it in now.
Be my guest. Highly recommend reading my analysis as I think it contextualizes the data. (But of course I would think that).
I'll check it out!
Incredible article and breakdown.
Assuming the reliability problem remains sticky, I'm really curious about what the world looks like in 2,5,10 years. Surely there's value in even unreliable AI. We won't be automating away all the jobs, but hopefully we'll have found better uses than the spray-and-pray workslop approach we're doing now. Software development will obviously be accelerated, but I think there will also be stranger applications that we figure out.
Thank you! And same here, don't doubt there will be a lot of value even if reliability issues remain. Even just modestly increasing productivity across most knowledge work would be a massive deal in aggregate.
Really appreciate this post. As you very concisely outline, there is too much focus on (top) capability, rather than reliability. METR is quite explicit about evaluating 50%/80% success rates, but this gets lost in translation. I do wonder about who the piece is mostly targeting. A few thoughts.
- Yes, we have not seen the productivity boom, but to be fair one of the first Brooking links (by Baily and Brynjolfsson) explicitly stated that we would be facing a j-curce and should not expect an immediate productivity increase.
- If OpenAI and/or Anthropic were just about to automate large parts of white-collar work, their valuations should be much higher than (almost) a trillion? My laymen interpretation is that the market does not expect the full automation, but GenAI to be helpful and be able to fully automate some tasks, and augment other task (bundles).
- Reliability belongs to systems, not models, and these systems require scaffolding, in particular for tasks that are not exceptionally well-defined and somewhat static. Many companies are working on how to create such systems, and many are probably realizing that it is more difficult than they had hoped or even anticipated. Thus, more botsitting and botshitting, than real agents doing somewhat automated work.
Overall: True, we have not seen a productivity boom. But I don't think the markets nor the companies are expecting it. OpenAI should be worth much more, and companies are not - at all - beginning to lay off people.
Agree with all of this, markets are definitely not pricing in the super-optimistic scenarios. And like I say in the article, I do except some measurable productivity gains at some point, though not dramatic ones. But the degree of AI utopianism / dystopianism out there, including in very high places, is shocking, and I'm partially targeting this group. Although, I do hope the article is clarifying for those with more realist perspectives too.
As someone who studies medical error, where reliably getting stuff right is still incredibly hard, I appreciate this. I do suspect that some hybrid LLM-symbolic systems with verification will improve reliability. I also think as the reliability issue comes into clearer focus (thanks to articles like this), more research will focus on it, and we should see greater improvements.
I've followed your takes on AI development with interest for a while, but this piece is just another level. Easily the best analytical piece within the broad "AI future" field I've read in a long time. This is the kind of healthy AI skepticism we need more of - and there are still many more avenues to explore. Kudos!
That's very kind Jonno, thank you!
Very interesting article and ideas, thank you for sharing
Thanks Anna!
If “The unavoidable price of reliability is simplicity.” [C.A.R. (Tony) Hoare’s Turing Award lecture, The Emperor’s Old Clothes (1980)] then the proverbial 'unforeseen and unwanted consequences' of extreme complexity (trillion+ parameter Gen AI that is not understandable) + agency (Agentic AI) may be unfathomable system fragility.
Will have to check out that lecture! But yes agree with that basic conclusion.
Thanks for this. An excellent essay!
The reliability gap maps cleanly to clinical AI because unreliable output is not just lower quality; it changes workflow ownership. For an OR-adjacent agent, I would want the eval to show what evidence it saw, what uncertainty stayed visible, and what condition forced it to stop before the room inherits a confident but brittle recommendation.
It can be and will prove increasingly useful but it's current unreliability renders the current valuations to completely unjustifiable. Chickens will roost.
I suspect the answer to AI unreliability is more AI. LLMs mimic humans, and humans are unreliable. How do we manage human unreliability? More humans.
The software industry is (maybe for the first time ever) a model here. Almost always when software fails it is because a human made a mistake. The software industry is built around processes of testing and review that screen errors down to an acceptable (but far from zero) level, and then react to address further errors as they appear. Almost the entire software lifecycle, except the tiny fun part at the start, is about addressing human cognitive failure.
Within that process there are multiple different human roles, but each of those roles can be formalized as encoding informal language into code or decoding code into informal language, so each of those roles can, potentially, be played by an LLM. Will they be as good as humans? Definitely not as good as the best humans. But will they be acceptable? Probably. And they're much faster, so to some extent you can replace ability with iteration speed. And if you have different models with only limited shared context writing specs, writing code, writing and running tests, doing hands on testing, filing bugs and reviewing code, and you do it faster than humans would be able to do it, is the net result as reliable as a human result? I don't see why it shouldn't be.
I have a suspicion though that the cost of inference is still too high though. Where we stand right now we could easily double the fully loaded cost of a developer if we let them use as many tokens as they might want to automate their whole workflow.
I actually thought about preempting this point in the article, but decided it'd take too long of a digression.
The basic issue is that, in practice, LLMs don't do a very good job of checking each other's work. There's a small, but interesting literature on this already. Here are a few papers to check out:
https://arxiv.org/pdf/2603.06612v1
https://arxiv.org/pdf/2503.13657v1
https://arxiv.org/pdf/2509.05396
https://arxiv.org/pdf/2603.04474
The main issue with your argument at a conceptual level is that you're conflating LLMs with a specific sub-category of high-competence humans who coordinate well relative to their context. But an arbitrarily chosen group of humans will often coordinate poorly, and if you read the papers above, you'll find that LLMs reproduce a lot of human-like coordination failures, such as group-think and propagating false beliefs.
Thank you for the references. They are interesting, but I actually think they point to different and more tractable problems than the one you suggest. No-one who ever worked in a software engineering organization would suggest that the participants are particularly competent or that they coordinate well! Essentially nothing ever works on the first attempt and communication between the different engineering teams is notoriously unreliable and contentious. And yet we do in fact succeed in producing useful software, at least sometimes. This is why I think this particular case is fruitful to think about in terms of LLM reliability - why does the human engineering process work in spite of poor one-shot reliability and noisy communication?
Start with a simple voting model like the ones described in the “Consensus is Not Verification” paper. As the authors correctly say, in order for consensus to improve reliability the errors have to be uncorrelated. LLM answers are notoriously correlated, so this does not work. But this is also true for humans! Ask two people to write the same code and they will make very similar mistakes. So we don’t do that (of course, there are other reasons).
Instead we set up adversarial-collaborative relationships, for instance between testers and developers. To pass, all the tests and the code have to be correct. With enough tests, the probability of all the tests and the code having the same conceptual error drop. Now LLMs are more correlated than humans, but they also don’t dislike writing tests nearly so much. I don’t know whether academic work has been done on this, but anecdotally people seem to have had good results having separate Claude instances write and test their code.
At a very handwavy level, most of the papers you provided seem to point to architectural naivety in currently proposed multi-agent solutions. To get good results from this approach, you would have to both combine different results by “and-ing” them to eliminate errors and polling them to establish consensus. You would have to construct scaffolding and roles that pushed agents as far as possible into uncorrelated outputs. And you need the right framework of iteration to get convergence over time. You need to work with the nature of LLMs - they are faster than humans and tireless, but they are less reliable, less flexible and do not learn reliably, but nonetheless I think there’s some value in looking at human organizational structures as a starting point.
I don't think so. In multiple places.
>I suspect the answer to AI unreliability is more AI.
The data presented in this post argues against this point. let's take a simple model. Say the LLMs you're working with have pass@1 of 50%.
If you just take the implementer LLM output and ship what it produces into production then you have a 50% chance on one run of getting a working implementation and 50% you ship something buggy.
Now add a verifier LLM with the same pass@1:
pass implementation - pass verification - (50%) (50%) = 25% (good)
pass implementation - failed verification - (50%) (50%) = 25% (bad - fail)
failed implementation - pass verification - (50%) (50%) = 25% (bad - ship bug)
failed implementation - failed verification - (50%) (50%) = 25% (good - fail)
If your verifier's check is uncorrelated to the correctness of what implementation is doing, you get no gain in reliability. You've just spent more tokens.
I'd also note, in only one case do you get the right code shipped into production - the top one so 25% chance of that.
And if you retry indefinitely on failure, the math converges to exactly your original 50% pass@1. Infinite retries with an uncorrelated verifier buys you nothing but token spend.
>LLMs mimic humans
Not in many important ways. One of the most important is reliability. When you do something regularly, you routinely get more and more reliable at that thing. LLMs do not.
>Almost always when software fails it is because a human made a mistake.
Yes, but the evidence shows that LLMs increase the number of bugs shipped into production: https://www.faros.ai/blog/ai-acceleration-whiplash-takeaways
If your goal is to minimize defects, LLMs work against that goal.
Right, this only works if implementer and tester agents make correlated correct decisions and uncorrelated errors, and if they can iterate and self-correct multiple times. The risk is that they make uncorrelated decisions, or make correlated errors. This happens with humans too. It happens more with LLMs, but LLMs can iterate more times and write more tests since they don't get tired or bored. The key is, can you construct the framework through which agents interact to eliminate the uncorrelated errors efficiently, and can you write the prompts to reduce the rate of correlated errors as far as possible. And once you've built that is it cost efficient?
If coding models don't learn at all (as opposed to just learning within their context window) and if their decisions are completely uncorrelated (as your math assumes) or too correlated, then any idea of using multiple agents to improve reliability will not work. But I don't think the situation is quite that bad. If it were that bad people would not find these things useful at all, and they clearly do.
People also find slot machines "useful" and junk food and cigarettes and opioid pills. Just because a lot of people do something or like something does not mean it's good for us.
There have been many times humanity has charged head long into fads that ended up being net negative for us. The question is - what are LLMs?