No wonder most people at Anthropic believe our jobs are in jeopardy, they've already made a system wherein the machines are treating them like pets.
I really don't get the tendency here from Anthropic to describe what's most likely churn? A high pressure amount of changes/PRs a day on a TUI to wrap a while loop? I'm not going to be a reductionist here so I'm going to cut it off there, I'm positive that the complexity of handling multiple terminal emulators is something that needs some handling (even if doing their own rendering engine was a self made problem). Yes I've vibe coded my own harness from scratch without writing any of it, I still know how it works.
I'm also not sure that they addressed point 2 in any actual capacity? They say "the system verifies itself" which is cool and is something that automated testing has done for decades, but they eschew the mental model of how the system works as a whole and prefer instead to have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
N.B. Changed "1000 PRs" to "A high pressure amount" to be vague since the article is vague here on what CC undergoes.
> have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
This is the crux. Take pause, because hidden here is a sea-change in our fundamental trust model of software...
There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug. By design, the system is too large for that, and everything hinges on Claude itself being able to find and understand the right slice of the system in any situation. As long as that very new invariant holds, everything will be all right. If it fails catastrophically, there is no recovery. You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.
The new trust model is: We trust Claude to always understand the system enough to debug anything or make any reasonable change we could want. The risk might be worth the reward. But it's worth explicitly laying out the deal we're making.
Tbf, it's worth pointing out that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
"Tbf, it's worth pointing that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
The real difference as I see it is that in the human maintained old style system, the human would take things slowly and ensure they understand any changes. Claude let loose will just say "yes, I know how to do this" and bork the system, because it doesn't have a concept of humbleness in the face of complexity!
Not only that, when you have a human designed and implemented system you know it's at least coherent enough that someone understood it at some point well enough to take responsibility for it. No such assurance with the llms.
Large company codebases are very much not like that. I've worked on systems (e.g., banking) that have accreted still-running code for as much as 60 years. Nobody understands how it works. Nobody ever understood how the whole worked, and many of the parts were never actually intended to work together.
But is this really a problem? If you have testing and you have version control via Git, you can roll back changes and investigate them.
I also don't even know if it's really true. Sure, maybe it's not coherent enough that a person understands it, but if an LLM wrote it, you can be ~99% certain (in my experience) that that LLM will understand it, and usually in less than the time it took to write it. And unlike humans, the LLMs don't die and take their knowledge with them, or change companies, or get ornery when you ask them to explain.
Also, in my experience, Claude doesn't really do what the parent is saying and dive head-first into complex problems and bork your code. It is fairly humble in the face of complexity. It constantly recommends breaking down larger problems into smaller ones that are more easy to grok.
But most importantly, anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful. Especially when they're writing code that's important. Unless you're an AI researcher pushing the bounds of what LLMs can do, which let's face it, the vast majority of programmers are not, it's almost always the case that you'll have your LLM writing intelligible code that you can understand and have it explain easily enough.
Sure, you can do that, but irreparable harm might have been already caused. Sure, that’s unlikely when your product is an LLM wrapper. But in something like a database, it could be catastrophic.
I mean it could be, sure. But I think people are generally speaking more careful with higher stakes code than they are with lower stakes code. I know a lot of people who are YOLO'ing it with LLMs, but who ask for more understanding, more reviews, more tests, etc., in trickier areas. Which is only natural. Just like people are more careful when carrying a baby than when carrying a pillow.
(Yes, there will be individual exceptions to this, just like with everything else in life.)
> anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful.
Is this a definitional argument? As in, if you’re not thoughtful, you must not be a decently skilled engineer? Because I see no reason to believe that people are going to be thoughtful about LLM-generated code. Just the opposite. It’s incredibly tempting to LGTM it.
> I see no reason to believe that people are going to be thoughtful about LLM-generated code.
I think that's true for low value code. But the more important and crucial and high-stakes code is, the more effort people put into it, generally. Even with LLMs.
I would love to know what the Venn diagram of people who complain about people not understanding their code, to people who can't describe the boot process of their system is.
I am not sure that "someone understood it well enough to take responsibility" means what you think it means. I have definitely come across old systems which have had addons bolted on to them in so many places that the overall behavior of the system is unpredictable and only mostly works.
Someone understanding something and the codebase being coherent are different things. I can only think that you have been lucky with your career placements if you are making that statement.
What you are saying is leaning on 2 premises that seem to not hold water
1. Code written by people is always more readable and maintainable than LLM code
2. Code written by an LLM is always less readable and maintainable
Can you really make those statements in all intellectual honesty?
I'd like that to be true, but many software service startups were stitching together node packages and selling the result long before LLMs became useful.
most large software companies are already too large for anyone to have a complete mental model of.
The idea that someone at say, Google, will be able to fit even half of the system architecture in their heads is not realistic. It also hinges on being able to find and understand the right slice of the system in any situation.
But we have all sorts of guardrails to make that easy - i.e logs, traces, etc
One of the fun things that you currently need to babysit Claude, etc, on, is that while they've been tuned to hell now to do more testing and even logging and such, they don't currently do enough log-and-debug-oriented specific passes/revisions to distinguish meaningful logging for meaningful debugging from noise that won't necessarily tell you what you need to know.
And you can ask them to address this. And you can ask them to look for that in all your PRs. Etc etc. And that helps a lot.
But it's not something that can be brute-forced in the same way "this failed a test, let me fix it" is. So without a long-term-memory brain holding all the pain of past incidents and knowing and applying ahead-of-time judgement about what to build, it's a challenge to turn the general guidance into the specifics for any unique piece of software.
They're trying to get there of course. We shall see what's next.
("Human code turns into this too" is also a curious rejoinder from the "don't read the code anymore" crowd because there are entire disciplines and areas of study that have emerged and evolved from the problems of large ad-hoc multi-faceted systems being difficult and risky to deal with.)
I don't think they're the same. AI code isn't just complex because it's huge.. it's complex because it makes no sense
If you have a large human-written system with a little thought put into it you should be able to find the specific pieces that handle the behavior you want and work on that, and if it's good then changing it should have a small blast radius and be covered with regression tests
In contrast, I've fully vibe coded a 15k line tool for myself with Opus and I have no idea how on earth it's architected after reading through it. It has "tests" but I don't trust them. It's just all over the place with no coherent design to it. It works, but it's making me more and more nervous to change it
Right, that's what I alluded to in my last sentence.
It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.
Having worked continuously this decade on one massively complex consumer facing app that interacts heavily with real-world scenarios, I can say that the problems of cohesion existed well before the agentic revolution. Tons of features built by myopic teams simply because they couldn't understand enough of the system (both online and offline, meat-space interactions) to make good design decisions. AI actually helps here by being able to perform deeper reading and analysis than any human can do. But then the risk is that it can also produce way more random non-cohesive features.
So far I kind of feel like it nets out to the same bottleneck being good judgment from decision makers. And you might say, no, leadership will just put more and more pressure to add more complexity just because we can and it makes it appear that we are moving fast. But when it comes to real-world apps, there is a backstop which is that things that are too complex and illegible end up not being fit-for-purpose for human problems.
Claude Code and other general purpose AI tools are not like this, because they are literally trying to optimize it for every single kind of knowledge work with zero constraints on use case or target audience other than chasing the next juiciest one they can with essentially unlimited resource. 99% of software is not like this, and when the dust settles I suspect software won't be anywhere near as different as the doomers or the maxxers are predicting right now.
Thank you! This is a huge sticking point for me. There is no knowing the code without experiencing the code. If nothing else, you have to read the tests to know they verify what you think they do. Even then, you might know it does what you want, but can’t know it doesn’t do anything it shouldn’t.
> wherein the machines are treating them like pets
I know some people will groan hearing the phrase again, but: "Reverse centaur." A centaur is a synergistic combination of two things, but now those things are in the wrong position: The human is the computer's tool.
Naturally, how "wrong" it is tends to depend on whether you ask the people who do all the work or the people who own all the computer-capital.
My favourite quote at the moment: "May not man himself become a sort of parasite upon the machines? An affectionate machine-tickling aphid?" Samuel Butler, Erewhon, 1872
When you get to the point that you are doing 1000 PRs I have to seriously wonder what even are you building, I hear about this a lot from a number of companies, but their “too many commits / too many PRs” problem never seems to give anyone from the outside the feeling they are innovating at groundbreaking speeds.
I think it's just a different focus on what matters.
There's clearly enormous (human) work that goes in to building and tuning their models.
The software they develop around their model is, relatively, trivial. It's not clear that enormous amounts of (human) energy need to go in to that code.
It seems largely a function of applying human energy towards the things that are marginally most important to generating you money and putting you in front of competitors.
I mean a harness around a model is pretty trivial, I’ve also built one (heavily relying on LLMs to build it).
With a few thousand lines of code over the course of a week or so of my limited evening energy, I have agentic loops, ide integration, and a limited selection of custom tools which are all very safe (no unintended changes), along with session management and awareness of git.
It works great, is customizable, and allows integration with any provider.
If they need 30k changes per day, it’s not due to the nature of the problem!
Oh, I mean I can't disagree there. But they've dug themselves into a hole where they're implementing custom ray tracing in the game engine that claude code runs on or whatever.
> implementing custom ray tracing in the game engine that claude code runs on
Haha, I don’t doubt it! Pretty damning overall for long-term maintainability of vibe-coded projects, even as the models are great at a lot of individual tasks
I mean, sure we can assume that (although the Anthropic article from the other day say that claude mostly builds Claude) but this is the Claude Code team which, I again am assuming so I could be wrong, focuses on the TUI/CLI interface to those models, not necessarily the models themselves. That coupled with the fact that this is how a large portion of their clientele interact with their moat, it would be odd to me if they didn't apply significant work towards this thing to keep the hype or productivity train moving
The Anthropic post the other day mentioned part of what helps Claude Code unlock some of this is the sheer ability to use more tokens than the rest to get something going and efficient.
This is like one of Google’s infrastructure blog. Google scale things are only relevant and appropriate for them. We had lots of bitter lessons from smaller engineering teams following the process with the justification that “this is how Google does it”.
Similarly, Anthropic can afford to do this because they have virtually unlimited tokens and labs to train models for their specific usecase. The rest of us need code comprehension lest we become dependent on continuous supply of tokens (which is a state these labs badly want to achieve)
So, unless you work at a lab, or have unlimited tokens guaranteed through the life of your company, it would do both the company and you good to continue to read the code and not let these 30k PRs per engineer be the norm.
An excerpt from the article makes my point
> Nobody on the team keeps a complete mental model of Claude Code in their head, and we don't think anyone needs to. When someone wants to understand a part of the codebase, they ask Claude to write an explainer for exactly that slice. It takes minutes and reflects the code as it is today.
Even then do we really have evidence that what they are doing is reasonable or practical for them at all? They are edging on like 3 9s uptime, what leaked of Claude code looked like soup, tons of security issues, they can't build a sandbox to save their company, and last I heard they aren't even profitable without adjusting their numbers for arbitrary things.
Think I'm good on taking advice from companies that aren't making money.
Exactly. What they’re doing, repeatedly, is setting a narrative that this is how software engineering should be. Always be reliant on tokens. Don’t worry about uptime or errors - agents can address it, don’t worry about security - agents can address it, don’t worry about code comprehension - agents will always be there for you.
I saw someone say the other day (in response to another blog post being well written) that Anthropic employees have a rule against using AI for writing/external comms? That seems clearly not the case here, normal people don't write like this:
> Most of the path is automated; a person's attention goes where the risk is
> The eval grows with every incident, and every change to the reviewer is scored against all of it.
100%, although if people are getting 90% of their interaction from a robot, one may start to pick up some vocal tics and colloquialisms from machine world. a kind of robo-pidgin.
It felt weird reading it because every once in awhile there were some well written sentences and coherent paragraphs, then it fell back into Claudisms.
this definitely seemed like something claude generated, i know because i frequently do all my research in chat and then ask it to generate a doc based on what we talked about and when i read what it generates the verbosity, the style it writes in makes me delete the whole thing. I would never subject readers to that.
> That seems clearly not the case here, normal people don't write like this:
I think when you repeatedly expose and immerse yourself to such a degree in reading and writing language, it brushes off on oneself. So I could also absolutely believe someone from Anthropic wrote this.
As a non-native, I see how susceptible I am to absorbing such style. There's no way at all for me to perceive anything wrong with the examples you present
It's a bit more subtle than things like "it's not X, it's Y". And even that isn't "wrong", it's just the way AI talks, and therefore an indicator that the author offloaded their thinking to an agent. I've read so much Claude output that it's easy for me to recognize its voice, the same way I could recognize a friend's voice in writing. The examples I quoted are a weird kind of circuitous sentence structure with a certain shape and rhythm that scream Claude to me. Here is how I might rewrite them in human speak:
> Most of the path is automated; a person's attention goes where the risk is
--> We automate as much as possible, focusing human attention on risky parts
> The eval grows with every incident, and every change to the reviewer is scored against all of it.
--> We run reviewer evals against every incident
But that's just my interpretation and my voice. I can't know for sure what the authors would have said if they took the time to write it themselves.
No human can stay on par with 30k pull requests per day ... they simply don't have control on what's going on and rely on AI itself to do the job.
PS: I really want to know what is the churn rate (and the cost per LOC) here. My gut is saying the cost per LOC is insane, and they are basically rewriting over and over the same code.
30k PRs is such a meaningless figure though, you can't make anything of it. It entirely depends what's living in that repo. Adding localized strings, configs, things that change frequently, can easily move numbers like this by orders of magnitude. How big are the PRs? Maybe there's an auto-bug-fixer agent that runs around the codebase making thousands of single-line fixes per day. Who knows. The number means nothing in isolation. We used to scoff at bad managers for measuring the performance of engineers using lines of code, now we're using the same figures to judge AI. We've become the bad managers.
Anthropic has amazing models. But they should really stop commenting on software practices. Most of what they say seem out of touch with what's happening in software shops. A lot of it seems hype driven. And the entire message looks to be written by AI.
It’s all marketing nonsense. Along with the “AI is dangerous” and “we’re all going to lose our jobs” messaging they do, it’s just to sell the services.
I'm thinking about high volume mass manufacturing as an example here. I like those "How it is made" type YouTube videos. You see a production line that has some kind of QA process where cameras + automated systems do spot checks at high speeds and these crazy actuators are punching out bad examples. These processes often also involve spot checks where some statistical sample is selected and viewed carefully.
I don't think we've gotten to that level of reliable review with code, but I don't see any reason we can't continue moving in that direction.
I really like those videos too, but a very big difference is, in manufacturing QA, the system outputs a very high number of copies of a very small number of possible types, so spot checks work fine from an operations research point of view, as the variance in output has a known upper bound.
For code, I think things go in the opposite direction: the system outputs artifacts that are all different among themselves (even more so now that we no longer distribute software via physical copies. Well, bits are always physical, but I mean we no longer ship floppies/CDs/DVDs).
That said, at least in my experience, code review has never been a direct part of factory-line-type QA. That QA is typically done by treating software as a black box and just verifying that it complies with a spec (often times based on a very loose definition of "spec" ...), and this is something that works equally well regardless of whether a human or a machine wrote the software. The type of QA that is done (among other things) by code reviews is checking whether the software's design is such that it will be economically feasible to maintain and extend it, for example.
The one part the code reviews takes a part of (but is in no way error-free) and that you just can't do with black-box testing (or it is much harder to do) is to look for things like backdoors. As our industry moves more towards "agents write software and tests validate that it works," I think a lot about "Reflections on Trusting Trust" and the new and wide attack surface added by people no longer reading the code shipped on their behalf.
That’s a category error. Software isn’t stamping out the same part again and again; if it were review would be simple. Analogies often mislead.
Of course Anthropic would love you to think you are heading towards that future so we have to bear in mind their livelihood depends on LLMs being the future of work and 30k changes a day being normal.
But why would we want 30k changes a day in a given product, what possible justification could there be?
I understand the skepticism and I can't claim I am right. But some points come to mind.
First, I think we overestimate how different code is from other code. We even joke about CRUD apps and how cookie-cutter they are. A lot of changes in those apps are: add some new features to a data access layer (SQL, ORM, whatever), add new routes, add new UI surface, etc. There is an entire industry around "Design Patterns" and people live by the DRY axioms. Things like Ruby on Rails exists because rebuilding these systems over and over is wasteful. I feel we are just moving further down that same path.
Second, your question of "why would we want" is totally backwards. Progress doesn't follow your desires. Progress doesn't require justification. Sometimes we adapt the environment to our needs, other times we adapt to a changing environment. Which case we are currently in is still undecided and you are free to fight for the future you prefer. But demanding an industry justify itself before it is allowed to change fails in the face of evolution. If it works, we will justify it in hindsight.
Those manufacturing videos are of someone making the exact same thing over and over. Months and years of testing go into ensuring that there is no change in the system, and continuous monitoring is put in place so that if any changes slip in, it can be undone. The design is set in stone, as it should be, and variations are bugs. There are months to years of cheap prototypes, iteration, design changes, pipeline tuning, and verification to ensure that nothing will need to change for the next million items.
If you want a software analogy to those devices getting churned off the assembly lines, it's the TCP streams coming out of your network card, with crazy checksums punching out bad packets.
We're orders of magnitude better at reliability with code, but we're regressing on that metric rather quickly. And the comment above deeply misunderstands both manufacturing and software.
The repetitive piece - the piece that needs that process-control - is moving from shipping code to its creation.
That's new. All analogies to 2022 software engineering need to be thrown out.
Software construction is now more like an industrial process, with the artisanal work shifting towards both
1. aligning what's built with what's needed (talking to customers!)
as well as
2. implementing and tuning your software construction process (ie. hi - we're Anthropic. We're burying ourselves in a mountain of code but it's okay because the fix is a second mountain of code, try some tokens the first hit's free!)
Back to my original point...
Writing code is now on the production line; making quality output demands industrial grade/scale verification (some involving humans of course).. which is a fascinating engineering challenge, and it's brand new!
Sure, I'd you want the exact same code created every time, I can automate that with cp. It's perfectly reliable, and checksums can catch the few times it fails.
Verification means nothing for industrial processes when the desired outcome is a moving target, so the first part of an industrial process is to develop a fixed target and aggressively remove variation.
Moving the target around is a research problem, before you feed the result into an industrial process.
Software solved industrialization so well we even forgot that we go from research to industrial with `npm` deployments or whatever your package manager of choice is.
You should go learn about test driven development and test first development in general.
It absolutely can be industrialized: the agent writes tests that conform to your requirements; the harness makes sure those tests pass before work can stop.
Have a look at https://codeleash.dev/docs/tdd-guard for my (dated but still interesting) take on this. Claude Code hooks are really useful to do the enforcement.
It's open source so no need to read my crappy generated docs - you can point your agent at the repo and ask it questions.
Those machines are not doing design work. They automate production of designs that had long development and testing lifecycles. Code is design. The compiler is the automated production process.
"Anthropic told investors that it will profit on a measure called adjusted operating income.
"According to the report, adjusted operating income excludes stock-based compensation, which is not unusual for technology companies. However, the report also quotes its gross margins - above 80% - exclude revenue shared with distribution partners, as well as the cost of training its models."
I wonder how they account for free credits given in this spend. We got given US$5k free API credits from them a couple months ago and even accounting for our Claude Code subscription spend, I think we're still ahead overall - they haven't made any money from us! And they just gave us another few hundred bucks a month of API credit as part of our subscription.
Antropic's revenue is up [1], [2], [3] and if you calculate adjusted operating income (by excluding most expenses like training costs) then by that measure it is profitable. However, if you factor in all the numbers it is operating at a loss.
Source? Only reports I’ve seen was they were profitable for one month in last quarter due to xAI discount of renting out colossus . Be interested to learn otherwise.
After investments. Investments they can cut if they need to. Their daily operation is very profitable, they just decided to use that to bring cash in and go on the crazy growing trajectory, instead of taking it slow and steady. But if AI progress suddenly stops, they can just scale investments back and became profitable immediately. Calling them unprofitable is highly misleading. It's not like they are an early venture company still trying to figure out their product-market fit, with totally inflated numbers based merely on a theoretical number of potential customers (while having close to zero paying ones).
I'm not arguing that their product is not valuable or that it doesn't have customers.
I'm stating that their current business model, prices, and behavior they are trying to instill is not profitable. For example, token maxing every code change is great for Anthropic and terrible advise for Anthropic customers.
The moment they stop training, they become irrelevant. The moment they become irrelevant their valuation crashes down to normal valuation ranges for a company with their revenue and future growth
They aren’t claiming that the doing less by itself changes the rate in this sentence. They are saying that the rate changes when you slow down, probably due to having more time to focus, which in my opinion is likely.
From the article: "merge with at most one human in the loop, who can be the author". Curious to see how this handles enterprise compliance requirements and insider threats?
- How was this approved for SOC2 compliance? Our SOC2 reviewer explicitly mentioned that all code needs to reviewed by at least one other human before it goes to production. Claude Code says they are SOC2 compliant (https://code.claude.com/docs/en/security).
- This assumes all engineers will not do anything malicious. What is preventing a Claude engineer from spying on their ex's or some other companies Claude searches by pushing malicious code? Recently an OpenAI engineer was caught planting a recording device at their own office.
I’m sure it’s not compliant. We have the same requirement for soc2. However the audits are pretty light and a company big enough can afford to have an auditor who looks past those things.
i think SOC2 is about audibility more than anything - my guess - and i’m not an internal auditor or anything - is soc2 needs auditing for human changes, but a machine is just a “system behaving on its own”
How is the ability of a human to understand the code via hands-on familiarity not a direct dependency of their ability to do manual code review? I don't get it. Having claude write an explainer (something I do a lot!) doesn't get you to the point where you can confidently assess a complex diff. And you've funneled your priors on what the diff says and does through the same system that already reviewed and approved it. This article isn't bad but it doesn't give enough weight to the human context-building part of the problem.
I think there is no magic trick here. The answer is simply that humans will no longer understand the code.
Although, following Anthropic’s ethos and taking this to its conclusion, I’m not even sure why they need human reviewers at all. What, they bring in a human to look at a diff of an “important” section of code that was written by agents and has changed a dozen times since last week through 12 PRs of churn? Obviously they don’t know what’s going on. So what’s a human to do? Do what the other approvers (if any) did: fire up a Claude to command it “verify this change has no regression; check carefully for edge cases”, then come back after a half hour of High Thinking to see “Change is clean (with some non-blocking findings below, in order of significance)” and click Approve. Not sure what the purpose of the human is in such a process.
Yes it used to be said, the code is not the asset, the people who know what the code does and how to change it are the asset.
Now we’re just going to outsource our capacity to design and review because it’s faster that way, as if the thing that was wrong with the software industry was that releases weren’t being pushed out often enough.
Before all came to this, I've been promoting reviews at $dayjob not as gatekeeping but as method to exchange information and build more team spirit. We are an internal service team working by "projects", which makes us a very disjointed team with barely any communication. (Part of it is also German work culture.)
And now this. Suffice to say, my efforts did not bear fruit. I feel more isolated than ever before.
All good ideas, I have been doing something similar for half a year. The most important thing is to reduce bugs and ensure the code does what you wanted. Writing code tests, logging, interacting with the running app are essential. For intent realignment I like to have the agent re-read all user messages posted in that project. It only takes a few minutes even on a 10,000 message project. Then use this fresh derivation of user intent to check code and tests. For long spans of work I prefer to generate a todo list which remains as a document in the project. Each closed gate tracks feedback from the agent on what happened. This task file is also reviewed twice - once before implementation and once after. So the formula is - review agents - tests - user message logs.
I'm somewhat surprised they would say Claude is writing 90% of their code at this point. Did they get tired of reporting new milestones, or has there been a plateau?
It looks like they are specifically optimizing for incidents/reverts and shipping faster
> The goal hasn't changed: less human time per change, and no more reverts than before.
Is this Goodhart's law in action? Less reverts and less incidents was measured and became the target. Is this then still a good measure? I guess Revenue is a more direct measure, but we don't have access to that one.
From my perspective as a power user, the product has improved substantially over the past six months. It has gained new features that are actually useful, lots of UI/UX/perf improvements, and bug fixes as well.
New features have actually been remarkable. For me they are almost always in one of two categories: things I've always wanted (but never actually bothered communicating to the devs), and things I didn't know I needed but did. The product team is clearly very tuned in.
The Workflow engine, the Mod system. I've tried to explore other coding harnesses and nothing is as powerful as claude code (barring investing a lot of effort into custom plugins for Pi or something). Other harnesses have workflow engines, but none are as powerful as Claude Codes.
I really like the the new ability for agents to start and manage other top-level agents. Makes it super easy to have one agent as the orchestrator for a feature and have it launch new sessions, answer questions from those agents, review their work and build merge trains. This was possible before, but required a fair amount of manual work. Now the experience feels first-class and it's really nice.
They are in for a surprise when they become publicly traded and have auditors requiring actual humans signing off on any code that touches financial data.
Here I am still unable to stop myself from doing line-by-line code review. I notice too many things (redundant code, duplicate code, overly complex architecture, etc.) by doing it that I feel it would be careless not to.
My advice is to start a totally vibe coded side project. You will never gain confidence until you experiment, find the failure cases, learn to deal with them, etc.
Choose something you want to do but don't really care that much about. Be more ambitious than you think you could be, have a high bar on the outcome but purposefully choose to completely ignore the actual code. Make it write tests, dashboards, monitoring - whatever you need other than code to feel confident.
Now is the cheapest time it will ever be to experiment.
I did this, but now I have a useful piece of software that I have zero clue regarding how it works. My only chance of "maintaining" it is giving it to another LLM to explain.
I am happy that I have a useful tool, but it simply isn't maintainable without a machine or me investing much time to understand it.
I don't see how the former is a sensible future for humanity. I am happy to say I won't be releasing this software I've created (I will find it useful myself), but deferring all understanding to a black box seems a poor future for brain activity.
One thing I do is ask the LLM to write me a walkthrough of the code. LLMs are pretty good at generating documentation like HTML slideshows or whatever else.
I mean, I've had useful software that I don't understand how it works where I am stuck hoping some small company continues to maintain it. And I've had some of those companies go out of business and leave me with useless software that I eventually have to replace once it no longer works on my machine.
Having the code that I can dump into an LLM to understand myself or have it alter it seems better than that.
I've been working on a large (~10MLOC) mostly vibe-coded project for months now. It's actually quite frustrating:
* The agent will fix a bug when I tell it to, but only in exactly one place. I'll notice that another place has that bug later and have to redo it. It's the sort of thing that I would know to check for if I were fixing it myself. Yes, I can ask it to look elsewhere for similar bugs, but even that's not super reliable.
* The user-facing text it generates is just as awkward as ever. I have to rewrite all of it, and sometimes it slips through. It's also hard to write text when you don't know what the code is actually doing.
* It doesn't have a good model of what kind of information is relevant to humans. It's just as likely to produce a paragraph of random jargon as a clear explanation of what's actually going on. It shows excessive detail for things that don't matter, and not enough detail for things that do. For example, it doesn't understand that $0 can logically mean "null" in terms of how humans think about things (e.g. "This item is free" vs. "This item is $0").
I kind of wish I hadn't gone down this path but now it's far too late, I can't really manually maintain this amount of slop. I'm quite sad if this is going to be the end state of this profession.
I have done this. It's exactly the reason I went back to doing code review. I don't look at the code with a fine toothed comb, but it does help me catch bad patterns early.
I guess the question is: did the project you worked on fail? As in, you couldn't achieve the outcome you were trying to achieve? Or at some point did you sneak a look at the code and recoil in revulsion? What was the failure state?
Was this with the latest models (Fable/Opus 5.5 or equivalent) or an older model in a 3rd party harness?
Are you in the camp of: "if at first you don't succeed, never ever try again".
Not parent, but project I was involved in was shipped very quickly, had many bugs, and burned a lot of customer trust.
This was a few months ago, so obviously not with the latest models, but:
1. That argument is evergreen. You will always be able to make it, and it doesn’t really add much. It is impossible to refute.
2. Gone back to - when writing code - having a strict mental modal and good control over the codebase. A mix of hand coding (to establish the mental modal) and prompting (for boilerplate). It’s not quite as fast, but the quality is much higher when you mix in using LLMs in quality enhancing capacities (review, debug, etc). This is with the latest model. Hands off still atrophies over time (even the boilerplate), and in some cases is very inefficient (as in wasting my time, nothing to be said about tokens).
> Are you in the camp of: "if at first you don't succeed, never ever try again".
This is great in some domains, but “don’t make the same mistake twice” is much more important in mine. I don’t have the luxury to ship low quality, especially when LLMs make even higher quality so much easier.
I think people are moving the goal post here. My post suggested starting a vibe coded side project, one where failure is low stakes and has no consequences. You are talking about shipping a production system to customers.
If you cannot tolerate failure then don't experiment. Don't mix the two.
I've done this, and always get subpar results. I cant get the LLM to make the minor tweaks I want to see, and now I have no mental model of the code to make the fix myself. Sometimes they arent minor things.
Ridiculous suggestion. If someone already takes issue with ai generated code (as well they should, it often is _still_ pretty bad), getting them to write more of it is useless.
Additionally, do you really believe that 'Now is the cheapest time it will ever be to experiment'? Why so? Because of token prices? Not a good omen if so.
I'm not sure what is ridiculous about suggesting people take a stance of openness and curiousness when confronted with new situations. I genuinely don't understand what people have to lose by building some throw-away experiments without reading the code.
I do believe the subsidies offered on Pro accounts driven by the current competition make access to the best coding models and harnesses as cheap as they will ever be.
>I do believe the subsidies offered on Pro accounts driven by the current competition make access to the best coding models and harnesses as cheap as they will ever be.
God I hope you're right. Maybe then this era of vibe slopping out photoshop clones that'll be abandoned within a week (while advertising them all over social media) will come to a close.
Issue: I am still unable to stop myself from doing line-by-line code review
Response: Try experimenting in a low stakes environment
From Google: "Exposure therapy is a psychological treatment designed to help people confront their fears, break cycles of avoidance, and reduce anxiety."
I think that depends on where you work. If your company demands that all code is reviewed then anyone who doesn't do that should be spoken to.
However, I suspect there are teams that are making different choices. In the most obvious example, I can do whatever I want on my personal projects (I hope you agree). And if I started a business and hire in programmers, I have some expectation on defining the culture, making hiring decisions based on my preferences, etc.
Whether or not such teams succeed will be a matter of fact, which I do not believe has been decided at this time.
I hate to say this, but you're now living in a world where the value of artisanal code is rapidly approaching zero. If you still cling to beautifully written code as your core qualification, you're having a rough future ahead of you.
And I say this with the bitterness of someone having lost a big part of what made me enjoy this line of work.
Because one may learn that those fears are overblown. Or they may learn that they cannot build any kind of software without reviewing every line of code.
The person is saying "I am still unable to stop" and that is most likely because they feel a sense of responsibility. If they choose a project where there is no responsibility, where it literally doesn't matter if the project fails, they may feel able to stop. And if they manage to make that project work, if they can succeed, then they will build up confidence.
It seems better to be able to work both on projects where they review all code as well as projects where they do not review any code. In fact, keeping their job in the long term may (or may not) require having both skills.
I guess they’re just saying “you’re holding it wrong”
Last time I chose to vibe code, I did it in a language I didn’t know so that I wasn’t tempted to try and comprehend the choices the robot made. I ended up with the application I wanted.
Caveat: it was code I would never have to touch again, I ran a script to generate the 140,000 static html pages that the rails app provided and then just uploaded them all to R2 buckets (million reads per month on free tier)
I think the intended takeaway is to learn to solve those problems without line by line review. Start by being hands off on a low stakes project, and see how the model fails. Then, come up with, and try, solutions that aren't you doing archeology.
This somehow assumes that the person above doesn't already do all the magic the people say work 100% of the time, and finds these issues. How does running into literally the same issues in a low-stakes project help?
I think it's a good assumption. Their comment implies they aren't harnessing the coding agent very strongly given the failure modes they mention.
Do you think it's better to experiment with high stakes projects? Do you think it'll be easier to avoid defaulting to line by line auditing of a project that is high stakes?
if you take yourself out of the loop it will hurt your skills and career. Engineering blogs used to have some best practices you emulate, but the big ai companies you cannot mimic without their infrastructure
Is it responsible to move slowly when your competitors are moving quickly? Depends on how your competitors perform. If they perform well, then your "responsibility" suddenly becomes your extinction.
No one cares about good code. It’s a means to an end. Your code doesn’t need to look pretty or be well documented or any of that stuff anymore because people are not looking at it. The only thing people care about is performant and error free code.
AI tends to add new wheels for a new feature in a large codebase, because finding, understanding, and generalizing the existing code to integrate a new feature will likely break existing features, cost more effort (and token) to implement and test. Human reviewers also tend to like these code since they are clean, easy to review, and get the job done, especially when they are not familiar with existing codebase.
The issue is that wheels like these accumulate in the codebase, and they inevitably entangle. When an upstream specification changes, when a new feature that works with many existing features are introduced, all these wheels need to be revisited, and in many cases they are not because the new test cases do not cover all the paths. The code base got larger and more complex, and contains more bugs than the unit tests can cover, especially when unit tests are often "bent" to fit the wrong behavior unnoticed.
The worst part about this is that if you aren't watching it, the AI will "solve" this with a thousand assert_3_plus_5_equals_8() test cases that just further clutter the workspace and the context window. Churn baby churn.
The models are pretty good at finding code duplication and eliminating it though, and it's usually part of my AI review process at this point
If you let it go buck wild on a massive change, there are those times where it sort of paints itself into a corner but rewrites are cheap (for now), so it's just not that big of a deal, imho
I’ve had multiple occurrences (with the latest models) of AI reading a poorly named db column and assuming it was one thing when it was another that resulted in a problem. It does not know when to stop or when to continue with determining what is true.
Literally 50 years of engineering history and culture cares about whether code is good. There are libraries worth of books about it, not as a means to an app but for its own beauty. Just read “The Elements of Programming Style”, for example.
Do what you want, write apps with AI and don’t review code, it doesn’t impact me. The claim that no one cares about good code is demonstrably false, and that’s all I’m objecting to.
if it was any other industry like medicine, food, or construction etc, could they get away with just shipping products where they have no idea whats in it?
where things like "it tastes good to me" or "cement didn't crumble when i hit it with a hammer" is good enough to ship we'd be inundated with lawsuits and customer claims... its a total joke where the software industry is headed
The way you phrase it makes me realize its almost like a GPL contamination. Eg, once you pull in GPL code, your project is infected and can never use a BSD (or other GPL incompatible) license.
Similarly, once you pull in terrible AI slop code, the project becomes unmaintainable by humans and you need to maintain it with AI.
There’s a little irony about the example you picked of the GPL, just because in the US the output of an LLM cannot be covered under copyright. The GPL works as an assignment of rights under copyright.
Code that isn't duplicated in 15 places, that doesn't read the whole database to fetch just 1 record, that doesn't do 15 database reads where one would suffice, that uses an optimized algorithm instead of an exponential one recreated badly from scratch... all that is also good code, and is just as important in LLM-slop era.
No code is error free. All code will need a fix of some kind eventually. When it does, it's better that the internals are not a total mess. In the same way that you don't want the insides of your car to be total alien nonsense either.
I think this is the future. Im more bullish than bearish. The only issue is that eventually human verification can no longer be possible and we will have to live under AI-verification assumptions....
What I’m more worried about is feature identification and how to monitor them. How can you get the product to the next level?
Honestly I’m not worried about code anymore in my codebase I’ve not had bug problems for a while and incidents get fixed automatically.
I monitor new features with an agent who checks recordings.
Most actually good features need still 20-30 iterations to design the userflow to actually be interactive, but then there is also another challenge to translate usage patterns into features and how to present functionality with the least amount of friction as possible.
How to efficiently run monitors without wasting your token spends. A best 'monitor' best practice would be nice.
Still this article was very helpful and gave me a lot of new ideas.
Genuine question I’ve been wanting to ask folks working on different types of things: how are your teams doing agent-assisted code reviews? Not automated reviews, but what techniques and tools are humans using to keep up with the larger volume of PRs that seem to trend larger and larger?
I think a lot of people like myself still want to feel in the loop but I don’t have the right tools to stay on top of the (generally good, productive, and still thoughtful) contributions from peers anymore.
Using agents for a first pass is a good way to speed up the review cycle. This way contributors can get near immediate feedback before a human needs to weigh in. This doesn’t replace the human review, but augments it. Each time your agent misses something, update your code review guidelines.
what are people using for automated code review these days? the claude code review product originally mentioned it cost "$15-25 per review" (which is absurd), but it seems like they've removed that from their docs with the launch yesterday. we've looked at various products in the space (coderabbit, greptile, etc.) but have gripes with each.
honestly i'm thinking it might be easier (and cheaper) to launch our own agent from GH webhooks and trigger `/code-review`..
We've had great success with a custom gh action on a self hosted runner. It sets up codex and all the same stuff a dev has on their machine:
So dbs, seed files, etc, and then just a review prompt and some skills and a few scripts.
It triggers on a new published PR or if tagged @bot-name. It's way better than the other services we've tried (which was admittedly ages ago) because you can get it to work exactly how you want it and it's dead simple.
With Codex, for the time being at least, you can use a subscription, so it's basically free if you're already paying for one. But on API pricing it's still not too bad as it doesn't produce that much output.
I set this all up in about half a day. Codex does 99% of the heavy lifting.
this: gh actions using an opencode harness (so we can use cheap models through OpenCode Zen). With the right model, and the right, scoped access, it's just a prompt away.
bugbot was costing us $1k per month for a small team. EVEN IF we decide to go with a paid or say something like GPT Sol for our custom harness it'll cost us maybe $100. We've already implemented all of bugbot's interesting features like cross-PR memory.
the cool thing is we can leverage this for things that bugbot cannot do. Just today I added a security-reviewer action reusing almost all of the code that we wrote for the code reviewer.
I've set up a claude code on a vm with a bunch of skills that can use git and the Azure Dev Ops MCP via a service user, the regular build agent pops a record in a db that serves as a queue/review and a bash script picks things up and fires off claude code in headless mode.
It's pretty effective and lets you do nice stuff like compare the code to the tickets, reply to comments and stuff like that.
Using a different lab's model to review the code to the one that wrote it. Claude model plans/codes, OpenAI model reviews, or vice versa, or DeepSeek or GLM or whatever.
I do it this way too. I think having the same lab's model review the code in a fresh session also helps, so I use both: I generally start a new Claude Code and a Codex and have them review the code with a focus on 3 things: 1. bugs, 2. cut corners, 3. code style.
This generally works for me. I think a solid spec is the key for the models to review against. I ask them to follow the spec, and if the spec doesn't say, follow first principles.
Also, if there is a user experience change, they need to notify me.
in a vacuum, sure. the average pull request will have multiple commits once opened (i.e. to address feedback or perhaps it was opened too early), each triggering a review. it's very easy to rack up $100+ per pr (and an eng may open a dozen prs in a single day)
I am happy people are testing these things out. I am also just going to wait a bit longer before really adopting things like this, I don't really trust it yet. A year ago this looked very different, and next year my guess is things will look different as well.
I'm astounded that I kinda ended up building the same agentic workflow they are talking about in this article and distributed it amongst the org (eng and product/designers).
On the other hand, I would have expected Anthropic to have a drastically more elegant SDLC.
The UI has improved substantially over the past six months. New capabilities are prominently featured and less-used ones are de-emphasized or removed. Every product goes through this type of evolution. The difference with Claude Code is that at the speed they iterate, it happens a lot faster and is therefore more noticeable.
customers don't like big changes to workflows they get used to; adding a new feature or a new button or menu item isn't a big deal but rearranging things so often start to feel random and shows little respect for the customer (or signs of any cohesive product design)
I'd be more interested to read on what 10% of the code is still getting written by humans and how it doesn't become a bottleneck at the pace they described.
Important/hard things are usually bottlenecks, its a good thing and usually by design. People should have more self imposed bottlenecks.
What are llm freaks in such a hurry for? Its like everyone is manically racing towards a finish line without realizing the finish line is you out of a job. Its like everyone is excited to be completely obsolete and poor.
My interest is mostly from the management perspective -- in 2025 I have seen a prime example when a heavily vibe-coding team chugged out half-baked products that other teams later had to deal with and this was not pretty and created all sorts of interpersonal issues; the same I fear may be happening here as well.
I'm looking at Claude Code and they chug releases daily and sometimes more often then this, if someone has to stop and code a critical path by hand or smth I'm wondering how this is handled.
--
On a separate topic, the "llm freaks" -- I know you intended this as an insult, but I'm actually proud to be called a freak again, and I thought this is the place where freaks are discussing things.
I didnt really mean it towards you, I meant it more generally but I can see how it came off that way.
While Im rooting for Anthropic to collapse under the weight of slop and their own contradictions. There's a leaked interview for Anthropic from earlier this year (I think) for their inference team. The interviewer seemed very competent and the quality of the questions were good.
This article is more likely a marketing gimmick in service of shelling shovels (for devs to dig their own grave) "hey look guys we automate everything with claude, you can too, just give us all your money!"
I built a library to have a way to check Claude Code's work against API docs, and asked it to write many tests on top, but even then I found Claude Code would paraphrase and drift from the docs. I also found that it passed its own smoke tests but failed when testing against the real APIs, so reviewing Claude Code's code is tricky.
I used to do it but now I only review code organization and structure.
Mostly I define the product behavior I want and discuss the technical implementation beforehand.
Once that's in a spec I let it build in my durable implementation engine. Automated verification, review rounds from different agent types across different models, issues triaged into meaningful clusters and flagged for false positives before finally being fixed.
That loops until the code is up to spec.
There is no special secret, LLMs need repetition and multiple passes.
They'll one shot bigger and bigger things in their default harnesses but they'll be filled with issues no matter what.
It makes sense for Anthropic to pitch "more code = more better" but as anyone who's been in this game long enough knows, "less is more" often enough. And good luck coercing an AI coding agent to solve a problem by removing things.
Personally, I find Claude's degenerate urge to just spew uncontrollable amounts of code quite counterproductive. Yes I get it, you get paid by the token. I'd like to see more reasoning, more understanding, and less generating.
Truth is, we are still very very far from actual AI software engineering. Some people are working on useful tooling, and (predictably, like anything good and useful) it is taking a long time. LLMs are great, and hopefully they will help us accelerate building the tools that bring this profession to the next level.
30k PRs in a month (or whatever) is not the brag they make it out to be.
If their only metrics are revert/incident rate then they're missing the long-term, cumulative impact of compounding complexity due to poor architectural decisions.
I snorted laughing at this. "We are experiencing tremendous growth" and it's just a codebase with 1,000 agents endlessly adding markdown documentation for every 1 line change.
I believe their peak month was >13k issues opened, but worth bearing in mind: "This issue has been automatically locked since it was closed and has not had any activity for 7 days.".
I just learned that manually invoked skills MUST be the _first_ thing in the prompt, e.g. you can't have `disable-model-invocation: true` and say "Hi, /grill-me on X".
Worse, everything after the /grill-me is injected a second time into the prompt as a <command-args> so your prompt is literally added to the context TWICE. (first time in the original user prompt.)
I swear Anthropic is intentionally bloating your context so you burn tokens on useless bullshit like this.
To be clear: you can certainly do "Hi, /grill-me on X" anytime you want. You can even write "hi, /grill-me and use /search-docs", even though the /search-docs skill absolutely doesn't work. But the Claude Code _harness_ won't autoload the skill into context, so claude-the-model has to go "hmm, where can I find that skill, when I call the Skill tool with that name it no worky, so now I have to search the filesystem for that skill...".
The <command-args> I know because I just audited my raw Claude transcripts via ccglass. You can verify by just asking Haiku "hey, /grill-me does this skill autoload for you?" and then "/grill-me hey, how many times does this text appear in your context?"
You can trivially doublecheck me. The above prompts won't even eat 1% of a 5 hr on 20$ plan.
I was running into something similar where I had it working on a single Python file. Maybe part of the issue is that I had memory turned off, and the conversation might have spilled into two conversations. I wasn't using version control either, but since it was a single file with about 1000 lines of code, it didn't seem too hard to eventually get all of my features back.
There was a post some months back called "Lines of Code Got a Better Publicist" or something along those lines. This is just more of that. I am not impressed by 30 thousand "changes" per day in the slightest without significant added context. When I logged in and prompted Claude today, do I feel like I got +600,000 change value from when I logged in a month ago?
lol right? So youre telling me they shipping 150,000 features this week? Or, maybe, they whack-a-moled 149,995 perf or bug regressions and shipping 5 new things. :shrug:
It’s only about 10 per engineer. I’m not at Anthropic and before 2026 I was regularly hitting 1400 commits / year on my GitHub. If agents made me 2x faster (they have, review is still slow; I currently have 40 PRs open) that’d be the same rate.
Right, but why do things this fast to begin with, especially for a product that's already publicly available?
I don't want to use a product that shifts under me as I use it several times a day. I want a product that actually moves slowly and works very reliably.
As an end user: why would I care about how many hundreds of releases a month are being pushed if maybe 1% of them have features I care about?
We have to test many different api's and models as we have over 600+ in our harness now, Sonnet 5.5 via cloud code provides the highest quality results.
Over the last few years we have added mcp tools, memory graph, md files and skills, automated everything to the point that on desktop the system runs itself, besides proceed/approved/explain.
We have 100's of PRs/jobs/projects and Claude builds down the list automated and only asks a few questions point form if needed.
I am not sure what it is you are saying. What I am saying: I would have supposed that Claude writes 99.x% of Claude's Code code, and that ~no code at Anthropic is written by humans anymore.
I agree finding out who is writing that 10% of the code in the AI company is important. Maybe time to identity them and put them on PIP since they are slowing down everyone else and giving a bad name to the AI company
No. 90% sounds like a low number. That implies something else is writing 10% of the code? People? Other models? What on earth is writing that other 10%?
No wonder most people at Anthropic believe our jobs are in jeopardy, they've already made a system wherein the machines are treating them like pets.
I really don't get the tendency here from Anthropic to describe what's most likely churn? A high pressure amount of changes/PRs a day on a TUI to wrap a while loop? I'm not going to be a reductionist here so I'm going to cut it off there, I'm positive that the complexity of handling multiple terminal emulators is something that needs some handling (even if doing their own rendering engine was a self made problem). Yes I've vibe coded my own harness from scratch without writing any of it, I still know how it works.
I'm also not sure that they addressed point 2 in any actual capacity? They say "the system verifies itself" which is cool and is something that automated testing has done for decades, but they eschew the mental model of how the system works as a whole and prefer instead to have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
N.B. Changed "1000 PRs" to "A high pressure amount" to be vague since the article is vague here on what CC undergoes.
> have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
This is the crux. Take pause, because hidden here is a sea-change in our fundamental trust model of software...
There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug. By design, the system is too large for that, and everything hinges on Claude itself being able to find and understand the right slice of the system in any situation. As long as that very new invariant holds, everything will be all right. If it fails catastrophically, there is no recovery. You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.
The new trust model is: We trust Claude to always understand the system enough to debug anything or make any reasonable change we could want. The risk might be worth the reward. But it's worth explicitly laying out the deal we're making.
Tbf, it's worth pointing out that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
"Tbf, it's worth pointing that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
The real difference as I see it is that in the human maintained old style system, the human would take things slowly and ensure they understand any changes. Claude let loose will just say "yes, I know how to do this" and bork the system, because it doesn't have a concept of humbleness in the face of complexity!
Not only that, when you have a human designed and implemented system you know it's at least coherent enough that someone understood it at some point well enough to take responsibility for it. No such assurance with the llms.
I'm guessing you've only ever worked at startups?
Large company codebases are very much not like that. I've worked on systems (e.g., banking) that have accreted still-running code for as much as 60 years. Nobody understands how it works. Nobody ever understood how the whole worked, and many of the parts were never actually intended to work together.
But is this really a problem? If you have testing and you have version control via Git, you can roll back changes and investigate them.
I also don't even know if it's really true. Sure, maybe it's not coherent enough that a person understands it, but if an LLM wrote it, you can be ~99% certain (in my experience) that that LLM will understand it, and usually in less than the time it took to write it. And unlike humans, the LLMs don't die and take their knowledge with them, or change companies, or get ornery when you ask them to explain.
Also, in my experience, Claude doesn't really do what the parent is saying and dive head-first into complex problems and bork your code. It is fairly humble in the face of complexity. It constantly recommends breaking down larger problems into smaller ones that are more easy to grok.
But most importantly, anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful. Especially when they're writing code that's important. Unless you're an AI researcher pushing the bounds of what LLMs can do, which let's face it, the vast majority of programmers are not, it's almost always the case that you'll have your LLM writing intelligible code that you can understand and have it explain easily enough.
> can roll back changes and investigate them
Sure, you can do that, but irreparable harm might have been already caused. Sure, that’s unlikely when your product is an LLM wrapper. But in something like a database, it could be catastrophic.
I mean it could be, sure. But I think people are generally speaking more careful with higher stakes code than they are with lower stakes code. I know a lot of people who are YOLO'ing it with LLMs, but who ask for more understanding, more reviews, more tests, etc., in trickier areas. Which is only natural. Just like people are more careful when carrying a baby than when carrying a pillow.
(Yes, there will be individual exceptions to this, just like with everything else in life.)
> anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful.
Is this a definitional argument? As in, if you’re not thoughtful, you must not be a decently skilled engineer? Because I see no reason to believe that people are going to be thoughtful about LLM-generated code. Just the opposite. It’s incredibly tempting to LGTM it.
> I see no reason to believe that people are going to be thoughtful about LLM-generated code.
I think that's true for low value code. But the more important and crucial and high-stakes code is, the more effort people put into it, generally. Even with LLMs.
I would love to know what the Venn diagram of people who complain about people not understanding their code, to people who can't describe the boot process of their system is.
I am not sure that "someone understood it well enough to take responsibility" means what you think it means. I have definitely come across old systems which have had addons bolted on to them in so many places that the overall behavior of the system is unpredictable and only mostly works.
Someone understanding something and the codebase being coherent are different things. I can only think that you have been lucky with your career placements if you are making that statement.
What you are saying is leaning on 2 premises that seem to not hold water 1. Code written by people is always more readable and maintainable than LLM code 2. Code written by an LLM is always less readable and maintainable
Can you really make those statements in all intellectual honesty?
I'd like that to be true, but many software service startups were stitching together node packages and selling the result long before LLMs became useful.
> you know it's at least coherent enough that someone understood it
30 years working in this industry and I'd say that's the exception rather than the rule. When it comes to large systems.
Yes, [citation needed]
most large software companies are already too large for anyone to have a complete mental model of.
The idea that someone at say, Google, will be able to fit even half of the system architecture in their heads is not realistic. It also hinges on being able to find and understand the right slice of the system in any situation.
But we have all sorts of guardrails to make that easy - i.e logs, traces, etc
One of the fun things that you currently need to babysit Claude, etc, on, is that while they've been tuned to hell now to do more testing and even logging and such, they don't currently do enough log-and-debug-oriented specific passes/revisions to distinguish meaningful logging for meaningful debugging from noise that won't necessarily tell you what you need to know.
And you can ask them to address this. And you can ask them to look for that in all your PRs. Etc etc. And that helps a lot.
But it's not something that can be brute-forced in the same way "this failed a test, let me fix it" is. So without a long-term-memory brain holding all the pain of past incidents and knowing and applying ahead-of-time judgement about what to build, it's a challenge to turn the general guidance into the specifics for any unique piece of software.
They're trying to get there of course. We shall see what's next.
("Human code turns into this too" is also a curious rejoinder from the "don't read the code anymore" crowd because there are entire disciplines and areas of study that have emerged and evolved from the problems of large ad-hoc multi-faceted systems being difficult and risky to deal with.)
I don't think they're the same. AI code isn't just complex because it's huge.. it's complex because it makes no sense
If you have a large human-written system with a little thought put into it you should be able to find the specific pieces that handle the behavior you want and work on that, and if it's good then changing it should have a small blast radius and be covered with regression tests
In contrast, I've fully vibe coded a 15k line tool for myself with Opus and I have no idea how on earth it's architected after reading through it. It has "tests" but I don't trust them. It's just all over the place with no coherent design to it. It works, but it's making me more and more nervous to change it
Right, that's what I alluded to in my last sentence.
It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.
Having worked continuously this decade on one massively complex consumer facing app that interacts heavily with real-world scenarios, I can say that the problems of cohesion existed well before the agentic revolution. Tons of features built by myopic teams simply because they couldn't understand enough of the system (both online and offline, meat-space interactions) to make good design decisions. AI actually helps here by being able to perform deeper reading and analysis than any human can do. But then the risk is that it can also produce way more random non-cohesive features.
So far I kind of feel like it nets out to the same bottleneck being good judgment from decision makers. And you might say, no, leadership will just put more and more pressure to add more complexity just because we can and it makes it appear that we are moving fast. But when it comes to real-world apps, there is a backstop which is that things that are too complex and illegible end up not being fit-for-purpose for human problems.
Claude Code and other general purpose AI tools are not like this, because they are literally trying to optimize it for every single kind of knowledge work with zero constraints on use case or target audience other than chasing the next juiciest one they can with essentially unlimited resource. 99% of software is not like this, and when the dust settles I suspect software won't be anywhere near as different as the doomers or the maxxers are predicting right now.
> But.... That's not knowledge?
Thank you! This is a huge sticking point for me. There is no knowing the code without experiencing the code. If nothing else, you have to read the tests to know they verify what you think they do. Even then, you might know it does what you want, but can’t know it doesn’t do anything it shouldn’t.
There are no shortcuts to true understanding.
> wherein the machines are treating them like pets
I know some people will groan hearing the phrase again, but: "Reverse centaur." A centaur is a synergistic combination of two things, but now those things are in the wrong position: The human is the computer's tool.
Naturally, how "wrong" it is tends to depend on whether you ask the people who do all the work or the people who own all the computer-capital.
My favourite quote at the moment: "May not man himself become a sort of parasite upon the machines? An affectionate machine-tickling aphid?" Samuel Butler, Erewhon, 1872
When you get to the point that you are doing 1000 PRs I have to seriously wonder what even are you building, I hear about this a lot from a number of companies, but their “too many commits / too many PRs” problem never seems to give anyone from the outside the feeling they are innovating at groundbreaking speeds.
I think it's just a different focus on what matters.
There's clearly enormous (human) work that goes in to building and tuning their models.
The software they develop around their model is, relatively, trivial. It's not clear that enormous amounts of (human) energy need to go in to that code.
It seems largely a function of applying human energy towards the things that are marginally most important to generating you money and putting you in front of competitors.
>The software they develop around their model is, relatively, trivial
30,000 changes a day is not my idea of trivial.
I mean a harness around a model is pretty trivial, I’ve also built one (heavily relying on LLMs to build it).
With a few thousand lines of code over the course of a week or so of my limited evening energy, I have agentic loops, ide integration, and a limited selection of custom tools which are all very safe (no unintended changes), along with session management and awareness of git.
It works great, is customizable, and allows integration with any provider.
If they need 30k changes per day, it’s not due to the nature of the problem!
Oh, I mean I can't disagree there. But they've dug themselves into a hole where they're implementing custom ray tracing in the game engine that claude code runs on or whatever.
> implementing custom ray tracing in the game engine that claude code runs on
Haha, I don’t doubt it! Pretty damning overall for long-term maintainability of vibe-coded projects, even as the models are great at a lot of individual tasks
I mean, sure we can assume that (although the Anthropic article from the other day say that claude mostly builds Claude) but this is the Claude Code team which, I again am assuming so I could be wrong, focuses on the TUI/CLI interface to those models, not necessarily the models themselves. That coupled with the fact that this is how a large portion of their clientele interact with their moat, it would be odd to me if they didn't apply significant work towards this thing to keep the hype or productivity train moving
> I really don't get the tendency here from Anthropic to describe what's most likely churn? 1000 "PRs" a day on a TUI to wrap a while loop?
From the article, the 30,000 changes a month are in the monorepo, and Claude Code is in a separate repo. So it’s not the TUI that has 1,000 PRs a day.
I could easily see 1,000 PRs a day on the backend. Big tech was doing more than that pre-AI.
When I worked at Facebook (pre LLM) we were around 50-100k per day. I imagine it's much higher now (quantity, probably not quality).
Fair enough, I did invent the 1000 a day because the article was vague, I'll edit my submission accordingly to also be vague
The Anthropic post the other day mentioned part of what helps Claude Code unlock some of this is the sheer ability to use more tokens than the rest to get something going and efficient.
This is a good answer
This is like one of Google’s infrastructure blog. Google scale things are only relevant and appropriate for them. We had lots of bitter lessons from smaller engineering teams following the process with the justification that “this is how Google does it”.
Similarly, Anthropic can afford to do this because they have virtually unlimited tokens and labs to train models for their specific usecase. The rest of us need code comprehension lest we become dependent on continuous supply of tokens (which is a state these labs badly want to achieve)
So, unless you work at a lab, or have unlimited tokens guaranteed through the life of your company, it would do both the company and you good to continue to read the code and not let these 30k PRs per engineer be the norm.
An excerpt from the article makes my point
> Nobody on the team keeps a complete mental model of Claude Code in their head, and we don't think anyone needs to. When someone wants to understand a part of the codebase, they ask Claude to write an explainer for exactly that slice. It takes minutes and reflects the code as it is today.
Even then do we really have evidence that what they are doing is reasonable or practical for them at all? They are edging on like 3 9s uptime, what leaked of Claude code looked like soup, tons of security issues, they can't build a sandbox to save their company, and last I heard they aren't even profitable without adjusting their numbers for arbitrary things.
Think I'm good on taking advice from companies that aren't making money.
Exactly. What they’re doing, repeatedly, is setting a narrative that this is how software engineering should be. Always be reliant on tokens. Don’t worry about uptime or errors - agents can address it, don’t worry about security - agents can address it, don’t worry about code comprehension - agents will always be there for you.
I saw someone say the other day (in response to another blog post being well written) that Anthropic employees have a rule against using AI for writing/external comms? That seems clearly not the case here, normal people don't write like this:
> Most of the path is automated; a person's attention goes where the risk is
> The eval grows with every incident, and every change to the reviewer is scored against all of it.
100%, although if people are getting 90% of their interaction from a robot, one may start to pick up some vocal tics and colloquialisms from machine world. a kind of robo-pidgin.
I've definitely picked up some weird claudeisms. I even know the em-dash keyboard shortcut now.
Yuuuup. Which I guess makes it even harder to detect what's AI and what's not.
Pangram reports 80% of the text is AI-generated: https://www.pangram.com/history/72309aad-6e3f-4002-bdc5-c062...
It felt weird reading it because every once in awhile there were some well written sentences and coherent paragraphs, then it fell back into Claudisms.
this definitely seemed like something claude generated, i know because i frequently do all my research in chat and then ask it to generate a doc based on what we talked about and when i read what it generates the verbosity, the style it writes in makes me delete the whole thing. I would never subject readers to that.
> That seems clearly not the case here, normal people don't write like this:
I think when you repeatedly expose and immerse yourself to such a degree in reading and writing language, it brushes off on oneself. So I could also absolutely believe someone from Anthropic wrote this.
As a non-native, I see how susceptible I am to absorbing such style. There's no way at all for me to perceive anything wrong with the examples you present
It's a bit more subtle than things like "it's not X, it's Y". And even that isn't "wrong", it's just the way AI talks, and therefore an indicator that the author offloaded their thinking to an agent. I've read so much Claude output that it's easy for me to recognize its voice, the same way I could recognize a friend's voice in writing. The examples I quoted are a weird kind of circuitous sentence structure with a certain shape and rhythm that scream Claude to me. Here is how I might rewrite them in human speak:
> Most of the path is automated; a person's attention goes where the risk is
--> We automate as much as possible, focusing human attention on risky parts
> The eval grows with every incident, and every change to the reviewer is scored against all of it.
--> We run reviewer evals against every incident
But that's just my interpretation and my voice. I can't know for sure what the authors would have said if they took the time to write it themselves.
It feels like it's trying to sell you something in every sentence.
The word “gate” was used.
No human can stay on par with 30k pull requests per day ... they simply don't have control on what's going on and rely on AI itself to do the job.
PS: I really want to know what is the churn rate (and the cost per LOC) here. My gut is saying the cost per LOC is insane, and they are basically rewriting over and over the same code.
30k PRs is such a meaningless figure though, you can't make anything of it. It entirely depends what's living in that repo. Adding localized strings, configs, things that change frequently, can easily move numbers like this by orders of magnitude. How big are the PRs? Maybe there's an auto-bug-fixer agent that runs around the codebase making thousands of single-line fixes per day. Who knows. The number means nothing in isolation. We used to scoff at bad managers for measuring the performance of engineers using lines of code, now we're using the same figures to judge AI. We've become the bad managers.
> Low - Docs, copy, internal-only features, additive changes behind a flag
> None required once automated review and security checks are green
How anybody can let an LLM write docs and not even read them is pure insanity.
It’s writing the docs for other agents obviously
Anthropic has amazing models. But they should really stop commenting on software practices. Most of what they say seem out of touch with what's happening in software shops. A lot of it seems hype driven. And the entire message looks to be written by AI.
It’s all marketing nonsense. Along with the “AI is dangerous” and “we’re all going to lose our jobs” messaging they do, it’s just to sell the services.
At 30k commits a day, is there anything that can meaningfully be called review?
I'm thinking about high volume mass manufacturing as an example here. I like those "How it is made" type YouTube videos. You see a production line that has some kind of QA process where cameras + automated systems do spot checks at high speeds and these crazy actuators are punching out bad examples. These processes often also involve spot checks where some statistical sample is selected and viewed carefully.
I don't think we've gotten to that level of reliable review with code, but I don't see any reason we can't continue moving in that direction.
I really like those videos too, but a very big difference is, in manufacturing QA, the system outputs a very high number of copies of a very small number of possible types, so spot checks work fine from an operations research point of view, as the variance in output has a known upper bound.
For code, I think things go in the opposite direction: the system outputs artifacts that are all different among themselves (even more so now that we no longer distribute software via physical copies. Well, bits are always physical, but I mean we no longer ship floppies/CDs/DVDs).
That said, at least in my experience, code review has never been a direct part of factory-line-type QA. That QA is typically done by treating software as a black box and just verifying that it complies with a spec (often times based on a very loose definition of "spec" ...), and this is something that works equally well regardless of whether a human or a machine wrote the software. The type of QA that is done (among other things) by code reviews is checking whether the software's design is such that it will be economically feasible to maintain and extend it, for example.
The one part the code reviews takes a part of (but is in no way error-free) and that you just can't do with black-box testing (or it is much harder to do) is to look for things like backdoors. As our industry moves more towards "agents write software and tests validate that it works," I think a lot about "Reflections on Trusting Trust" and the new and wide attack surface added by people no longer reading the code shipped on their behalf.
That’s a category error. Software isn’t stamping out the same part again and again; if it were review would be simple. Analogies often mislead.
Of course Anthropic would love you to think you are heading towards that future so we have to bear in mind their livelihood depends on LLMs being the future of work and 30k changes a day being normal.
But why would we want 30k changes a day in a given product, what possible justification could there be?
I understand the skepticism and I can't claim I am right. But some points come to mind.
First, I think we overestimate how different code is from other code. We even joke about CRUD apps and how cookie-cutter they are. A lot of changes in those apps are: add some new features to a data access layer (SQL, ORM, whatever), add new routes, add new UI surface, etc. There is an entire industry around "Design Patterns" and people live by the DRY axioms. Things like Ruby on Rails exists because rebuilding these systems over and over is wasteful. I feel we are just moving further down that same path.
Second, your question of "why would we want" is totally backwards. Progress doesn't follow your desires. Progress doesn't require justification. Sometimes we adapt the environment to our needs, other times we adapt to a changing environment. Which case we are currently in is still undecided and you are free to fight for the future you prefer. But demanding an industry justify itself before it is allowed to change fails in the face of evolution. If it works, we will justify it in hindsight.
The second paragraph hardly makes any sense, was it the output of an LLM?
stillpointlab.com , "Get more from your LLM"
Exactly. 30k changes a day look like brute force. No one wants that except the one charging for generating those changes.
Those manufacturing videos are of someone making the exact same thing over and over. Months and years of testing go into ensuring that there is no change in the system, and continuous monitoring is put in place so that if any changes slip in, it can be undone. The design is set in stone, as it should be, and variations are bugs. There are months to years of cheap prototypes, iteration, design changes, pipeline tuning, and verification to ensure that nothing will need to change for the next million items.
If you want a software analogy to those devices getting churned off the assembly lines, it's the TCP streams coming out of your network card, with crazy checksums punching out bad packets.
We're orders of magnitude better at reliability with code, but we're regressing on that metric rather quickly. And the comment above deeply misunderstands both manufacturing and software.
The repetitive piece - the piece that needs that process-control - is moving from shipping code to its creation.
That's new. All analogies to 2022 software engineering need to be thrown out.
Software construction is now more like an industrial process, with the artisanal work shifting towards both
1. aligning what's built with what's needed (talking to customers!)
as well as
2. implementing and tuning your software construction process (ie. hi - we're Anthropic. We're burying ourselves in a mountain of code but it's okay because the fix is a second mountain of code, try some tokens the first hit's free!)
Back to my original point...
Writing code is now on the production line; making quality output demands industrial grade/scale verification (some involving humans of course).. which is a fascinating engineering challenge, and it's brand new!
Sure, I'd you want the exact same code created every time, I can automate that with cp. It's perfectly reliable, and checksums can catch the few times it fails.
Verification means nothing for industrial processes when the desired outcome is a moving target, so the first part of an industrial process is to develop a fixed target and aggressively remove variation.
Moving the target around is a research problem, before you feed the result into an industrial process.
Software solved industrialization so well we even forgot that we go from research to industrial with `npm` deployments or whatever your package manager of choice is.
You should go learn about test driven development and test first development in general.
It absolutely can be industrialized: the agent writes tests that conform to your requirements; the harness makes sure those tests pass before work can stop.
Have a look at https://codeleash.dev/docs/tdd-guard for my (dated but still interesting) take on this. Claude Code hooks are really useful to do the enforcement.
It's open source so no need to read my crappy generated docs - you can point your agent at the repo and ask it questions.
Those machines are not doing design work. They automate production of designs that had long development and testing lifecycles. Code is design. The compiler is the automated production process.
Honestly systems like this resemble real engineering more than manual human review does
I dont believe it, its FOMO marketing to me.
It's conducting/steering/oversight/supervision, not reviewing.
And so the wildly unprofitable AI company explains how it token maxes it's code and review process.
Authors should really try and view this article from the perspective of someone who has to think about cost to use the models.
Anthropic has been profitable for the past two quarters.
"Anthropic told investors that it will profit on a measure called adjusted operating income.
"According to the report, adjusted operating income excludes stock-based compensation, which is not unusual for technology companies. However, the report also quotes its gross margins - above 80% - exclude revenue shared with distribution partners, as well as the cost of training its models."
From https://www.morningstar.com/news/marketwatch/2026091414/the-...
I wonder how they account for free credits given in this spend. We got given US$5k free API credits from them a couple months ago and even accounting for our Claude Code subscription spend, I think we're still ahead overall - they haven't made any money from us! And they just gave us another few hundred bucks a month of API credit as part of our subscription.
How so? Can you link to the numbers?
Antropic's revenue is up [1], [2], [3] and if you calculate adjusted operating income (by excluding most expenses like training costs) then by that measure it is profitable. However, if you factor in all the numbers it is operating at a loss.
[1] https://finance.yahoo.com/technology/ai/articles/anthropic-v...
[2] https://fortune.com/2026/09/29/anthropic-leaked-ipo-prospect...
[3] https://www.cnbc.com/2026/10/15/anthropics-third-quarter-rev...
Source? Only reports I’ve seen was they were profitable for one month in last quarter due to xAI discount of renting out colossus . Be interested to learn otherwise.
Profitable as in after expenses.
https://youtu.be/T-oXyXwD6sE?si=tauzXVyGmQ6MhI_5
After investments. Investments they can cut if they need to. Their daily operation is very profitable, they just decided to use that to bring cash in and go on the crazy growing trajectory, instead of taking it slow and steady. But if AI progress suddenly stops, they can just scale investments back and became profitable immediately. Calling them unprofitable is highly misleading. It's not like they are an early venture company still trying to figure out their product-market fit, with totally inflated numbers based merely on a theoretical number of potential customers (while having close to zero paying ones).
I'm not arguing that their product is not valuable or that it doesn't have customers.
I'm stating that their current business model, prices, and behavior they are trying to instill is not profitable. For example, token maxing every code change is great for Anthropic and terrible advise for Anthropic customers.
The moment they stop training, they become irrelevant. The moment they become irrelevant their valuation crashes down to normal valuation ranges for a company with their revenue and future growth
That's factually incorrect, and shouldn't be allowed here.
We're talking about being actually profitable, not fake profitable.
“I think that, every time you see the word EBITDA, you should substitute the words "bullshit earnings.” ― Charles T. Munger
And what they're reporting is adjusted even further than EBITDA
Yes when you subtract expenses and capex
> One colleague said slowing down does lower your error rate, but mostly because you're doing less of everything
That’s not how rates work
They aren’t claiming that the doing less by itself changes the rate in this sentence. They are saying that the rate changes when you slow down, probably due to having more time to focus, which in my opinion is likely.
From the article: "merge with at most one human in the loop, who can be the author". Curious to see how this handles enterprise compliance requirements and insider threats?
- How was this approved for SOC2 compliance? Our SOC2 reviewer explicitly mentioned that all code needs to reviewed by at least one other human before it goes to production. Claude Code says they are SOC2 compliant (https://code.claude.com/docs/en/security).
- This assumes all engineers will not do anything malicious. What is preventing a Claude engineer from spying on their ex's or some other companies Claude searches by pushing malicious code? Recently an OpenAI engineer was caught planting a recording device at their own office.
I’m sure it’s not compliant. We have the same requirement for soc2. However the audits are pretty light and a company big enough can afford to have an auditor who looks past those things.
i think SOC2 is about audibility more than anything - my guess - and i’m not an internal auditor or anything - is soc2 needs auditing for human changes, but a machine is just a “system behaving on its own”
How is the ability of a human to understand the code via hands-on familiarity not a direct dependency of their ability to do manual code review? I don't get it. Having claude write an explainer (something I do a lot!) doesn't get you to the point where you can confidently assess a complex diff. And you've funneled your priors on what the diff says and does through the same system that already reviewed and approved it. This article isn't bad but it doesn't give enough weight to the human context-building part of the problem.
I think there is no magic trick here. The answer is simply that humans will no longer understand the code.
Although, following Anthropic’s ethos and taking this to its conclusion, I’m not even sure why they need human reviewers at all. What, they bring in a human to look at a diff of an “important” section of code that was written by agents and has changed a dozen times since last week through 12 PRs of churn? Obviously they don’t know what’s going on. So what’s a human to do? Do what the other approvers (if any) did: fire up a Claude to command it “verify this change has no regression; check carefully for edge cases”, then come back after a half hour of High Thinking to see “Change is clean (with some non-blocking findings below, in order of significance)” and click Approve. Not sure what the purpose of the human is in such a process.
Yes it used to be said, the code is not the asset, the people who know what the code does and how to change it are the asset.
Now we’re just going to outsource our capacity to design and review because it’s faster that way, as if the thing that was wrong with the software industry was that releases weren’t being pushed out often enough.
Before all came to this, I've been promoting reviews at $dayjob not as gatekeeping but as method to exchange information and build more team spirit. We are an internal service team working by "projects", which makes us a very disjointed team with barely any communication. (Part of it is also German work culture.)
And now this. Suffice to say, my efforts did not bear fruit. I feel more isolated than ever before.
All good ideas, I have been doing something similar for half a year. The most important thing is to reduce bugs and ensure the code does what you wanted. Writing code tests, logging, interacting with the running app are essential. For intent realignment I like to have the agent re-read all user messages posted in that project. It only takes a few minutes even on a 10,000 message project. Then use this fresh derivation of user intent to check code and tests. For long spans of work I prefer to generate a todo list which remains as a document in the project. Each closed gate tracks feedback from the agent on what happened. This task file is also reviewed twice - once before implementation and once after. So the formula is - review agents - tests - user message logs.
i am tired of listening about other ppls agent setups. Everyone thinks they've discovered something cool and unique but They are all the same.
Its like telling other ppl about your dreams. Not interesting.
I used to keep a dream journal because I thought it might be interesting to recall all of my dreams.
Eventually I did, but it was hours and hours of effectively random stuff happening that I had to write down or dictate with speech to text.
It was a waste of time and I'm now glad I forget almost all of my dreams. My point is even my own dreams are uninteresting.
I'm not that interested in my own agent setup either - usually the tools become better on their own after a few months and render it all obsolete.
Much like the output of agents themselves
So true.
For real, my eyes completely glaze over, there is nothing interesting in it.
> Security gets its own automated review too. ... You either fix it or close the thread with a short reason, which is how you clear a false positive.
I hope this is only for low-risk false positives. Dismissing a critical finding as a false positive should still involve AppSec.
You could improve revert rates while simultaneously introducing more security vulnerabilities.
And the linked "claude-code-security-review" explicitly states in the README
> This action is not hardened against prompt injection attacks
That's a significant limitation for an AI security review control, no?
Mfw tui needs a billion prs per day.
Opus 5.5 is obviously stellar but come on what are you doing lol
They need to add tamagotchi pets to the TUI so you stay awake while approving commands.
I'm somewhat surprised they would say Claude is writing 90% of their code at this point. Did they get tired of reporting new milestones, or has there been a plateau?
It looks like they are specifically optimizing for incidents/reverts and shipping faster
> The goal hasn't changed: less human time per change, and no more reverts than before.
Is this Goodhart's law in action? Less reverts and less incidents was measured and became the target. Is this then still a good measure? I guess Revenue is a more direct measure, but we don't have access to that one.
From my perspective as a power user, the product has improved substantially over the past six months. It has gained new features that are actually useful, lots of UI/UX/perf improvements, and bug fixes as well.
New features have actually been remarkable. For me they are almost always in one of two categories: things I've always wanted (but never actually bothered communicating to the devs), and things I didn't know I needed but did. The product team is clearly very tuned in.
All in all, I'd say their approach is working.
I'm curious what has improved so much. To me Opus 5.5 is very good but so was 4.6. I try to forget the versions in between...
not even one tangible example?
The Workflow engine, the Mod system. I've tried to explore other coding harnesses and nothing is as powerful as claude code (barring investing a lot of effort into custom plugins for Pi or something). Other harnesses have workflow engines, but none are as powerful as Claude Codes.
And lets not forget the `claude agents` view and the backgrounding system. A++
I really like the the new ability for agents to start and manage other top-level agents. Makes it super easy to have one agent as the orchestrator for a feature and have it launch new sessions, answer questions from those agents, review their work and build merge trains. This was possible before, but required a fair amount of manual work. Now the experience feels first-class and it's really nice.
They are in for a surprise when they become publicly traded and have auditors requiring actual humans signing off on any code that touches financial data.
Here I am still unable to stop myself from doing line-by-line code review. I notice too many things (redundant code, duplicate code, overly complex architecture, etc.) by doing it that I feel it would be careless not to.
My advice is to start a totally vibe coded side project. You will never gain confidence until you experiment, find the failure cases, learn to deal with them, etc.
Choose something you want to do but don't really care that much about. Be more ambitious than you think you could be, have a high bar on the outcome but purposefully choose to completely ignore the actual code. Make it write tests, dashboards, monitoring - whatever you need other than code to feel confident.
Now is the cheapest time it will ever be to experiment.
I did this, but now I have a useful piece of software that I have zero clue regarding how it works. My only chance of "maintaining" it is giving it to another LLM to explain.
I am happy that I have a useful tool, but it simply isn't maintainable without a machine or me investing much time to understand it.
I don't see how the former is a sensible future for humanity. I am happy to say I won't be releasing this software I've created (I will find it useful myself), but deferring all understanding to a black box seems a poor future for brain activity.
One thing I do is ask the LLM to write me a walkthrough of the code. LLMs are pretty good at generating documentation like HTML slideshows or whatever else.
I mean, I've had useful software that I don't understand how it works where I am stuck hoping some small company continues to maintain it. And I've had some of those companies go out of business and leave me with useless software that I eventually have to replace once it no longer works on my machine.
Having the code that I can dump into an LLM to understand myself or have it alter it seems better than that.
Mermaid is a great output format for this type of information. Flowcharts, sequence diagrams, etc.
I've been working on a large (~10MLOC) mostly vibe-coded project for months now. It's actually quite frustrating:
* The agent will fix a bug when I tell it to, but only in exactly one place. I'll notice that another place has that bug later and have to redo it. It's the sort of thing that I would know to check for if I were fixing it myself. Yes, I can ask it to look elsewhere for similar bugs, but even that's not super reliable.
* The user-facing text it generates is just as awkward as ever. I have to rewrite all of it, and sometimes it slips through. It's also hard to write text when you don't know what the code is actually doing.
* It doesn't have a good model of what kind of information is relevant to humans. It's just as likely to produce a paragraph of random jargon as a clear explanation of what's actually going on. It shows excessive detail for things that don't matter, and not enough detail for things that do. For example, it doesn't understand that $0 can logically mean "null" in terms of how humans think about things (e.g. "This item is free" vs. "This item is $0").
I kind of wish I hadn't gone down this path but now it's far too late, I can't really manually maintain this amount of slop. I'm quite sad if this is going to be the end state of this profession.
I have done this. It's exactly the reason I went back to doing code review. I don't look at the code with a fine toothed comb, but it does help me catch bad patterns early.
I guess the question is: did the project you worked on fail? As in, you couldn't achieve the outcome you were trying to achieve? Or at some point did you sneak a look at the code and recoil in revulsion? What was the failure state?
Was this with the latest models (Fable/Opus 5.5 or equivalent) or an older model in a 3rd party harness?
Are you in the camp of: "if at first you don't succeed, never ever try again".
Not parent, but project I was involved in was shipped very quickly, had many bugs, and burned a lot of customer trust.
This was a few months ago, so obviously not with the latest models, but:
1. That argument is evergreen. You will always be able to make it, and it doesn’t really add much. It is impossible to refute.
2. Gone back to - when writing code - having a strict mental modal and good control over the codebase. A mix of hand coding (to establish the mental modal) and prompting (for boilerplate). It’s not quite as fast, but the quality is much higher when you mix in using LLMs in quality enhancing capacities (review, debug, etc). This is with the latest model. Hands off still atrophies over time (even the boilerplate), and in some cases is very inefficient (as in wasting my time, nothing to be said about tokens).
> Are you in the camp of: "if at first you don't succeed, never ever try again".
This is great in some domains, but “don’t make the same mistake twice” is much more important in mine. I don’t have the luxury to ship low quality, especially when LLMs make even higher quality so much easier.
I think people are moving the goal post here. My post suggested starting a vibe coded side project, one where failure is low stakes and has no consequences. You are talking about shipping a production system to customers.
If you cannot tolerate failure then don't experiment. Don't mix the two.
I've done this, and always get subpar results. I cant get the LLM to make the minor tweaks I want to see, and now I have no mental model of the code to make the fix myself. Sometimes they arent minor things.
Ridiculous suggestion. If someone already takes issue with ai generated code (as well they should, it often is _still_ pretty bad), getting them to write more of it is useless.
Additionally, do you really believe that 'Now is the cheapest time it will ever be to experiment'? Why so? Because of token prices? Not a good omen if so.
I'm not sure what is ridiculous about suggesting people take a stance of openness and curiousness when confronted with new situations. I genuinely don't understand what people have to lose by building some throw-away experiments without reading the code.
I do believe the subsidies offered on Pro accounts driven by the current competition make access to the best coding models and harnesses as cheap as they will ever be.
>I do believe the subsidies offered on Pro accounts driven by the current competition make access to the best coding models and harnesses as cheap as they will ever be.
God I hope you're right. Maybe then this era of vibe slopping out photoshop clones that'll be abandoned within a week (while advertising them all over social media) will come to a close.
[flagged]
Please don't do this here.
(You're welcome to make your substantive points thoughtfully, of course.)
https://news.ycombinator.com/newsguidelines.html
Issue: I am still unable to stop myself from doing line-by-line code review
Response: Try experimenting in a low stakes environment
From Google: "Exposure therapy is a psychological treatment designed to help people confront their fears, break cycles of avoidance, and reduce anxiety."
The thing is that's not an issue in the slightest, it doesn't need therapy!!
It's those who don't do it who'd need a talk..!
I think that depends on where you work. If your company demands that all code is reviewed then anyone who doesn't do that should be spoken to.
However, I suspect there are teams that are making different choices. In the most obvious example, I can do whatever I want on my personal projects (I hope you agree). And if I started a business and hire in programmers, I have some expectation on defining the culture, making hiring decisions based on my preferences, etc.
Whether or not such teams succeed will be a matter of fact, which I do not believe has been decided at this time.
I hate to say this, but you're now living in a world where the value of artisanal code is rapidly approaching zero. If you still cling to beautifully written code as your core qualification, you're having a rough future ahead of you.
And I say this with the bitterness of someone having lost a big part of what made me enjoy this line of work.
That's not wrong right now, we'll see in the future.
But I have no interest in working as a vibecoder, there are many other jobs that I'd enjoy more
No. The issue is: "I notice too many things (redundant code, duplicate code, overly complex architecture, etc.) during review"
How does "low-stakes environment" fix that?
Because one may learn that those fears are overblown. Or they may learn that they cannot build any kind of software without reviewing every line of code.
The person is saying "I am still unable to stop" and that is most likely because they feel a sense of responsibility. If they choose a project where there is no responsibility, where it literally doesn't matter if the project fails, they may feel able to stop. And if they manage to make that project work, if they can succeed, then they will build up confidence.
It seems better to be able to work both on projects where they review all code as well as projects where they do not review any code. In fact, keeping their job in the long term may (or may not) require having both skills.
yolo'ing a code complexity generator doesn't make it suddenly stop being a code complexity generator.
I guess they’re just saying “you’re holding it wrong”
Last time I chose to vibe code, I did it in a language I didn’t know so that I wasn’t tempted to try and comprehend the choices the robot made. I ended up with the application I wanted.
Caveat: it was code I would never have to touch again, I ran a script to generate the 140,000 static html pages that the rails app provided and then just uploaded them all to R2 buckets (million reads per month on free tier)
I think the intended takeaway is to learn to solve those problems without line by line review. Start by being hands off on a low stakes project, and see how the model fails. Then, come up with, and try, solutions that aren't you doing archeology.
This somehow assumes that the person above doesn't already do all the magic the people say work 100% of the time, and finds these issues. How does running into literally the same issues in a low-stakes project help?
I think it's a good assumption. Their comment implies they aren't harnessing the coding agent very strongly given the failure modes they mention.
Do you think it's better to experiment with high stakes projects? Do you think it'll be easier to avoid defaulting to line by line auditing of a project that is high stakes?
This is why I like GitHub Copilot's autocompletion, I'm right there guiding and policing the code as it appears.
if you take yourself out of the loop it will hurt your skills and career. Engineering blogs used to have some best practices you emulate, but the big ai companies you cannot mimic without their infrastructure
You're unable to stop being responsible? Tragic!
Is it responsible to move slowly when your competitors are moving quickly? Depends on how your competitors perform. If they perform well, then your "responsibility" suddenly becomes your extinction.
Apple's iPhone Duo seems like it'll be a huge hit.. how many years after the competitors?
They've built an entire strategy around moving slow and doing it better
If they move quickly by shipping crap they're just running towards a wall, the responsible thing is to not follow them.
fomo nonsense. not everyone has to jump off the cliff just because you are.
No one cares about good code. It’s a means to an end. Your code doesn’t need to look pretty or be well documented or any of that stuff anymore because people are not looking at it. The only thing people care about is performant and error free code.
People saying this are telling on themselves that they have no idea what quality code is. It's not about looking pretty or nice variable names.
Code quality matters to AI just as much as it did to people
Do you have any data backing this statement? Or is it just your experience?
Personal experience:
AI tends to add new wheels for a new feature in a large codebase, because finding, understanding, and generalizing the existing code to integrate a new feature will likely break existing features, cost more effort (and token) to implement and test. Human reviewers also tend to like these code since they are clean, easy to review, and get the job done, especially when they are not familiar with existing codebase.
The issue is that wheels like these accumulate in the codebase, and they inevitably entangle. When an upstream specification changes, when a new feature that works with many existing features are introduced, all these wheels need to be revisited, and in many cases they are not because the new test cases do not cover all the paths. The code base got larger and more complex, and contains more bugs than the unit tests can cover, especially when unit tests are often "bent" to fit the wrong behavior unnoticed.
-
Yep.
The worst part about this is that if you aren't watching it, the AI will "solve" this with a thousand assert_3_plus_5_equals_8() test cases that just further clutter the workspace and the context window. Churn baby churn.
I haven't had this issue since like last year with frontier models. What are you using?
All the models do this. I'm literally right now watching Opus 5.5 duplicating code even after telling it to fix duplication.
-
The models are pretty good at finding code duplication and eliminating it though, and it's usually part of my AI review process at this point
If you let it go buck wild on a massive change, there are those times where it sort of paints itself into a corner but rewrites are cheap (for now), so it's just not that big of a deal, imho
-
still waiting for a better result than "a anon-optimizing compiler where progress is impossible": https://www.anthropic.com/engineering/building-c-compiler
Funnily enough, the models learned from GGC and LLVM, yet, cannot produce something half-decent.
I’ve had multiple occurrences (with the latest models) of AI reading a poorly named db column and assuming it was one thing when it was another that resulted in a problem. It does not know when to stop or when to continue with determining what is true.
Literally 50 years of engineering history and culture cares about whether code is good. There are libraries worth of books about it, not as a means to an app but for its own beauty. Just read “The Elements of Programming Style”, for example.
Do what you want, write apps with AI and don’t review code, it doesn’t impact me. The claim that no one cares about good code is demonstrably false, and that’s all I’m objecting to.
> it doesn’t impact me
so long as his software isn't used for anything that can affect you
Not true, LLM’s will struggle with badly organized and designed large codebases just the same, though likely with less errors and profanities.
The biggest concern is that AI can doxx you and find where you live if your code sucks. If that doesn't scare you, nothing will.
> No one cares about good code. It’s a means to an end.
I do, my coworkers do, my friends do, etc etc.
I think a more generous interpretation of the parent comment would have been: "No one [in business] cares about good code" .
Your end users certainly don't either, heck they'll gladly sit there and scroll through their phone until the 50GB React app loads.
That’s a fair interpretation, thanks for the clarification!
I don't care about the opinions of end users with poor taste as much as I care about the opinions of end users with good taste.
Good code is a means to the end of the code being error (and vulnerability) free, maintainable and performant.
And if people are not looking it, those people don't care about their customers.
where things like "it tastes good to me" or "cement didn't crumble when i hit it with a hammer" is good enough to ship we'd be inundated with lawsuits and customer claims... its a total joke where the software industry is headed
The way you phrase it makes me realize its almost like a GPL contamination. Eg, once you pull in GPL code, your project is infected and can never use a BSD (or other GPL incompatible) license.
Similarly, once you pull in terrible AI slop code, the project becomes unmaintainable by humans and you need to maintain it with AI.
There’s a little irony about the example you picked of the GPL, just because in the US the output of an LLM cannot be covered under copyright. The GPL works as an assignment of rights under copyright.
This is, of course, tangential to your point.
Code that isn't duplicated in 15 places, that doesn't read the whole database to fetch just 1 record, that doesn't do 15 database reads where one would suffice, that uses an optimized algorithm instead of an exponential one recreated badly from scratch... all that is also good code, and is just as important in LLM-slop era.
No code is error free. All code will need a fix of some kind eventually. When it does, it's better that the internals are not a total mess. In the same way that you don't want the insides of your car to be total alien nonsense either.
I think this is the future. Im more bullish than bearish. The only issue is that eventually human verification can no longer be possible and we will have to live under AI-verification assumptions....
We're already there with our pipelines and suspect a few more out there more concerned with measuring outcomes at this point.
What I’m more worried about is feature identification and how to monitor them. How can you get the product to the next level?
Honestly I’m not worried about code anymore in my codebase I’ve not had bug problems for a while and incidents get fixed automatically.
I monitor new features with an agent who checks recordings.
Most actually good features need still 20-30 iterations to design the userflow to actually be interactive, but then there is also another challenge to translate usage patterns into features and how to present functionality with the least amount of friction as possible.
How to efficiently run monitors without wasting your token spends. A best 'monitor' best practice would be nice.
Still this article was very helpful and gave me a lot of new ideas.
Token peddler says best way to do something is using more tokens.
This is not HN-worthy
Genuine question I’ve been wanting to ask folks working on different types of things: how are your teams doing agent-assisted code reviews? Not automated reviews, but what techniques and tools are humans using to keep up with the larger volume of PRs that seem to trend larger and larger?
I think a lot of people like myself still want to feel in the loop but I don’t have the right tools to stay on top of the (generally good, productive, and still thoughtful) contributions from peers anymore.
Using agents for a first pass is a good way to speed up the review cycle. This way contributors can get near immediate feedback before a human needs to weigh in. This doesn’t replace the human review, but augments it. Each time your agent misses something, update your code review guidelines.
what are people using for automated code review these days? the claude code review product originally mentioned it cost "$15-25 per review" (which is absurd), but it seems like they've removed that from their docs with the launch yesterday. we've looked at various products in the space (coderabbit, greptile, etc.) but have gripes with each.
honestly i'm thinking it might be easier (and cheaper) to launch our own agent from GH webhooks and trigger `/code-review`..
We've had great success with a custom gh action on a self hosted runner. It sets up codex and all the same stuff a dev has on their machine:
So dbs, seed files, etc, and then just a review prompt and some skills and a few scripts.
It triggers on a new published PR or if tagged @bot-name. It's way better than the other services we've tried (which was admittedly ages ago) because you can get it to work exactly how you want it and it's dead simple.
With Codex, for the time being at least, you can use a subscription, so it's basically free if you're already paying for one. But on API pricing it's still not too bad as it doesn't produce that much output.
I set this all up in about half a day. Codex does 99% of the heavy lifting.
this: gh actions using an opencode harness (so we can use cheap models through OpenCode Zen). With the right model, and the right, scoped access, it's just a prompt away.
bugbot was costing us $1k per month for a small team. EVEN IF we decide to go with a paid or say something like GPT Sol for our custom harness it'll cost us maybe $100. We've already implemented all of bugbot's interesting features like cross-PR memory.
the cool thing is we can leverage this for things that bugbot cannot do. Just today I added a security-reviewer action reusing almost all of the code that we wrote for the code reviewer.
I've set up a claude code on a vm with a bunch of skills that can use git and the Azure Dev Ops MCP via a service user, the regular build agent pops a record in a db that serves as a queue/review and a bash script picks things up and fires off claude code in headless mode.
It's pretty effective and lets you do nice stuff like compare the code to the tickets, reply to comments and stuff like that.
Using a different lab's model to review the code to the one that wrote it. Claude model plans/codes, OpenAI model reviews, or vice versa, or DeepSeek or GLM or whatever.
I do it this way too. I think having the same lab's model review the code in a fresh session also helps, so I use both: I generally start a new Claude Code and a Codex and have them review the code with a focus on 3 things: 1. bugs, 2. cut corners, 3. code style.
This generally works for me. I think a solid spec is the key for the models to review against. I ask them to follow the spec, and if the spec doesn't say, follow first principles.
Also, if there is a user experience change, they need to notify me.
How about your pattern?
> the claude code review product originally mentioned it cost "$15-25 per review" (which is absurd)
If an engineer paid $150,000 a year reviews some code manually for about 30 minutes, the cost of their time is about $36.
in a vacuum, sure. the average pull request will have multiple commits once opened (i.e. to address feedback or perhaps it was opened too early), each triggering a review. it's very easy to rack up $100+ per pr (and an eng may open a dozen prs in a single day)
I am happy people are testing these things out. I am also just going to wait a bit longer before really adopting things like this, I don't really trust it yet. A year ago this looked very different, and next year my guess is things will look different as well.
Poorly I'm afraid, given how many new bugs make their way to the claude code CLI every update.
I'm astounded that I kinda ended up building the same agentic workflow they are talking about in this article and distributed it amongst the org (eng and product/designers).
On the other hand, I would have expected Anthropic to have a drastically more elegant SDLC.
That explains why the UI keeps changing.
How is this a proper product? Every time you open the desktop app the buttons may be on a completely random place
The UI has improved substantially over the past six months. New capabilities are prominently featured and less-used ones are de-emphasized or removed. Every product goes through this type of evolution. The difference with Claude Code is that at the speed they iterate, it happens a lot faster and is therefore more noticeable.
You can call it however you want, but in the end it is just poor engineering. Imaging building a bridge that needs construction every week
Software is very different than other engineering disciplines. It is meant to be malleable. I think you need a better analogy.
Is it, though? Imagine NASA software having to change in-flight every day...
Claude Code is not in-flight software though.
customers don't like big changes to workflows they get used to; adding a new feature or a new button or menu item isn't a big deal but rearranging things so often start to feel random and shows little respect for the customer (or signs of any cohesive product design)
I'd be more interested to read on what 10% of the code is still getting written by humans and how it doesn't become a bottleneck at the pace they described.
Important/hard things are usually bottlenecks, its a good thing and usually by design. People should have more self imposed bottlenecks.
What are llm freaks in such a hurry for? Its like everyone is manically racing towards a finish line without realizing the finish line is you out of a job. Its like everyone is excited to be completely obsolete and poor.
My interest is mostly from the management perspective -- in 2025 I have seen a prime example when a heavily vibe-coding team chugged out half-baked products that other teams later had to deal with and this was not pretty and created all sorts of interpersonal issues; the same I fear may be happening here as well.
I'm looking at Claude Code and they chug releases daily and sometimes more often then this, if someone has to stop and code a critical path by hand or smth I'm wondering how this is handled.
--
On a separate topic, the "llm freaks" -- I know you intended this as an insult, but I'm actually proud to be called a freak again, and I thought this is the place where freaks are discussing things.
I didnt really mean it towards you, I meant it more generally but I can see how it came off that way.
While Im rooting for Anthropic to collapse under the weight of slop and their own contradictions. There's a leaked interview for Anthropic from earlier this year (I think) for their inference team. The interviewer seemed very competent and the quality of the questions were good.
This article is more likely a marketing gimmick in service of shelling shovels (for devs to dig their own grave) "hey look guys we automate everything with claude, you can too, just give us all your money!"
Interested to know how they handle ordering changes. Sounds like with that many changes, you're bound to hit lots of conflicts.
I built a library to have a way to check Claude Code's work against API docs, and asked it to write many tests on top, but even then I found Claude Code would paraphrase and drift from the docs. I also found that it passed its own smoke tests but failed when testing against the real APIs, so reviewing Claude Code's code is tricky.
I simply don't want to review AI code in depth.
I used to do it but now I only review code organization and structure.
Mostly I define the product behavior I want and discuss the technical implementation beforehand.
Once that's in a spec I let it build in my durable implementation engine. Automated verification, review rounds from different agent types across different models, issues triaged into meaningful clusters and flagged for false positives before finally being fixed.
That loops until the code is up to spec.
There is no special secret, LLMs need repetition and multiple passes.
They'll one shot bigger and bigger things in their default harnesses but they'll be filled with issues no matter what.
It makes sense for Anthropic to pitch "more code = more better" but as anyone who's been in this game long enough knows, "less is more" often enough. And good luck coercing an AI coding agent to solve a problem by removing things.
Personally, I find Claude's degenerate urge to just spew uncontrollable amounts of code quite counterproductive. Yes I get it, you get paid by the token. I'd like to see more reasoning, more understanding, and less generating.
Truth is, we are still very very far from actual AI software engineering. Some people are working on useful tooling, and (predictably, like anything good and useful) it is taking a long time. LLMs are great, and hopefully they will help us accelerate building the tools that bring this profession to the next level.
30k PRs in a month (or whatever) is not the brag they make it out to be.
If their only metrics are revert/incident rate then they're missing the long-term, cumulative impact of compounding complexity due to poor architectural decisions.
Well, that explains the Github repo with 5000 issues, maybe about a third of which are random regressions of previous behavior.
I snorted laughing at this. "We are experiencing tremendous growth" and it's just a codebase with 1,000 agents endlessly adding markdown documentation for every 1 line change.
https://i.imgur.com/lYjn8xq.png
their claude code changelog 'fixed' % over time.
https://i.imgur.com/KJcem6c.png
I believe their peak month was >13k issues opened, but worth bearing in mind: "This issue has been automatically locked since it was closed and has not had any activity for 7 days.".
I just learned that manually invoked skills MUST be the _first_ thing in the prompt, e.g. you can't have `disable-model-invocation: true` and say "Hi, /grill-me on X".
Worse, everything after the /grill-me is injected a second time into the prompt as a <command-args> so your prompt is literally added to the context TWICE. (first time in the original user prompt.)
I swear Anthropic is intentionally bloating your context so you burn tokens on useless bullshit like this.
Oh /loop was the crowning achievement. Basically guarantees you'll run a prompt at max context without caching over and over.
I have done exactly this as recently as yesterday, so I’m not sure why it doesn’t work for you.
To be clear: you can certainly do "Hi, /grill-me on X" anytime you want. You can even write "hi, /grill-me and use /search-docs", even though the /search-docs skill absolutely doesn't work. But the Claude Code _harness_ won't autoload the skill into context, so claude-the-model has to go "hmm, where can I find that skill, when I call the Skill tool with that name it no worky, so now I have to search the filesystem for that skill...".
https://github.com/anthropics/claude-code/issues/92769
The <command-args> I know because I just audited my raw Claude transcripts via ccglass. You can verify by just asking Haiku "hey, /grill-me does this skill autoload for you?" and then "/grill-me hey, how many times does this text appear in your context?"
You can trivially doublecheck me. The above prompts won't even eat 1% of a 5 hr on 20$ plan.
I was running into something similar where I had it working on a single Python file. Maybe part of the issue is that I had memory turned off, and the conversation might have spilled into two conversations. I wasn't using version control either, but since it was a single file with about 1000 lines of code, it didn't seem too hard to eventually get all of my features back.
30 thousand changes merged per day? no. like just no. what are we doing? I'm trying to understand
There was a post some months back called "Lines of Code Got a Better Publicist" or something along those lines. This is just more of that. I am not impressed by 30 thousand "changes" per day in the slightest without significant added context. When I logged in and prompted Claude today, do I feel like I got +600,000 change value from when I logged in a month ago?
I’m horrified by it. There is no possible justification for it but it does explain their buggy products.
lol right? So youre telling me they shipping 150,000 features this week? Or, maybe, they whack-a-moled 149,995 perf or bug regressions and shipping 5 new things. :shrug:
I, too, can write and merge 30000 changes per day. Even 300000 if I've up to.
Of course, just like "line of code" is a worthless metric, "change merge per day" is;
(automerged PR, JOB IS DONE)The premise is clear, is one agent can do what developer can, but quicker, why not speed up the whole workflow to match it?
Have you used Claude Code?
Have you seen the benefit of the churn of 1000 PRs a day?
It’s only about 10 per engineer. I’m not at Anthropic and before 2026 I was regularly hitting 1400 commits / year on my GitHub. If agents made me 2x faster (they have, review is still slow; I currently have 40 PRs open) that’d be the same rate.
> 40 PRs open
Ah, so you've privatised the benefits and externalised the costs.
Right, but why do things this fast to begin with, especially for a product that's already publicly available?
I don't want to use a product that shifts under me as I use it several times a day. I want a product that actually moves slowly and works very reliably.
As an end user: why would I care about how many hundreds of releases a month are being pushed if maybe 1% of them have features I care about?
Speed is not the product.
More code output isn't a metric of success per se. Goodhart's law.
Humans don't operate at AI speeds. We're making the car go from 0 to 5000 mph in 1 second and the drivers are turning to goo in the seat.
haha I do like that analogy.
We have to test many different api's and models as we have over 600+ in our harness now, Sonnet 5.5 via cloud code provides the highest quality results.
Over the last few years we have added mcp tools, memory graph, md files and skills, automated everything to the point that on desktop the system runs itself, besides proceed/approved/explain.
We have 100's of PRs/jobs/projects and Claude builds down the list automated and only asks a few questions point form if needed.
Very impressive now, compared to 6 months ago.
You can view it if interested: www.vekmem.com
Very impressive indeed https://imgur.com/a/xbwdkWc
Remind me to invest more in the clouds they are burning those tokens in.
FWIW, Pangram identifies several sections as 100% AI-written.
I would have been surprised were this not the case
And yet they can't fix the janky bugs when you scroll or show/hide chain of though, commands executed, configuring plug-ins from the options menu etc.
I wonder how they maintain the quality of tests for such a massive codebase and frequent commits.
That's the trick: they don't
Claude Code is great, but do you really push that much (new) code every day?
Sorry, but how the hell does a TUI harness have 30,000 changes _a day_?!
The 30,000 number is across their entire monorepo, which they say in the article Claude Code does not live in.
everything is shipped with auto reviews behind a flag. then you need a human approver to launch
good model
That's a shockingly low number.
1000/day in what sounds like a single repo is a low number?
I am not sure what it is you are saying. What I am saying: I would have supposed that Claude writes 99.x% of Claude's Code code, and that ~no code at Anthropic is written by humans anymore.
I agree finding out who is writing that 10% of the code in the AI company is important. Maybe time to identity them and put them on PIP since they are slowing down everyone else and giving a bad name to the AI company
The 30,000 number in the blog is the number of changes landing per day, not per month!
to achieve what?
Honestly not sure that there are that many effective changes a day given how little complexity there is to their public facing product.
No. 90% sounds like a low number. That implies something else is writing 10% of the code? People? Other models? What on earth is writing that other 10%?
They use Codex?
If the entire body of this article isn't just an "LGTM" then I know they're lying.
spoiler: they don't
90% ?
A random twitter said that, 6 month ago, it was 100% !
DECLINE ! SELL THE STOCK OPTIONS ! (wait, there are none, confused!)
Title is: How the Claude Code team does code review
I don't care how they do it.
I vibe coded my harness that is superior to CC in functionality, and it still doesn't flicker.
I don't need to hear their bullshit.
Anthropic's solution to anything is to simply burn more tokens. Because of course.
Anyway, flagging as misinformation.
tl;dr: We don't, claude does. Make of that what you will.