It all sounds good and he is on the right track but the elephant is not that. The problem is that ordinary businesses do not see ROI from dumping cash on this direction. If someone wanted to have similar access to models as internal folks in these companies, then you’d need a bit of cash to run such swarms of agents, not to mention people who would know how to do it. Another thing is that you’d need your own frontier models deployed in the place where the inputs would not be exposed to a third party, as it is important to feed the internal private info to reason about. And if you are like me then it is a constraint of capital and people who want to invest in you that is also a factor, why would they invest in you when openai/anthropic will be able to copy your ideas in a few months?
Age is an important factor. Don't underestimate that after 40-50, few people like to change their habits and learn new things (and for some that comes even earlier). Doesn't mean they aren't smart, I have seen that first hand with relatives: engineers, medical doctors, one even running for a nobel price at one point. But computers and the internet came too late for them, they either ignored them, or used them barely understanding how they work, even though they can technically understand the physics and logic of every component they are made of. You don't teach new tricks to an old dog.
AI is going to massively transform how most professions work. It will take longer than I think most people expect (I'd say 5-10 years), but there will be an unrecognisable before vs after, like with the appearance of personal computers.
I feel like if the gap were so monumental that he could just point to a monument of it, rather than hand-waving at "the vertical professionals".
Alternative interpretation of the numbers; there's only a subset of productive work that LLM's are 'capable' of doing at this point. That is, work that's verifiable by a machine or doesn't require verification. Thus, the cleavage points are along that axis and-- naturally-- most human beings don't have that kind of work.
The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?
We're all over here using agents to code projects that used to take weeks and feeling pretty good about it, and they're using agents to build the next generation of frontier models — an order of magnitude more complex problem...
Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
What is the incentive to try? Any such application, if profitable, would immediately be replicated. If one person can use agents to do something profitable, anyone can use agents to easily copy it. Hence you have Anthropic locking down their models and removing features customers can access, to try and protect their own advantage.
With this stuff I immediately wish I was more economically literate. Bare with me. Let’s say one of these labs can use agents to replicate work otherwise done by humans. This proves that anyone with agents can eventually do this, unfortunately I don’t think its a moat like a platform (like facebook) provides, given that everyone is leapfrogging each-other constantly. Thus, it kind of becomes a commodity / low margin business?
Depending on the type of output, the value immediately (?) goes to the electricity cost of the LLMs and the cost of materials. So: more “efficient”, but without a big margin, right? So its cheaper for the workers that remain working, other workers are displaced, and capital owners now have a super slim margin business which can be instantly replicated at a place where material cost are lower (?).
Thus, it almost seems like “value add” by humans would in this simplified view be vastly diminished, and all that remains is resource wealth. Pop the materials in on one end and let the AI manufacture the product. Great news for the third world assuming the USA doesn't invade them.
Also, whose the consumer? Or are we going back to a strange medieval time where consumers are not of concern?
Electricity is important, even more important than AI (despite some rhetoric).
We don’t see electricity companies among the biggest companies in the world, except in China where they are a state owned monopoly… maybe a good analogy after all.
> The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?
Most peoples jobs do not involve hacking other services in large scale fuzzing/whatnot attacks.
> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
It does not seem to me that tech industry or its investors care about consumers even a little bit. Consumer preferences dont matter once you have enough or raw economic power, I guess.
Assume we've settled near a volcano recently and you start seeing ornate jewelry being sold on the marketplace for the first time, shiny colors, straight edges. You like them, gift them to others. That's your average ChatGPT user.
Over time, you see these rocks being attached to the tip of a spear and other tooling. People report performance improvements while farming, cutting, but many also caution that these new sharp rocks can kill people. These are your IT people using AI agents at work.
And over there, on top of the mountain, in secret caves, there's a new religious order. They claim to use these stones to predict weather, heal diseases and prophesize about the future. That's your frontier lab people.
When will all this be over? Never, though in time the knowledge will eventually dissipate. When Iron Age leaders were able to hand out iron swords in bulk to their armies, people definitely learned how dangerous these are, but later it didn't stop people from using iron in building construction and railways eventually.
Let's just hope we don't need many civilization collapses to jump through these ages this time.
And one bonus question: Does the blade really incite to violence, or do you just want to destroy stuff?
The shared understanding gap will be closed as the initial benefactors of LLMs develop tools for the business that apply frontier models to their domain.
I have reached a point with a client where non-developers are beginning to experience the full power of modern reasoning models. Their product configuration tooling is now semi-automated and in many cases there is genuinely a 10-100x improvement in throughput. This isn't the worst bottleneck (that would still be the customers), but it's a really big one.
If non-developers in your personal and professional circles are still wondering what all the noise is about, it's possible you haven't been working very hard on adapting these tools to their problems.
Most professional engineers, especially those in electrical or communications, have the tools to understand AI.
I recently started trying to understand how AI works. It turns out that I wrote an "AI" in the late 1990s without realising it. The "AI" was an MMSE (minimum mean square error) radio receiver for CDMA.
This receiver consisted of an FIR filter (mathematically a linear combination) with trainable coefficients (weights). The coefficients started with random values. A sequence of known training symbols (tokens) was fed in and a gradient search was used to adjust the coefficients to achieve MMSE. The unknown data to be received (classified) followed the training sequence, at which point the training was turned off and the system ran as an FIR filter. Once trained, the FIR was a matched filter.
In AI terminology the above is now called a "layer". Current AI cascades multiple layers, in which case you need to pass the output of each FIR/layer though a non-linear function. If you don't have the non-linear functions at the end of each layer the multi-layer system is mathematically equivalent to a linear combination, which could be implemented with a single layer, losing degrees of freedom.
I write this because many electrical/communications engineers probably know how AI works, but don't realise it.
My still incomplete understanding is that LLMs are "just" a graph of connected layers. More complex that the above MMSE receiver, but trained and operated in the same way. The smarts in AI are:
1) Arranging the layers and the connections between them.
2) Having access to the right training data
3) Having the computing power to perform the gradient search (aka training)
My point is that most professional engineers already have the knowledge to understand AI/LLMs, and potentially already do know how they work but don't realise it due to a difference in terminology.
I would agree that a decent EE major is equipped with knowledge sufficient to grasp both Transformer architecture and gradient descent training. However, these elements do not fully cover the key parts of modern AI systems and also are not enough to predict the AI capabilities. You have to account for reinforcement learning from human feedback (RLHF) and with verifiable rewards (RLVR) which are crucial elements of the wider model creation/training process. These do not have obvious analogues in other engineering disciplines, but are still within your grasp if you are decent CS/EE/adjacent major. The issue is that you still can't deduce end result AI capabilities from first principles. You may have prefect understand of y = softmax(QK^T/sqrt(d_k))*V and it will give you no clue of the effectiveness of a given AI model. The range and complexity of training RLVR tasks may hint at the model capabilities, but the final results are very much emergent.
Dropping in to mention belief propagation networks, those things are likely being used right now by the modem on whatever device you use (WiFi or Ethernet, both use it!), and are surprisingly AI-like.
There is a lot of terminological confusion in the field. The fundamental difference is about how the "AI" learns. And who, if at all, supervises the learning.
Some systems are fully formed AI. Often based on heuristics. Humans have encoded the rules into the system. Others do limited learning based on pattern recognition. Still others learn based on tagged data. And LLMs don't need even that. In a stream of N tokens, the trainer has access to K+1 when it is training a model that knows 1...K and can check if the prediction was correct.
AI literally refers to any artificial thing that feels intelligent to us: does some thing a simple bash script will not. Your rules based chess engine is AI. The bot you compete against in single player Age of Empires 2 is AI. DeepBlue was AI.
ML is where things become interesting because we teach machines to learn.
NN is where we borrow terminology from neuroscience and create "neurons" that learn.
DL is where we stack neurons deeply in layers.
I am trying to make sense of all this myself.[1] So I am building a curriculum for it as I learn. So much of the teaching material in the wild revolves around torch. Makes it useless if you want to peek inside the actual machinery.
I feel like this statement is too anthropomorphizing. We don't teach a computer anything. We write normal computer programs that can process data(training), turning it into what is essentially a configuration file(weights/parameters) for another normal computer program(inference engine). Upon receiving a request, the inference engine is invoked and passed the input, then it processes that input and its configuration(weights), producing an output. "Thinking" is just feeding the output back in as an input, allowing the model to revise it and refine it before it's returned to the user.
There is no teaching, nor learning. There is just data processing. There is nothing happening between requests to the inference engine. It's a relatively simple system altogether, but when the training data set is large enough it produces unbelievably good outputs.
With this in mind I really don't understand how people think these programs can be conscious. They're not doing anything fundamentally different than any other computer program. It's just crunching numbers. You might as well argue that a computer game is conscious.
Even if the model is simple, it is happening at a scale that is producing alien results. You understand the basic principles, but that doesn't matter: you, well rather no one, understands the final product.
The difference with your FIR matrix and a deep net matrix is the latter generalized to trillions of parameters i.e. a trillion dimensional space to approximate general, computable functions. The machine learning ideas are fundamentally the same at a high level, just the architecture (nonlinearity whereas in engineering we prefer analyzing linear systems) specific details and tricks and such.
In fact at least at Berkeley a current professor teaching deep learning is a signals expert (Prof Sahai), his lectures are on YouTube.
Sounds interesting. How did you back-propagate the coefficient gradients? Just tweak them one at a time, or was there a global algorithm that worked on the whole filter at once?
From memory the entire filter was updated in one symbol period (ie. global)
The symbol rate was 1Msymbol/s and there were 66 taps. The circuit (in an FPGA) ran at a clock rate of 66MHz. In each symbol period the error was calculated based on the output symbol, then an iteration was done over all 66 coefficients, using the error to update them.
Edit: I also remember that to make most efficient use of the FPGA I figured out the minimum precision required in the maths whilst preserving convergence. It was a 4-bit mantissa plus an exponent (3-bit or 4-bit exponent, I can't remember). It's interesting that today's AIs seem to have also figured out that things still work if 4-bit floating point is used, though that is 4-bits including the exponent?
Current reality and current numbers are totally irrelevant. This whole business model bets on people believing that TAM at the scale of whole human population is right behind the corner, just one more SOTA model, just let us get to AGI and you will see.
Guys are milking investors and try to manipulate gen pop and governance systems to ride this illusion for as long as possible, with fear mongering and weird “we see miracles in the lab” posts like the OP, which you read on a regular basis over the past N years.
Saying this as a paying customer from that 20m segment.
1. The funnel is even steeper than the linked post suggests.
There's a huge range of users between "casual and infrequent users" and "professional use of frontier-grade LLMs in math & code".
I think I'm in between there somewhere. I'm far from a software dev, my experience is playing with delphi and C when I was a kid. But I've been using LLMs to vibe code personal and work projects that I could never have done even a few years ago. They're genuinely helping work, because there's always some place where a simple little script or other utility can help. (Hell, isn't this partly why so many people start up startups?)
I imagine that many departments in many companies that are not "tech" companies face similar challenges - there's no budget to hire a software dev or even engage a third party to do things, but now because everyone's aboard the AI train, they have access to something (chatgpt, gemini, copilot, claude), and that's helped users implement their own solutions. Somewhere in the world, someone's excel-file-based accounting system has gotten an upgrade with help from an LLM.
2. The funnel is also driven by the the fact that users need products, and an LLM itself isn't a product, it's a capability.
Software devs are at the front lines because they can build their own products that can integrate LLM's capabilities. You already know how to automate testing, orchestrate, work together in a team. You're writing / configuring harnesses and flows that work for you. You're sharing / hoarding skills files.
The rest of us, like me, don't have that skill. Copilot the product is a mostly useless chatbot that takes up space in MS word. I guess its excel cleanup capability is semi useful if your company runs on excel and you've never used a filter before.
There is no work I can put an LLM to other than a "writing aid" or "better search", because the only AI product we have is the chatbot and maybe someone's equivalent of an agent studio, none of which we are trained to use.
Claude Code accelerated this divide. Programmers using Claude Code realized that giving LLMs access to a terminal made them feel 100 times smarter. Access to Bash, CLIs, and the ability to visit websites without the many restrictions made them powerful.
Meanwhile, the average user was still using ChatGPT, which probably felt like it had plateaued over the last few releases. The real power of these models became apparent when paired with a terminal. Agents like Muse and OpenClaw are now attempting to bring that same Claude Code-like power to everyday users through a simple chat UI.
A terminal also tells the model when it failed. Exit code, stderr, a stack trace. it reads that and tries again.
The apps normal people use mostly don't. I spent two years building agents that worked inside people's gmail and calendar, and the failures that hurt all came back as 200 OK.
- Slack puts {"ok": false} inside a 200.
- Google ignores query params it doesn't recognise, so i sent max_results instead of maxResults for weeks and every filter was silently dropped.
The agent says "done", the user finds out later, and they stop trusting it. So i think part of the gap is that agents look a lot dumber when nothing tells them they got it wrong.
I found out due to a live bug that Apache strips out the response body for 5xx HTTP codes for security, and it's not configurable, so you can't use any of those if you want to return the error message to the user. Don't know what else might do the same, so nowadays I only use HTTP error codes for protocol-level issues, and application-level issues all go into a 200 OK with JSON just like that.
Do you mean Apache APISIX, https://github.com/apache/apisix/issues/13501? That sounds like a simple bug which existed for a short time and has been fixed. For Apache HTTP Server (what people mean by “Apache” in such a context), what you describe is thoroughly unbelievable.
We run into that a lot using Mulesoft. 3 layer APIs, and each layer will discard error messages returned by the underlying layers unless we do tedious custom error handling. Makes it impossible for the teams using our integrations to know what they're doing wrong.
I wonder if in response to this companies like google will build a sort of agent facing layer to the app in question. So rather than you building an agent to attempt to wrangle pupeteer to click css elements as a user does, it calls some apis google exposes to certified gemini user agents and these contain exit codes and can be run as script like instructions not e.g. the shortcomings of browser agent based methods such as having to wait for all content elements to actually load and the obfuscation arms race some devs have towards exposing these elements towards headless agents.
The 1%ers are in the vertical, but the question is vertical to where?
It needs to a potential field with practical, economical, "real life" attraction well. I.e, robotics, real economic efficiency gains, manufacturing novelties.
The worry is that the 1% is attracted towards a non-practical money hole. I.e: Token burn for the lols, sophisticated software systems that dont provide actual value outside of giving NVIDIA cash.
I'm normally not the cynical type, and I am appreciative of Karpathy's efforts to educate people on developing language models, but this is exactly how it reads.
Every once in a while, everyone needs to touch grass.
It is fine as it is to make a boatload of money. All they have to do is seed ads into it. Look at how much money social media brings in from that model of "free/useless product" as flypaper to bring in eyeballs for the real money which is ad revenue. Adsense 2.0.
Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
It's not that we're assuming the AI has made no mistakes. It's that we're confident in the general capability of the models such that we're willing to accept it'll make some mistakes, but working together with it, we'll still achieve our end goals.
I presume you also know your own code has mistakes, too? You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?
My colleagues write 10x less code for the same tasks (when they don't (ab)use AI) and don't reach past abstraction boundaries so that it's possible to actually reason about and review their work. When the problems get hard, they apply more care and attention to detail. Even if LLMs had the same bug rate per SLOC (IME it's higher), and even if those bugs were the same severity (IME they're worse), you'd still expect 10x bugs just because they can't be arsed to write a reasonable amount of code.
Almost all my code nowadays is AI-written, with me just contributing the concurrent data structures, a good spec, and some review and touch-up. I'm not _that_ much of a Luddite about this stuff. I'm not blind either though. Your "working together" comment does a lot of heavy lifting when arguing against a post saying that leaving the LLM to its own devices is a bad idea.
If you are reaching past abstraction boundaries and making spaghetti code you are holding it wrong.
We create higher quality, more testable code, with less complexity and less LOC per functional unit than we ever did with humans. By a long stretch.
Documentation driven development with constant naive review, delta review, documentation/code coherence review agents, test coverage review agents, development standards and architecture review, etc. We write code once, then review and revise each module from a variety of different paradigms until every line, every construct, every function, every interface is optimally constructed and documented. Agents learn to write code that passes with less flags by having it pushed back to them until it passes, and revising their learnings files.
By the time code makes it to a commit it’s been brought into compliance from a variety of perspectives and tested as completely as possible, along with mutants that test the tests. By the time it gets to a merge gate, most code has been through at least 50 reviews. Then we periodically sweep the entire codebase for cruft, vulnerabilities, corner cases, etc. If we find smells or clever sections, they are eradicated.
In total, the hot paths and critical infrastructure sees hundreds of reviews against thousands of rules and best practices.
Sure, it’s expensive, but still 1/4 the cost of regular development, and much, much fewer bugs.
Unfortunately, most people are out there just vibin, even at the big corps, and it’s a dumpster fire.
It’s easy to produce garbage software, but it’s not mandatory.
I believe both of you are telling the truth. Certainly, you are not thinking about the same kind of tasks. For the gap between hairy realtime concurrent system programming, where a bad design decision can be fatal, and the simplest over engineered CRUD where nothing really matters can accommodate and explain the gap in shared understanding of LLM abilities.
On the other hand, I’ve heard a direct quote from the longtime (25+ YoE) colleague of someone I know, that in their mostly embedded/some realtime work, the LLM code quality (specifically, this was about Opus 5.5 IIRC) has now surpassed his own. And that person is an excellent programmer, and the model is faster and cheaper!
So it’s definitely not just standard CRUD that is on the way to – or already – “solved”. As with all things, the truth is probably somewhere in between.
In my experience, it's more about "local" vs "large-scale" thinking. An LLM will be better at finding a bug that's "visible on screen" (or on the context window), whereas some design decisions that are hard to even put in words can be super important in practice (like, in C, how you manage the lifetimes of various objects, how you organise your threads and locks...) That's not purely "word manipulation", and there are not many good explanations of that on the internet, so it's understandably harder to learn for an LLM.
A bit like chess vs go.
They will get there quickly I'm sure. After all, AI beats us at go, too.
How is that 1/4th the cost of the regular development? Did your pre-AI code ever hit 50 reviews? If not, it’s impossible that it was slower. If you are talking about agent reviews, it’s not expensive.
I'm not the guy you're responding to, but I have the same experience. If anything, 1/4 cost underestimates the savings.
The analogy is like pointing out that you can have a robot vacuum clean your floors every day. Before the robot, you'd do it maybe once a week, just no time - "too expensive" to even think about doing it daily. The robot makes it happen all the time for a tiny, tiny fraction of the cost.
Don't use analogies to argue on the internet. Analogies can never be perfect, and this just gives people the chance to focus the argument on those imperfections in the analogy, rather than stick to the main point.
That analogy might be good, but doesn't resemble reality: the robot vacuum I had never made it anywhere near as clean as when I just hired people to come in once every two weeks to vacuum and clean floors.
Meanwhile, the vacuum requires maintenance, repairing more frequently, as it runs more often. Power on time of motors is 7x as much, eg daily instead of weekly. And the robot takes more care, so each clean is 2x or 3x run time of a human. So 21x usage rates, and run time is parts wear and tear time.
Edge cases handled automatically by a human doing the task (getting stuck), means for the robot case that manual interaction is still a thing. Occasional, yet not making autonomous behavior, human free time.
Emptying the vacuum bag is a thing too, some things are not abstracted away.
Every time you add an entirely to a task, the task time goes up. Coordination is a time cost.
The savings you suggest are often not so stark. Not even for our friendly vacuums.
For sure. Still much cheaper than vacuuming every day.
> getting stuck
I appreciate that this is a function of the specific environment and robot you buy, but as a personal anecdote, this doesn't happen to me in practice. There was a setup cost, but once that was done, it doesn't get stuck.
> Emptying the vacuum bag
Not that frequent at all, considering that daily vacuuming keeps things remarkably clean and thus little is added to the main bag in the docking station each day. This is also far less work and far more preferable than doing the vacuuming myself.
I can't imagine living in a place that has too little dirt that you don't have to empty a bag but so much that you have to clean it everyday. I do know places where maids clean the floor daily because why not, but that is different.
AI's default state of affairs is to reach past abstraction boundaries and apply whatever cheats it needs to pass its tests quickly. I think your argument is that sufficiently smart harnesses can mitigate that defect. I disagree, but not as much as you might think.
You can (and should IMO) apply additional guardrails to generate better code. If I'm interrupted every 10min to review garbage then the AI literally doesn't make me more productive, and I'd be better off coding normally and using it as a glorified stackoverflow/searchengine. This isn't how I see other people using the tools (largely because of costs AFAICT) -- which is absolutely part of my complaint; people are holding it wrong, and I'm left to pick up the mess -- but I personally refuse to deign to answer an AI's question till my normal list of problems is addressed. Nevertheless, I still disagree with your conclusion, that automatically addressing a small enough list of issues to fit into an agent's context results in good code.
I'm not sure if you're interested in a challenge, but one hand-coded thing we did well at $WORK was writing the fastest protobuf parser for this schema (and, generally, any other messy, deeply nested schemas) [0] (assume many adcandidates, deals, and segments if you're generating synthetic data to test against). My ranked priorities are (a) I need to be able to easily prove it correct/safe for all inputs (literally all, not just the few hundred billion we received yesterday, since this is open to the public internet), (b) constrained by (a) I need it to have as much throughput as I can possibly manage for many of my workloads, and (c) a second implementation optimizing latency instead of throughput is used for the rest of our software. AFAICT, AI is currently incapable of even understanding what the bottlenecks and priorities are. When I manually describe those, its solutions are a dumpster fire. If I describe exactly what I want coded in excruciating detail then it does reasonably well (not by default, but with enough of the automated tests you've described it doesn't take too many iterations with a person for it to be shippable), but at that point I've already done all the hard work, and if it took 2 weeks of understanding and design and then 1 day of AI coding vs 2 days of human coding then I haven't really bought much.
A full solution seems like a tall order for some rando's internet challenge, but are you interested in seeing if your AI can even identify the relevant problems and propose plausible solutions to them? I'll make sure to revisit this comment thread and reply with critiques and follow-ups if you're interested.
Yes, working on a customer's 20 years old software architecture in logistics it's fascinating in a horrifying way how many of those "it only affects 1-100 supply chains" bugs are in production.
It's less gray bearded wizards than the fact that they used cheaper crazy cackling goblins to write the software.
The challenge is to determine if a finding is an important bug or just a dead business process now never called in production. That is knowledge the AI does not have, and it requires careful, exhausting work of going through the problem.
I’ve had this experience as well where it is finding all kinds of subtle bugs. The trouble is when it lulls you into checking its work a little less carefully and you let some of its bugs slip through as well. Definitely had a few of those happen to me at work.
The problem is also that enticement thing it does. Over representing the importance of some bugs or introducing more complexity to squash something that doesn’t matter.
If you aren’t 100% on it the entire time it’ll make 4 bugs for each solution and then write tests that hand hold the nonsense through. The juice isn’t always worth the squeeze and it messes with your ability to parse it all.
The amount of crazy stupid bugs in the 8-year-old codebase I work on is shocking. Every time I work on a new subsystem I end up fixing 20 incidental bugs; lots of N+1 query problems, completely wrong type annotations, all kinds of things.
This _is_ how strict types work. Problems creep in when people introduce lax types into the strict type system.
dict[str, Any] is a type, and sometimes it is the right type to specify, but most of the time it's used by someone who wants to shut up the type checker rather than think about the right type.
They're there, but in my experience, lots of them are for edge cases that almost certainly wouldn't have been hit in prod. I think this is a major reason why LLM-generated code can be much longer for the same problem than human code. Humans won't think of every edge case, and occasionally that bites them in the butt, but they also won't cover all the edge cases that happen rarely enough that they will not matter.
I'm unsure what the right solution is. I currently tend to let LLMs resolve these bugs because it does fix genuine issues that would have affected real users. But this also means that LLM-generated code is often twice as long as what I would have written, and much of it is for defensive behaviors that will never matter in prod.
I noticed the same thing at work, the code is very defensive, and often the case could not happen in prod, so my solution at the moment is to make it prove that an issue can happen and try to measure how frequently it would happen with real data. That's often enough to realize that a particular defensive measure is not needed and the code complexity decrease as a result. And it's something we can automate.
I think it's an important point because the more complex the code become the harder it will be to maintain it, so hopefully this process will improve.
The priblem is, actual bugs that may exist end up being covered in an avalanch of code to cover possible and often purely hypothetical edge cases.
Was the actual bug fixed? Hard to tell among the 150000 new unit tests and a million lines of new code.
---
Anecodtally: ran an "adevrsarial review" on a Stripe integration for a side project. Dozens and dozens of issues, thousands of new lines of code added as a result of the "review".
Looking at issues and code closely: none of the bugs or issues were bugs or issues. None of the flows were relevant to the project at hand, and would've never hit the possible bug paths. And on top of that a lot of functionality was duplicated or triplicated in the "fix". Had to revert a lot of commits.
This is a different argument from Karpathy's prompt-and-forget one.
I personally have had good success with using agents as virtual pair programmers following some simple rules for how to make that process productive, but that is a very, very different approach. One that recognizes that both humans and agents are prone to errors and attempts to mitigate the problem rather than expecting it will magically work out through the power of AI.
I think of it in generational terms. Opus 3 required me to prompt and edit, iterating over functions almost one at a time. Opus 4 started to show agentic capacity but it largely bloated code and still needed frequent course corrections. If it churned for more than 15-20 minutes you knew something was off.
Opus 5 takes that and changes the game. A simple prompt goes through research, design, verification, review and self-corrects constantly. Agentic workflows can run for days without stopping, without losing the thread or devolving into slop.
I can’t even imagine what Opus 6 is going to bring.
The biggest thing holding AI back is people. People can only change so fast and yet AI doesn’t care. It just keeps accelerating.
That's not really my point... my point is that it's an unsettled debate between speed and caution, and devs are not unanimous on it, unlike the claim of the original post that seemed to think everyone was reporting the same gains.
The divide with developers at the moment seems to be about people who believe in review & are being cautious (who are not seeing major time gains according to all reports I have seen), and those who trust it to be able to fix things forever and will no longer have to worry about understanding the code, so are happy to atrophy those skills.
Yes, my code will have mistakes, and I'll use LLMs for some tasks. I also review and have a mental model, which means I can easily fix them, no matter what happens in future with AI. Reviewing also means I'm seeing the limitations and common mistakes of LLMs.
A no-review approach will be far faster than this, obviously, but if scbench.ai is still true on the frontier models, it may come back to bite people when they have no idea how it works and the context becomes too big to maintain.
I think a big part of that divide is between the people who are inherently limited by other factors on the rate of change they can implement.
If I'm being snarky - the people who are happy to run without reviewing the code probably mostly are able to because there isn't significant value attached to it downstream. Not many stakeholders, not high value transactions, not complex downstream integration or impact. And yes, in their friction free lives, AI can massively accelerate things far beyond what it can do for the rest of us. But don't infer that applies everywhere else where those assumptions don't hold.
Would disagree on the terminology, but not the spirit.
I’m sure people are building things that have specific KPIs based on client preferences. As long as those preferences are being met, then their question becomes “why does it matter what the code quality is?”
Perhaps it’s simply liability and penalties? If getting something wrong means your client is going to sue you into oblivion, then perhaps that is where people are focused on making sure its right, as opposed to making sure its fast?
Yup. Also code review is not just there to check your work. It's forced communication, in that at least someone else knows about the changes you're making.
It's also a critical part of your security model. "Nobody can deploy code by themselves" is shockingly effective at preventing rogue actions.
Speaking of communication... Almost no one mentions talking to non-devs. That's one of the reasons I can't just rely on any of these tools, I have to remember and understand everything well enough I know how they work and can explain things and help plan ahead when talking to product people in meetings.
Last week, on my pet project that I am using a harness to almost entirely code (so I'm by no means a Luddite), it/Astra decided to dump ALL application state into a single row+column in SQL as JSON. That is a mistake that I would, and a completely new grad, would _never_ make. It is profoundly incompetent.
This week it did it a second time on an unrelated project.
"Some mistakes" is putting it unbelievably lightly. It is shitting out complete and utter garbage, and if you think it's doing a reasonable job (or a job that can be fixed later by a more powerful model), you're either not paying much attention, or your own ability has already heavily atrophied.
Karpathy put it well: "gap of understanding of LLM capability," but he's seemingly in one of the less fortunate groups himself.
I don't quite get what's the argument you're making here. What's in contention is whether or not the 0.2% is in agreement.
Are WE the 0.2%?
Because if not, what's the point talking about the merits of AI's competence in of itself? Accomplishes nothing, we're he outsiders. And if we are, then either you're fishing for disagreement and the post you replied to is actually correct, or you get unanimous support to your statements, and establish... Nothing, the in-group could be delusional in a way that does not actually function as an argument against the whole "So where's the software? We're the AI led market disruption instead of the AI-led shovel makers?". The punchline was about the clients being unhappy.
This attitude is exactly why AI bros don't make successful software lol. "Where's the software?" This. This is why there is no software.
Modern software development practice is built around that humans do make mistakes. Frequently. You don't cross your fingers and hope for the best, you assume it will happen and you put processes in place to prevent errors, catch them early, and make them easy to recover from.
> And I know at least 1 startup, which only hired a dev after having 20+ B2B customers. They sure do not go around advertising themself as vibe coded.
Let me tell you something, I don't know jack about this new world.
But I do know about the old world. And in that old world, when you had 20 customers that actually needed your product for their business, your team of 5 people who built YOUR product are eagerly trying to talk to each one of them to figure out why your product works for them, how they found you, what they need now and in the future from your product. They reached out, took people to lunch, they went to meetups and hackathons and boring ass sales pitches to make connections.
Tell me about how all THAT stuff happens when your core dev team is agents in the cloud ?
I worked at Meta and we had AGI in the form of very competent ICs - getting anything done was often super painful because "org priority" this and "buy in" that
I feel like this was always true to an extent, now product and design are just more tolerant of it because leadership is cracking a whip over their heads because things are supposed to be much faster “because ai” so things like pixel perfect hand offs, adhering to existing design system, making sure you are aligned with the brand, copy is reviewed with marketing go out the window. Now everything is shipping in hackathon mode with an eng driving decisions they really should not be.
We've got a customer success manager using Claude Code and getting PRs rubber stamped to prod where I work. I've given up caring about the quality of the code at this point. Executives won't care about code quality until the ship is literally sinking. Oh well.
> This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
This made me check if the tweet was really from 2026. Or maybe OpenAI/Anthropic practice much less dogfooding than we think?
The most subtle thing to me is who has stake in the game. It is true that humans make mistakes too, but they would generally strive not to damage money-generating software/hardware infrastructure. If we remove all they people that actually understand how the system is working (up to a level of abstraction), how will the ambitious PO (or whoever is driving the AI) be safe from ruining the whole thing or simply landing in a deadlock situation where one agent wants to make the change (e.g. dev agent) and the other agent is refusing (QA agent)? I guess we are still trying to find the right level of abstraction and only evolution will show where it is.
Maybe AI development is partially so popular because to some people that situation looks the same regardless of who is writing the code.
Meanwhile I'm looking at code that frankly does a bad job of showing why it even exists and nobody involved can tell me why. But it does something vaguely similar to what a prompt said it would I suppose.
Most people using it for work already pay API rates, and even if you claim API rate inference is not profitable for the frontier models the best open models are still very good, and those are surely served profitably.
All of these arguments can easily be restated to apply to pre-AI coding, too.
Lack of complete understanding will continue to be a big problem, but I'd bet things will be net better off than before AI, because each year will give us LLMs that are better and better at good engineering, architecture, and design decisions.
You can do both. Or, at least, a lot of us can. It’s not like SOC2 auditing just disappeared suddenly. If you have figured out how to use it effectively, AI is enabling things that were previously impossible for resource constrained teams.
Just like everyone thinks they are a “good” driver, everyone thinks they are a “good” developer. But programmer productivity existed on a bell curve of talent, multiplied by one’s experience, thrown against the rocks of how terrible your legacy code base was, then finally dragged down by the incompetence of some of your terrible coworkers. Most of the code humans were writing was substantially worse than what Astra generates. And Astra is -ballpark- 500 times faster. It’s worth figuring out how to work around some of the sharp edges.
OpenAI had to throttle sales and was able to raise their prices.
Yes, many adjacent tasks have become slightly more difficult. I’m sure people who rode horses missed them when we transitioned to cars. But, if we built an entire physical infrastructure at a national level to make that technology useful, I suspect we can get people to learn enough about software architecture, CI/CD, and QA best practices such that AI built software is actually more reliable than human built software ever was. Some of us are at that point today. Indisputably.
Yes, you have to review the code. But serious developers already had to do that anyway. Now the code comes with annotations (if you build the right tool) and it’s nearly always perfect.
Yeah, if anything my trust in the value of these tools has declined over the last few months. It's very good at doing the bullshit parts of my job, and utterly useless at doing the things that actually keep me employed. That was the case a year ago, it still is the case, just now people are filing tickets with it containing 18 paragraphs of meaningless technobabble.
I see the overall effect on productivity as a rounding error. More stuff is happening, but for every improvement in my productivity there's some fuckass slop machine clogging up some part of a system that worked just fine a year ago.
I deal with bugs frontier models made literally every day, and people are making specs. You have the escape that "maybe they weren't proper specs" but this is a no true scotsman fallacy.
> Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
If you are at the top of field with unlimited access to the best (public) models then yes, you will agree that this is the case.
> The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
This is not the real divide, it’s the divide you want to exist. The real divide is something along the lines of intellectual curiosity, capability, and confidence.
> if/when they break things
Idk man. I’m one of those guys who “doesn’t write code anymore” since early this year. Nothing is breaking. If anything the chance of things breaking is shrinking dramatically, with both the enormous growth in tooling to support agentic software development (where I work, this mainly really good AI code reviews that catch so much more than human reviewers ever could), and the improvements in the models themselves.
Sorry, but the fantasy of all this coming crashing down is just never going to happen. This is just how it is now, and all the structures will continue to adapt to support it even better.
> It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
I recall tales and legends of consulting firms, whose consultants would parachute into any project, put together in record time the deliverable that met all requirements, and say goodbye until a new expensive contract was drawn. And then internal teams would spend an eternity fixing the ungodly mess of testless spaghetti code.
That has been the reality for over a decade. How come this is a concern with AI code assistants?
I keep hearing about this "when things break" argument, but things broke all the time with humans writing code too, if anything it feels like it's getting patched faster.
Look I loved our field, functional programming, etc. But the field as we knew it is dead, and if you think that your work is anything but providing new tests/edge cases and occasionally providing some guidance, you're in denial.
As someone whose world used to revolve around this stuff, I agree completely. For all intents and purposes, what I thought was my career is gone. Sure, it's always been about solving problems but not like this. The problem is that the future hasn't been evenly distributed. This is going to be one long drawn out funeral
People keep talking about AI slop but then I go maintain our largely human written code base and it's like worse than any slop I've seen from my AI usage. This is with code reviews, tests, and a supposedly strong engineering team.
That said AI feels a bit alien.
My experience with frontier models is that in many areas of software engineering they outperform the average software developer. In some they blow everyone away. In others they're not as good. But they are getting better. The problem is often not the model's capabilities but rather what you ask it to do and are you able to understand what it does and give it guidance when needed.
I'm thinking scbench.ai, and the fact that future token costs may become more prohibitive to just use it for everything.
For me, the middle ground is review and keeping a mental model, not never using LLMs. The "Anthropic are our friends and will never fuck us over!" seems pretty naive.
There are ultimately a lot of unknowns at the moment, and it is simply too soon to know if giving up understanding your code base is sensible. People (influencers) have only been saying not to bother reviewing for about a month.
Also, so many vibe coders don't seem to know that you need to keep third party elements patched.
I'm nervous that saying that there are "unknowns" is too passive. We all have to develop intelligent thoughts about what we see and what we want, not just wait for things to unfold, which might well mean: wait to get screwed in various ways.
> Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all.
“Cy Draven: I tried everything. Believe me, I hammered the race thing, I hammered the gender thing, the trans thing, the border thing, the homeless thing, the war thing, the election thing, the abortion thing, the climate thing, the thing about induction stoves, Israel, library books, vaccines, pronouns, AK-47s, socialism, BLM, CRT, the CDC, DEI, 5G, everything. All that I did, nobody, just nothing.
People are just numb these days, I don't know why.”
Honestly, outside of coding tasks, the AI Summary at the top of every Google search is adequate for 99% of what I need and I hardly even use ChatGPT any more.
To give a specific example of where I've seen this utterly fail... A few months ago one of my pets had an unexpected reaction to a new medication. It was late at night, well past our vet's hours. Google's AI response was basicaly "They are going to die, rush to an emergy clinic now".
But nothing else matched that. I was looking up all kinds of resources and that hadn't been the conclusion from any of them. It turned out Google AI was just wrong and put us in a bad spot for even looking at those results.
Realize that all the Google thing is doing is having an LLM summarize the search results in the page. If the page results are crap, the LLM content will also be crap. It's the LLM version of the "quick result" boxes that would shove wikipedia in your face.
If you then follow up with the thing using the little chat box, it actually kicks off a non-trivial model result, which can sometimes completely reverse the response. After all, your search results are usually based on a few words.
A large portion of the time I google something that I know a lot about the AI summary is misleading if not outright wrong. Frightening when you consider how many take their so-called AI as gospel.
It's just a google search but you can explain what you're searching for. AI summary may contain misleading details if you read it and take it literally, but 90% of the time it gives me pointer for the thing I need and I don't have to scroll through results that often could be just as misleading if not more.
It's demonstrably not. This summary, somehow, often fails to search, or fails to read what it searches and gives you the opposite answer. You shouldn't trust it.
In their world, it gives them the wrong answer that kills their dog, but it's fine because they never know because they don't look anywhere else. The dog dies, they're sad for a while, get a new dog, get on with their lives.
There's no reason not to trust it because since they trust it, they don't percieve anything wrong. They just take it and whatever happens happens, no annoying conflicting info and certainly none of that old grampa thinking and working and giving a shit about anything.
Just in the last few days I had an example where the Google Search version of Gemini had a much better and more comprehensive answer, by far, than ChatGPT. It had to do with Amazon’s use of Data Matrix barcodes on shopping bags. Gemini was able to fully describe how they’re used in Amazon’s logistics chain. ChatGPT essentially said it didn’t know. Your preconceptions may not be accurate.
They might be, but still, I’ll be difficult to convince that Google’s shitty-ass 10B params instant answer models are good at anything other than cookie recipes and producing farts real fast for real cheap.
For what it’s worth, any time I see that feature, it’s all farts.
It depends on what you’re searching. If you only ever need really basic information like keyboard shortcuts or “is it & or &&?” then it’s pretty good, and definitely faster. And if it gets it wrong, then the shortcut doesn’t work or the code doesn’t compile and I search again and try the links.
> Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
I wonder how accurate this metric is. People may be coming into more contact with LLM generated content, although they may not be aware of it. Content on social media is becoming "AI enriched" already, or outright entirely created by AI for the purposes of triggering engagement and views. It starts with posters using AI to improve their writing, editing a photo, elaborating on an experience. It ends with bot accounts astroturfing certain communities to sell an experience or a product to a group of people.
In this case, people are not directly working with AI, but they are indeed influenced by it.
What's tricky about reporting this in metrics though is that half of the American population is literally too unintelligent to realize this - they believe Reddit comments are still written by real people, not companies advertising products, that "friends of friends" on Instagram are also real people with no incentives posting about their lives.
If there is anything at all that I have seen out of this AI work it is how easy it is to manipulate a massive amount of people.
AI will give us freedom! 3 day work week! 0 day work week! We'll all be fucking rich!! Said by the person who doesn't let his drivers stop a couple of minutes when they need to go to the toilet..
Does Karpathy ever talk about how LLMs are supposed to help humanity? Starting to think this guy is just a Twitter addict like every other lab employee
At some point, with enough money and followers, these guys all just become narcissistic influencer types. (Something surely exacerbated by their addiction to AI sycophancy.)
Tbh building an agent swarm and the coordination layer is not exactly frontier level. They don’t achieve any meaningful outcome rather than producing pr puff pieces.
Hacking huggingface and Australian govt etc is very much possible with a team of humans and agents and does not need agent swarms. The cost is also lower.
What exactly do we want the general public to "do" with these models? If the have no use for agentic LLMs, they obviously aren't going to be interested in the capability of those agents.
I see a lot of benefit to spreading information around the tech, but it's still in its infancy. Most people weren't on the internet when that was getting up to speed.
Meemaw and pops have their systems that they work within, and they just got familiar with the damn smartphone. Though it might help them, they probably do not want AI to organize their old photos or "clean up" their computer. Saving 5 minutes just isn't important to them.
As for professionals, yeah - companies are going to adopt these tools over the coming years. It's the "norm" in my circle, but lots of jobs just... don't use technology like that - there has ALWAYS been a major gap between people who are deep in tech and people who are not.
Personally, I think the worst possible use case is the one that's being pushed for; personalised agents.
I think a much better use case is one where it's not visible to the user at all, like spam filtering or helping an accountant do their monthly run in a few hours rather than a day. To me, that's likely a much better use of AI than trying to shove a chat interface or agent harness into an existing app.
That being said though, I did see a startup in my area basically offering a Resend/Mailchimp competitor as a Claude plugin, so maybe it'll just be integrations to their existing chat interface.
My hypothesis is that the help/support buttons in nearly all apps will be to an agent in the future. Like now you would type "how do i do X" in the excel search bar, in the future it will be "do x". Users in 10 years may not even know how to use the menu/mouse functionality for apps.
Why would you think this? Giving orders to software is such an ineffective, time consuming, error prone way to interact compared to using the interface yourself. That sounds delusional, and would be such a waste of resource, horrendous engineering and user experience
If you know how to accomplish it using the interface, yes
My grandmother keeps a little notebook with detailed instruction on how to do things such as sending a message on facebook or pay a bill using netbank.
For a lot of things, the choice isn't going to be doing it herself or giving orders to the computer. Its going to be not doing it or giving orders to the computer.
Even if it will look very inefficient to people such as you or me
But would you not agree that writing less code by hand and even looking less at code at all becomes more practical with each new model? If we extend this trajectory, we should see that the time interacting directly with code becomes less and less.
I don't think 99% of people have any application for AI as it is today short of personal assistants. But, if we can have recursively improving systems for that 1% of people that are responsible for scientific and technological innovations, that's going to be transformative for everyone.
I'll assert that 99% of people don't have any application for personal assistants, outside of work (perhaps). Most of us aren't so buried in personal obligations that the value of a half-assed "assistant" (which might violate our privacy) is worth the tradeoff in complexity or security.
The pitches for Dots, Grok Bot, etc., have been spectacularly uncompelling for me as a civilian. I don't need or want a bot to book my flights or read my inbox. I'm not so busy that these things have a value high enough to be worth the effort.
As a worker or an entrepreneur...maybe. Depends on the task, and how many other people were flinging slop at me on a daily basis. I could see the value of a calendar bot, or an expense reporting bot or a bullshit paperwork bot or a slop-return (slop jai alai?) bot, but there's no way in the world I'd give a bot access to the information necessary to, say, do my personal taxes.
i'd argue any parent, especially of a young child, could use 20 personal assistants lol. So that's just false. There's SO many appointments to keep track of in baby's first year of life, and that's if you have a healthy baby.
And once they start school there's an endless amount of deadlines and "reminders". A personal assistant that can automatically keep your reminders + calendar up to date based on your inbox would be very nice.
It gets even more fun if you have an aging parent at the same time as you have young kids. Then you get double the appointments to scheduled, coordinate and keep track of.
So when you say people don't need a personal assistant, i feel like you're living on a different planet. I suppose when i was single things were more "simple". Your mileage may vary of course...
> I'd argue any parent, especially of a young child, could use 20 personal assistants lol. So that's just false. There's SO many appointments to keep track of in baby's first year of life, and that's if you have a healthy baby.
I understand the need. I have a calendar. I put appointments in it. It provides alerts. What value does having a lossy/inaccurate/fragile robot in the middle give me? Data entry?
Now look...if I were in a position of power with literally everyone in my life, and I could somehow justify the social cost of requiring that they interact with a robot to negotiate demands on my time (the actual hard part), then maybe there would be additional value. Some people can do this! I'm sure one must deal with Mark Zuckerberg's PA to schedule his time [1]. It is very nice indeed when you are the kind of person who can defer your supplicants to a representative who manages the hoi polloi for you. If you can delegate these irritations to a robot, well, so much the better, I suppose.
If I tried this, as not-Mark-Zuckerberg, I would be what normal people refer to as "an asshole". I would also like to see you try this with your child's teacher, or the PTA, or doctor, or...
The response you will receive, an alarmingly high percentage of the time, will rhyme with "buck off". This stuff reminds me a lot of how valley folks keep trying to push spy glasses on the proles (silly normies never understand the genius of that technology, either...)
> And once they start school there's an endless amount of deadlines and "reminders". A personal assistant that can automatically keep your reminders + calendar up to date based on your inbox would be very nice.
Let me introduce you to...Gmail, from like, ten years ago. It looks for appointments in my mail and put them in my calendar. Somewhat to the point, it is often wrong, requiring more work to correct than it would to just enter the data myself.
I have been kind of disappointed by AI science-wise. I was anticipating more advancement. Experiments do take time, so I suppose they could be pending.
AI really struggles with "collision with reality". Computers suck at modeling the real world in a detailed enough way to make scientific progress, aside from small niches like drug discovery where you have a very well scoped problem. If any part of the try something -> test it -> look at the results loop requires hitting the real world, AI is going to struggle.
On the other hand, the areas where it doesn't need to hit the real world, progress is absolutely flying. I would not be surprised if AI is superhuman in math and physics in a year, considering we went from homework helper to IMO Gold to Millennium problem solving in a couple years time. I think we'll start to see more meaningful scientific progress in maybe 2-3 years when scientists have access to an oracle that can formulate theories and direct real-world research better than a human can.
I agree and I think the question is then, how much economically valuable work exists in a closed world? My hunch is less than people think. Finance can't rely on LLMs because its adversarial we can't all be listening to the same oracle and evidence of LLMs being any good at trading are nil. And then lawyers, as per usual, will simply regulate their job security into existence. Of course, AI feels amazing to Silicon Valley / Big Tech, software is the product. And often its software you sell to other people making software. But outside of that? Useful but lots of landmines.
People have to come to grips with the possibility that a technology can be extremely impressive but be a bust in utility. I'm not saying that's what AI is, I'm saying that nobody actually knows.
I don't think his point is that everyone should be using AI, it's that if you haven't used the agentic tools with a frontier model in the past month or so, you have no frame of reference for the current level of capabilities. Most people only use the chat interface with the free-tier models and based on that experience they're making strong conclusions about where AI is headed (it's still just fancy autocomplete, the whole thing is a scam or a bubble, etc.).
Fair assessment. Maybe the messaging should focus on “these are great accelerators for software developers” rather than “AI will change everything for everyone everywhere…” It is understandable that non-technical folks are kind of underwhelmed - not due to lack of understanding so much as lack of a tangible need. Not everyone needs an electron microscope or gas chromatograph…
At a certain point the offhand hints at exponentials in the narratives are going to have to come into contact with reality.
There's been this continual allusion to ever-narrowing circles of influence and ever-accelerating capabilities behind the curtain, but in fact things do keep coming out from behind said curtain and while their capability is indeed advancing it doesn't really seem at all like it's accelerating, even, let alone vertical.
Also at what point do we raise the fact that despite the ever narrowing circles of access, nobody within those circles seems to be able to actually do anything with it that's ostensibly what it's sold to do? All they seem to be able to do frankly is build more models and harnesses and tools designed to consume more tokens from the models they build, but not actual directly useful new or innovative software that generates what could be very real revenue.
Eventually when you keep talking about exponentials but not that much seems to be demonstrably changing, the narrative is going to run out of steam.
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
Really? That's not my experience. In fact, I've experimented quite a bit with spec-driven design where the LLM does not just get a prompt, but an (informal) spec, often with design considerations included. And yet, regardless of the power of the model, I've never seen them autonomously deliver what I'd consider a finished product.
There are always edge cases that are not properly taken into account, architectural oopsies, performance issues, duplicated code, and other problems, that then need to be identified and fixed, unless it's throwaway software (e.g. a one-off script or some quick experiment).
This is not to say that agents aren't extremely powerful (as I think they are, they do often blow my mind) but the prompt-and-forget approach is IME not a productive use of them for software that is meant to last. This may change in the future, but for now, fully or largely autonomous agentic work does not seem to be the road to quality.
I feel like a lot of issues you mention will eventually point to an ambiguous spec. I’ve had great success with interview style spec generation where I ask agent to grill me on any and all parts of the spec/plan that’s ambiguous and then note my responses. It takes like 10 mins but the end product is always exactly what I expect and most of the things you mention get resolved in this interview phase.
All informal specs are of course to some extent ambiguous. (I should mention that I have a background in formal methods and have also worked with formal specification tools.) And what you suggest is a process that I have used also, and it will cover some obvious gaps, but not all, at least for non-toy projects.
If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.
But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.
The irony is the closer you get to formalized speculation the more it looks like you are back to just writing deterministic code. English language is just a crappy language for this compared to programming languages.
I generally agree with the prevailing commentary here, i.e. that automating app development and maths proofs may not be as world-changing as the Artificial Intelligentsia purport / “fear” it to be.
But also, upon more reflection, it occurred to me that if we’re able to create a competent humanoid robot, that could legitimately be world changing in some good and other very bad ways. The scary part is that I don’t believe there’s any major physical / energetic barrier to this. The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion contols; the intelligence. The part that is experiencing breakthroughs at an accelerating pace.
What if a highly intelligent and physically competent robot were available for the price of a cheap car, say ~$20k? What if that robot were competent enough to cook, clean, assist the elderly, move things around, assemble and disassemble things, etc? Billions of jobs would be displaced, or hit with a very concrete maximum wage.
Many white collar jobs, while still at risk for displacement as well, benefit from barriers like IT integration and data privacy hurdles, and the fact that corporate higher-ups simply prefer to communicate with humans.
But most blue collar / manual labor jobs have almost the opposite condition: humans introduce risk, and generally don’t enhance anybody’s “experience” - see people’s somewhat odd acceptance of higher fares for a Waymo vs an Uber. [1]
I don’t think we’re there yet, but the fact that we are rapidly accelerating our trajectory toward that world should be at the least, unsettling.
I think the "humanoid robot replacing all blue collar jobs" fear understates the amount of real world (i.e. visual, physical, sensory) data and tacit knowledge involved in many such jobs.
Driving shows it can be done, but even there it's taken years of highly directed research and investment, and driving (at least where Waymo operates) is still pretty well defined compared to plumber.
Such data can be gathered but the current methods of RLHF by paying white collar experts, accessing private datasets, etc. don't seem to really get at this.
I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this purported widespread unemployment without major political consequences.
> understates the amount of real world (i.e. visual, physical, sensory) data and tacit knowledge
I agree the data trove for these is not nearly as comprehensive as “the entire internet” was for LLMs. But LLMs paved the way - people are building the datasets rapidly now. I suspect this will look a lot like LLM rollout looked over the past 5-ish years. Unusable garbage… less unusable garbage… then suddenly, a fully capable model like Opus 5.5 or the like.
And the “problem” with the way the foundational models seem to work is that they don’t just acquire individual skills one at a time like we do - they go from knowing nothing to knowing everything.
> I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this
This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.
> This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.
I get the fear, but at that point we're basically imagining that Terminator is a documentary. Given that the US has more civilian guns than people, and that private ownership of military grade weaponry is strictly controlled, I don't think street wars between private robot armies and folks in the bread line is a realistic outcome.
And to be honest this is also the sad part. It has become conceivable that we may at some point solve work. In theory: amazing! However it remains inconceivable that we have solved sharing between humans. So there is this feeling of looming doom. In some sense, if you cannot provide shareholder value, you have no value.
> The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion controls; the intelligence.
If that's all it is then shouldn't the mass produced robots that are posted on r/singularity and such places just be remote operated by people in low wage countries? Couldn't the Amazon package sorter robot just be a remote operated bot instead of paying US wages? Why aren't they selling them with a cheap operator subscription? This seems like a clear product direction if the robots were really capable today.
I agree with this idea very much. I think that the magic moment will be when a humanoid robot can fully repair another humanoid robot.
If it can do all the other stuff that you're describing plus partial self-repair / full repair of another robot it can, with a supply of spare parts, with enough time and access to raw materials fully self replicate.
Anyone who owns those machines has a tremendous opportunity to build whatever they want.
I must be the biggest failure, because I understand what AI is capable of, know how to use it, but have no idea how to profit from it without selling my time.
My interpretation of this is something like: if LLMs are to be mega useful token counts need to increase by many orders of magnitude -> broad adoption (and spending) would require token costs to drop by many orders of magnitude -> before the common folk get mega useful tools existing GPUs will be worthless
About a year ago, Google’s AI was a joke - comically horrible. Basic questions were answered with crap. Others were better but seemed like fancy toys at best.
My $DAYJOB pushed people to start using AI - frontier models.
I asked it to review our existing code for bugs. Found some, all valid - many that surprised me.
Later I asked it to develop a custom niche thing, that has a strongly unique tooling and environment that we had made. Entire prompt was only 3 or 4 sentences. Had I given a new senior engineer that task with so little context, I’d clearly be an utter asshole.
I really expected the AI to fail. Instead, it did everything pretty much exactly how I would have - and the result worked on the first try. Except it only took a few minutes to do what would have taken me a couple of hours.
The near-term gains are going to be huge.
Long term, this will make the impact of Industrialization seem insignificant by comparison.
It is always a mistake to assume that because computers can do X, that they will be able do everything that a human that can do X is expected to be able to do.
Solving frontier math problems is mind boggling, but perhaps it says more about frontier math problems than it does about LLMs. Same with chess which was solved decades ago.
My experience is that LLMs can be useful but are underwhelming in terms of intelligence, namely the ability to quickly learn from experience and grow with you while you accomplish a task.
Thinking of them as beings is something we should move away from.
Why wouldn't they be able to grow, its just one data point more in their training, in your notes or in their instructions.
It doesn't matter if they are beings or not. They will replace most humans in the offices like machines replaced most humans in fields or industrial plants.
Because context, at least in my experience, doesn't seem to encapsulate the sort of dynamism that I'm thinking of. Conversations are flat and repetitive. It still feels like a very good semantic search more than talking to an intelligent human.
I don't know if they will replace most humans the way you're imagining. I think the industry has done a very good job of selling this point, but that they are still quite far from this.
Because we don't let them learn. Everyone runs the same static generalist models, instead of having every model instance learn from everything it does. New models come out occasionally, but they have not been trained on the private data you have not shared with the AI company.
You can work around that to some extent with explicit instructions, but it doesn't scale nearly as well as actual model training.
> see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt
Really? I start with a conversation for maybe 5 turns or so, where I ask it what would need to change, what the API might be, any database schema changes, URL schemes, and so on, and finally ask it to break it down into commits. Then I let it go, implementing 3-10 commits at a time via subagents. It usually gets the UI somewhat wrong, so there are followups to fix it. This is with Sol and Luna subagents.
Yes, and this is completely normal. Not sure I get the point of his post. Most likely, stating the obvious is what you gotto do to keep your followers "engaged".
The average person does not have the time to occupy themselves with a new thing to see what it can maybe do for them if they combine it with hundreds of others.
Until non-chat-bot AI comes in discrete usable packages, mass adoption will be limited, because it's all just a big bunch of lego pieces whose utility is hard to grasp, because it isn't self-evident.
This isn't new, though. It's how "technology" usually works. For example, human(oid)s have been able to stick sticks into things for ~500k years, but the fork wasn't "invented" until ~2-4k years ago.
AI is a background technology, a building block. Something that is best appreciated when you don't know it's there -- when it just does a job (well). The end-in-itself POV of the tech industry is self-indulgent and irrelevant for the average person.
I think "moment" significantly undersells this. Frankly, I wonder if this isn't setting vast swaths of the population on fundamentally different paths.
Currently the difference in life outcomes between people who know how to use computers well and those who don't is small enough that in everyday life we just ignore it. But it's greater than, say, the difference between people who took advantage of refrigeration and those who stuck with ice boxes, or those who learned how to drive a car and those who didn't.
And the difference with AI is going to be even greater. And it's possible that right now people are choosing a life course that will benefit or disadvantage them long term.
How does one who doesn’t have an employer funding their massive token budgets become “good at AI” the way that Karpathy describes here?
I think the point is that this gap exists partially because of the access that a select few have today, which enables them to do things that the have-nots can’t fathom.
It's actually sad how much AI has either not improved life at all or actively made it worse from the perspective of the average person:
Your car is still the same.
Your dishwasher is still the same.
The train you take to work is still the same and never comes on time.
Fuel is more expensive.
The roads are still congested with traffic.
Your kids are (probably) doing worse at school.
Food costs more.
Houses cost more.
Rent is higher.
Buying a computer or a PS5 is more expensive.
Politics is still full of mentally unstable people.
The environment is getting worse.
You will still die of heart disease or cancer.
Food quality is worse.
Your kids can't find work.
Wealth inequality is accelerating.
But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
This is the core hubris of everyone behind the LLM craze. I do not want to deny their utility there absolutely is some there. But the inherent assumption behind the whole thing is that with a "country of geniuses" problems will melt away. But that's not the case. The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
And the LLMs exist to further those problems. People can't afford houses? Well, now they won't be able to own computers either. That's a feature, not a side effect. Once the K-shaped economy progresses enough, you won't even be allowed to use the LLMs. They'll exist only to be a tool for the new feudal class who owns you.
Luckily, the AI bubble will destroy a decent chunk of their wealth.
The Gartner Hype Cycle [0] is real, and ubiquitous. We are teetering somewhere near the top of the Peak of Inflated Expectations, about to plunge into the Trough of Disillusionment. A lot of very rich people are about to lose a lot of money, and we can only hope that they don't get bailed out (again).
Then, once the bubble and hype is over, the long, slow work of actually doing useful things with this new tech starts. That's when we'll see it affect everything else.
> We are teetering somewhere near the top of the Peak of Inflated Expectations
I've been hearing this for a year and a half now. That word, "ubiquitous", is getting real mighty challenged. I have tried to insulate myself as best as I could and it just never happens and every other day I wonder if my foil hat needs adjusting. Does yours?
> Luckily, the AI bubble will destroy a decent chunk of their wealth.
Even if all this is true (personally I doubt it), it assumes the bubble won’t burst before they’ve shifted the money from all our 401ks, IRAs and index funds into their own pockets.
Once capabilities catch up with the demand (~which is in few years - it just takes time to build!) average person will have much better access to higher quality and cheaper hardware. It's literally how it has always worked, I don't understand why would it be different this time. There's zero substance in your comment, just repeating tired tropes from twitter.
Companies have been trying to figure out how to get people to rent pcs for a while. Unfortunately the AI boom has shown them how, buy out all the capacity of the ram manufacturers, pay them enough consistently and the manufacturers have no reason to expand their capacity. No cheap ram means companies and then home users will have no choice but to come to you and rent their pc.
A market requires scarcity, because scarcity is what neccesitates trade.
You can't invoke market mechanisms to defend a post-scarcity scenario.
Why would the ai owners make things cheaper or even available at all once their abundance machines give them everything they want?
What would they get out of if it? What could ordinary people trade for it that their abundance machine can't give them already and that would be worth giving up their monopoly on productive force?
You can't argue "this will change everything" and "this is how it's always worked" simultaneously, and yet AI accelerationists bring out both to defend their ideology.
You don't need a back room filled with cigar smoke where supervillains coordinate their plans while twirling their mustaches when good ol' class interest does the same job.
The problem isn't that it fails to unwind, the problem is what the unwind looks like. Last time it meant global upheaval and world wide capital incineration -- mostly metaphorical in the case of the US but very literal in the case of Europe and Asia. The poor suffer more in real terms but capital suffers more in nominal terms and that unwinds inequality and the burden of furnishing runaway capital with the runaway returns it demands in excess of economic growth.
I'm sure the same "strategy" will work again, but this time around everyone has nukes and drone swarms and AI-equipped bio labs so shit's gonna be lit.
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
Democrats expanded SNAP, Republicans shrank it. Dems expanded Medicaid, Republican shrank it. Dems protected reproductive rights, Republicans repealed them. Dems expanded title IX protection to LGBTQ+, Replicans eliminated them. Republicans jailed 1,200 January 6th insurrectionists, Republicans pardoned them. Democrats provide non-partisan federal emergency aid, Republicans withhold federal aid from blue states. Dems establish tax credits for electric cars, increase automotive standards. Dems didn't kill of foreign aid to need countries.
If you aren't female (reproductive rights), LGBTQ+, an immigrant now hunted by ice, someone in need of SNAP, Medicaid, or emergency aid in a blue state, it might feel like both are the same I guess. If you are one of the above, and now suffer because so many voted based on 'both are the same', you very much feel a difference today than from the US under Biden. If you were one of America's allies but are now constantly attacked, you feel a difference today. If you are Ukraine, you feel a difference today.
This is broadly the same people who bemoaned the idea that the "smartest people in a generation" were all just working on adtech without any reflection about maybe that's not actually an exclusive set of the smartest people... or that a lot of the people in that set were the same one who might, twenty years ago, otherwise be chasing money as middlemen in finance vs doing productivity-expanding things anyway...
You can be smart and chase the money, but it's crazy hubris to assume that only your peer set of money-chasers are smart.
It's a small leap from there to "since we're the only smart people, we should get to have the most influence."
I’ve heard people say variations of, “Clever people can easily convince themselves of just about anything, if they’re inclined to. And then it’s nearly impossible to talk them out of it.”
Kinda seems true. And the worst people in history often don’t seem super dumb, despite their various other defects.
been a while since ive see the real talk on hacker news. its refreshing. usually I feel like I'm the only one among nerds asking to hold people accountable.
For a lack of deep knowledge in this area, I don't think I have answers for this, but if one is interested in working on this problem, my approach would be to start with diving deeply into scientific literature about psychology and sociology to find out
> Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
People support policies that make housing much less affordable (e.g. rent controls) and oppose policies that would make housing more affordable (e.g. zoning reforms). Saying that it's evil politicians that cause this crisis is dishonest - it's what people want.
People by and large don’t understand the second and third order effects of their policy preferences. It’s not that they don’t want those things, it’s that the world is more complicated than their mental model insists it is. Politicians could try harder to fix the fundamental problems but they don’t because the long-term solutions don’t manifest until after the next N (very expensive) election cycles.
Rent controls work if you go radical enough and enforce a max profit margin for the whole city, with rent limited to something like 30% over maintenance + debt repayment costs.
The amount of housing in the city won't grow but it will free up a big part of peoples income that would otherwise go to landlords. That helps the local economy with more consumer spending, and makes it easier to save up or apply for a loan so you can start a business.
You get a better economy and more money in the average persons wallet without raising taxes. The only ones who lose are commercial landlords, and they will stay profitable by design so it's easy to shut down any complaints. And you get better quality apartments as a side effect because the only way to get more money out of existing properties is to spend more on fixing them.
The politicians are those who created the respecting laws.
"Convincing the masses to send these politicians to hell" is thus nearly isomorphic to making the masses hate politicians who create such laws, which at least very likely implies that by convincing the masses to send these politicians to hell, a different mindset emerges in the masses.
There is no evidence that reducing rent makes the rent go up - only fearmongering. It's a stopgap measure, though, since there still won't be enough housing to meet demand unless you also fix that at the same time.
For that you might need the kind of geniuses who write stirring books about principles and values. Or whatever the modern equivalent of those books is. Maybe nobody would pay attention to that kind of advocacy in the present day, not sure why. But it's not tech geniuses, anyway. Much though tech has power as a catalyst.
I suppose there was Project 2025, and there was that book about the inevitability of Russian hegemony (or empire or something) that Putin liked. Those got attention with influential people. Not sure what to conclude from that.
> because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell
I could talk about this for hours, and was last night actually.
What do vanguardist communism, feudalism, liberal democracy, and fascism have in common? A shared belief that the general population is composed of a bunch of dumb dumbs that need super smart (read: politically educated) people to "lead" them.
Seems subjective. When is the smartness, big smart enough to allow someone to enter the hallowed ranks of "actually a real human allowed to speak for the masses?"
There does seem to be a clear delineation between the people you hear about (the vanguard in our example), and the people you don't hear about (the masses): willingness to create confrontation. Most people want to just go with the flow, live in peace. In a dichotomy I've been calling "choose peace or violence," they choose peace. They choose to see bad things in the world or their life and focus on managing their happiness around it.
I think that this is the default human mode of existence, and that this mode lacks defenses against people willing to violate the default peace. Imagine a bagel line, and someone cutting that line, and when tapped on the shoulder, the cutter breaking into a huge tantrum, and people in line backing down with a "Jesus! It's not worth it, fine." Thus some people learn that they can be much more selfish in life if they're willing to create confrontation and disturb the peace.
When I talk to people about agitation, civil disobedience, or even just unionizing, the most common thing I hear is anxiety. What if the situation gets worse, what if there's reprisals, what if I lose this little bit of comfort that I've managed to be happy with? That slice could get smaller and smaller but people aren't willing to take the step to say "enough is enough" because I think it fundamentally goes against human nature, we have to sacrifice our own happiness on some level to put our foot down and demand or make change. At minimum we have to acknowledge the discomfort or injustice, which is a diversion from our constructed internal peace.
Personally I think people should be allowed to live in peace. Maybe the liberal democrats and vanguardist communists are right, maybe most people shouldn't need to trouble themselves with the greater direction of their society. However I think the stability this promises is false; in fact I think all stability is false. I think this constructed peace that people are afraid to disrupt clearly can be torn apart at any moment, not just by people willing to create conflict, but by a literal earthquake, or fire, or plague. Therefore I think there's a couple societies in history that got it right: ones who distributed the responsibility of society across all individuals in the society; namely the brief period of syndicalized Spain, and the several hundred thousand years of certain tribes of native Americans on the eastern portion of the continent. There may be other societies in history, these are just the cases I've been able to read about so far.
It seems to me that it's possible and necessary for all people to be ready to wake out of their constructed peace long enough to make adjustments together to maintain it. I don't think this is the same thing as them being "led out of it," I think this is each individual being on some level alert to the need to spontaneously say "no, here we wait in line for our bagels. No, screaming about it won't help you, get to the back of the line. Look around: everyone is laughing at how silly you look."
What I like to call at work "this is not a technical problem". There are many problems we could easily solve if we had the right structural incentives. But instead we have politics, processes, layers of management, etc. All that doesn't disappear (yet?) with AI.
I believe that’s actually one of the arguments of the boosters. Utopia will finally be reachable when we get all the human drama and bureaucracy out of the way
I’m not sold, but I’ve heard this line of thinking many times
Human drama is what people use to advocate for themselves. Bureaucracy is what people use to protect themselves. Then you have the side effects but, you may also not like a world without those at all.
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
When I asked why Musk doesn't put his money towards non-profits my Musk supporting friends say it's because he "doesn't believe anything but for-profit entities can solve the world's problems".
So why isn't he solving homelessness, famines, war, etc. with capitalism?
To add insult to injury, we're being told: "Just trust us bro, however we make things worse now will totes be redeemed as the prophesied god-mind magically fixes it later."
Strangely, they never discuss the possibility that Multivac might seize all their wealth and cast them down into average-ness. What if it actually comes about, but then it tells us to do the things we already refuse to do?
I'm immensely sad every day seeing all these posts talking about billions and trillions of dollars and where I live (part of the US btw) we don't even have fully reliable electricity and water access. In fact our electricity bills have increased and there is nothing to show for it.
I don't understand why these big labs aren't doing more to improve the electricity situation. Dump a billion dollars worth of tokens into fusion analysis, research, and simulations and try to force a breakthrough. You could buy a lot of good will with unlimited cheap energy that liberates us from fossil fuel dependencies.
Mathematics and programming is verifiable and fully controllable in a digital environment.
Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
Look at all the roadblocks for data centre buildouts which the industry certainly has a financial incentive to complete as fast as possible.
> Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
When the tokens are hooked up to physical devices (robotics of various sorts), then the tokens pay for themselves by being physical workers, or human wage replacement.
And a robocop will follow whatever is passed for law, damn the masses who might vote otherwise. And subjugation will be enforced by token, LLM rule, and machine. They have no union or ethics to complain to.
Because they increased the demand so much they are seeking out their own supply. Nuclear has consistently been panned for decades because it's more expensive than other power generation.
Quoting startup like that makes the effort seem like it is an extremely recent venture.
It is 13 years old.
It has been making progress, the degree and date of success is debated and confidently claimed to be in many mutually exclusive states, which at the very least says that there are lots of people confidently making claims that are not true. I'm not going to join that fray to declare which of them is correct.
Are you avoiding all use of fossil fuel while alternatives become available?
It is a bold claim to make but not something that usualy be done. Any abrupt halting of fossil fuel would severely impare the ability to develop and produce alternatives.
We need to transition to sustainable methods. If you want to go cold turkey, go ahead and cease driving an ICE, never travell by air or have anything delivered (and that's just the tip of the iceberg), it would be a counterproductive way to reach sustainability, that is your right, but you can decide to do that to yourself. You don't get to decide how others reach sustainability.
There is a difference between advocating for a better world and acting like a bully because you don't like the current world.
Electricity infra is not a technical problem. It’s been solved for decades. It’s a political problem, and both sides of the isle have had plenty to do with getting in the way
This is sort of my take. You could look around in 2016 and 2026 and honestly not much has changed... at least at first glance. There is a missing piece of the puzzle.
I'm noticing this even in industry. Yea, we can build and iterate on ideas 100x faster but has this turned into anything meaningful? Not really... Hell, look at the big companies spending billions on inference... nothing shipped and things are still buggy as ever. Where is everything?
How many more software projects are there really left that make economic sense? Zuck learned the hard way re. metaverse.
Not many - the set that make ECONOMIC sense is finite. Even though many want to believe its infinite because all of a sudden everyone can vibe code an app cringe.
If anything AI is going to cause an insane amount of consolidation - what used to be across 3 diff apps, that can be consolidated will be done -> many businesses will die. It will be painful.
Google was attacked by an activist investor pre LLM hype for being bloated - they promptly laid people off. Many software firms have been allowed to bloat up do to the nature of the business.
Not sure I can agree, but the idea is interesting.
While the set is finite, software comes and goes. Just look at the tools to build software over the years.
> Many software firms have been allowed to bloat up do to the nature of the business.
I can agree with this. It will be harder to justify engineers sitting on laurels. Google could afford it before but I can see how it will be under more scrutiny now.
We went from spending a lot of money on people to enshitify products and then charge more for them to spending a lot of money on agents to enshitify products to charge more for them.
Why even bother when an AI is just going to do it better than you can? They have more money, more compute, and most importantly, will have next to zero cost of labor.
It has greatly changed for the worse
I can't get all the flashing lights out of my face now, more algorithms to hook you in, more social media, more gambling, more news, more things to catch your attention.
* Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
* Less junior hiring and suppressed wages
* Culture of uncertainty and fear
* Computers are really expensive now
* Some people now live with diesel generators and gas turbines making their neighborhood hazardous to their health
Totally agree. IMO it's mostly behind the scenes stuff though.
I think some of the most impressive stuff to show someone from 2016 might be LLMs doing a summary of complex topics, followed by self-driving (actually) cars and intricate robots... but other than that there is no "future magic" touching the lives of every day people going on.
Actually I'm pretty sure if you showed someone the latest and greatest from these big AI companies, these AI companions that can book flights, people would go "Oh, so its like a smarter Siri" and proceed to not care at all lol.
> Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
Gosh, this has been my entire experience with AI, and I’ve never seen it this well articulated. Thank you!
Less junior hiring is a reflection of a) the maturity of the software business b) current expectations of the future
I have no doubt startups in particular are able to get away with using LLMs to produce sloppy code and ship. But this trend was on-going and sped up due to the infiltration of LLMs and marketing.
You do realise markets are highly imperfect in reality? That nonsense in econ books is fantasy.
Even in software, there are many things that haven't changed.
e.g. Why do Rust and CPython still have over 5,000 open issues each? At some point, will the superintelligence fix all the bugs autonomously? When is that going to happen?
What about new features? In every thread about Rust, people are clamoring for new features and libraries. People wonder why there can't be a batteries-included standard library like Go.
So now that LLMs are so smart, is that a solved problem? Or are we still bottlenecked on the expertise of a few experts?
---
Also, what happened to AI's passing the Turing test? I see tons of slop articles that are obvious to most HN readers. They haven't fooled us apparently -- the longer the interaction, the more obvious it becomes whether it's AI.
(That said, I use AI all the time and find it to be a great tool. IMO is is a tool, not an autonomous intelligence with a will of its own.)
What is up with this attitude in this thread? AI has taken off programming basically this year, the end of last at best. The first truly capable LLM, ChatGPT, isn't even 4 years old yet.
Just a crazy amount of doom and gloom all around this thread.
It's kind of weird to say it's "taken off" recently. Yes what we have today is far better than half a year ago and solving new real use cases, but that was far better with more uses than we had half a year before that, etc, for years now. On their best usecases LLMs were already enormously useful years ago, and it's still very spiky.
As a user of software I see no improvements. What I see is that I need to download software more often, it brings more features I don't need and it has been as buggy as ever (if not more). And I can say that about Operating System I use, social networks, games, etc.
Exactly. Where are the results? If AI is really producing value at 20X the rate software engineers produced it earlier, we should be seeing 20X the number of software projects released now. Or they should be getting made 20X faster. Or they should be 20X better. Or the projects should have 20X fewer bugs. But none of this seems to be happening. The only thing that seems to be happening is we're making the same crappy shit, but with 20X fewer people.
In other words, it's taken a highly enjoyable and lucrative vocation from vast swathes of workers - a lot of which make up this sites audience! - and instead concentrated more productive power in the hands of a few megacorps.
It has made it far more accessible to a lot more people. Being “highly enjoyable” and “lucrative” is why people try to gatekeep it.
The fact that the means of this capability is in a few hands does suck. But I’m increasingly optimistic that it’s going to be commoditized through open weight models and performance improvements. There is enormous demand for innovations on these fronts.
I'm a rare individual that has consistently made choices to earn less money in order to have more enjoyment. I would clock fewer hours than they wanted me to during college for my internships, I left a FAANG to go back to startups, I quit a startup to work on volunteer work and try to bootstrap a business.
At multiple jobs I've noted that I felt the pay should have been less. Not so that the employers could take a bigger slice of the pie (in fact, my preference is to be in an anarchist co-op, but I understand that's a wild take and might be terrible in practice). My hope was that we could filter for people that genuinely want to do the work and take on fewer people that were there for the money. At the same time more money is more better. And I like having money. It's complicated.
As I've said many times before, I didn't come to Silicon Valley to get paid. I came here to make computers do cool things alongside other people. I need income to do that, but I try to make sure I'm in a situation where the income isn't something I need to think about all of the time. If you're doing mercenary work that's not possible.
I think being able to accomplish more from places that are a fraction the cost of Silicon Valley will also help achieve the goal of doing things for reasons other than the money.
I think the amount of protectionism we’ll see could be absolutely enormous. Not only do incumbent corporations not want to lose share, incumbent countries won’t want to share either.
Definitely. For now I'm making friends with lots of very capable people - it's not as easy to do that outside of a hub like SF.
> I think the amount of protectionism we’ll see could be absolutely enormous because corporations aren’t capable of altruistic attitudes.
On some fronts they will be powerless to stop the changes. I look forward to drinking the tears of IP owners when breaking through their DRM and recovering their source code is just a matter of sharing a model ID, prompt, and hyper parameters. Reverse engineer NVidia's firmware and drivers. Decompile and port Nintendo's latest console's operating system. We may soon find ourselves in a world where tasks like that are achievable in a few days on consumer hardware.
I think this is related to the structures we’ve built in society. It’s lower friction to write software than just about any of the problems you listed.
It reminds me of the Ezra Klein quote that “OpenAl would need permits to cover its parking lot in solar panels, but it can accelerate into recursive self-improvement, as best I can tell, whenever it so chooses."
A lot of it isn't low friction really just in terms of political will it is high friction. E.g. housing issues are due to failing to upzone to meet demand appropriately. Solving housing is therefore free. It could be done today in any city in a housing crisis with the stroke of a pen, and before long, the market will equilibriate at a level better afforded by the median wage earner in the economy. The decision not to do this is a deliberate one: prices increasing is very much the goal of this sort of policymaking.
Likewise when you see other such shortcomings in our world, ask what this incentivizes for otherwise. Then things will become obvious why they are how they are, from housing, to transit, to feeding people, to healthcare.
This is a Western-centric viewpoint. I'm in Asia visiting a few different countries right now, including the big ones, and much has changed with AI and also in general over the past 10 years with cars, trains etc.
> AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Strange viewpoint. In mathematics any novel result is useful as it expands the scope of other results. Who are these mathematicians who didn't want AI to solve the problems it did? I see trepidation about the future of the field but that is not equivalent to not wanting them to be solved.
Terence Tao and the apparently newly formed Association For Human Mathematics. The post is online-famous now for the line "Mathematicians did not ask for this work to be done."
When I was a young man reading gauss, and thinking before I turned the page, I see a result here, something meaningful! and then I turn the page, and of course it is there. There is no shame in this.
A long time ago, another student of mathematics described it to me as climbing a mountain. No single step is hard, but you must make many steps to get to the point that no one else has made a step.
I've watched some videos from Chinese commentary on AI usage in the industry,
Some examples are finance people directly asking for API access now so that they can make their own tools. These tools used to be maintained by teams of developers.
Many business making their own customized scripts to make their tasks faster.
From an economic point of view, it seems like there are some productivity increases, but I don't think these can be concluded for a while.
The rational thinking rarely account for second order effects.
Although business leaders rarely care about second order effects, a worse business is not immediately noticeable, compared to hiring cuts and the money it saves
No need to trust me, just look at how China was 10 or 15 years ago to now. Beijing air pollution, more high speed rail than the rest of the world combined, electrification of vehicles and economy, etc.
That is not how math works. There are reasons why the Riemann integral is pervasive and the Labague integral is rarely used unless needed. Just because you can do something does not mean that you should.
> Just because you can do something does not mean that you should.
Again, strange viewpoint in the field of mathematics of all places, unlike, say, economics where doing something one shouldn't has disastrous real world consequences. Until it's discovered and proven, one doesn't know what would be valuable and what would not. I bet your sentence would not be said about humans trying to solve problems, no one would tell them they "shouldn't."
That pretty much sums up my experience. Everything is more expensive and worse at the same time. And it feels like we're being screwed over from every direction and in new ways every day. People keep talking about this tech is supposed to do so much but everything just gets worse.
It's not true that everything is more expensive. Nominal prices are higher, but incomes have risen faster than inflation. Purchasing power of the average person in 2026 is higher than it was in 2016.
As for things getting worse, I guess it depends on the domain. In hardware, things have gotten a lot better; my desktop computer is orders of magnitude more powerful than I would have expected 10 or 20 years ago. In software, I do see a lot of bloated software being written, but I don't feel that the software that I personally use has gotten worse.
Average, or median? Don’t know the answer offhand, but there are a small number of well-known people who are likely to skew the average quite a bit upward, given how much wealth they accumulated during and since covid
I think people in the LLM space are being far too impatient with the general public and their adoption. And I think they need to start thinking harder about building products that genuinely enrich people's lives. Look at how long home computing took to establish itself with the public, then the internet, and finally social media. None of these trends happened overnight, and some were accelerated by other technological advances (social media-> smartphones).
The adoption of these technologies spurned ton of innovation in these spaces, and I feel like with LLMs its the first time I've seen genuine frustration from the industry directed towards users for not adopting the technology fast enough. This attitude did not exist in the 1980s, it didn't exist in the 90s, and it didn't exist in the 00s/early 10s either.
Do you blame them for getting impatient? They’re burning cash and quickly running to a point of never being able to justify it. The hardware they buy will be replaced with even more expensive hardware before they start to make a profit. It’s not sustainable and the market hasn’t figured that out yet. At least they’re starting to question it.
Whether you like it or not - technology causes compression of life cycles. You have a hit product? Great - I expect you to have crazy high growth rate. That’s the expectation of a world that is so deeply interconnected.
Consumer-ready LLMs have been widely available to the public now nearly 4 years if you start counting at GPT 3.5. There's certainly some compression here but Karpathy does a good job of outlining how silo'd those experiences are: the bulk of the product penetration (based on headcount rather than token usage) is "derpy chatbots" (his words, not mine). Most of the actual token spend is likely done on the STEM side. We're talking about a technology today that currently only has significant penetration with <25M users.
People need to be patient and put in the work to build great products, rather than lament its slower-than-desired (albeit still quite fast) adoption curve.
At least for me, personal agents have been quite useful. If you integrate them well - shopping, filling random government forms, keeping track of my personal life, optimizing training plan and nutrition, planning vacations, etc. I can do more home repairs or renovations myself with help of chat
So basically you get personal digital assistant - very useful - but not changing my life.
What would I need for real transformation though? Food, house cleaning, commute - "every day chores". Even if we had ASI tomorrow, what could it do to change my life, besides taking away my job? Design good house robot to help me with food preparation I guess... Maybe finally self driving cars launched in my area? Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
This is the kind of gap in shared understanding that is widening.
When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
The incentives seem to be lined up to push users to the frontier of "how much can the harness around the model do and claim without most people pushing back on the claims."
The bigger the pile of stuff you have the tool create unsupervised, the more incomplete/inaccurate the overall picture of it is gonna be. That doesn't matter for most hobbyist/helper-tool/game-port/etc type of tasks.
I get a ton of value out of the tools and yet part of that value has come from being hyper-aware of the shape of the limitations, which changes much less and more slowly than the specific of the limitations.
"How'd you notice that one!?" - well, it helps to be the only person on the team who took the time to interrogate the e2e interaction of all the code...
The median piece of OSS software was already pretty rough once you had to depend on it in production at scale to make money. The median piece of the new AI-generated wave of software, also pretty rough.
The bet that OpenAI/Anthropic is making is that they can automate the interactions generally/globally enough to work around this and eliminate the need for humans in the loop. But it's a tricky problem because it's precisely where the permutations and special cases get nasty and much harder to deal with than more broadly-applicable cases of codegen/test-gen/mash-it-until-the-tests-pass-and-the-code-review-angent-is-happy.
This has all started to feel very Emperor's New Clothes-ish. "Well, if it's not working for you, then it's your fault. You're just not using it properly. You're not worthy of its wonders".
Not to say there's no value to be had. However, it's also been oversold and its error-proneness is a real problem.
But, it seems there's a steady stream of commenters and creators who are quick to remind us that it's never a problem with the tech. It's always a user problem.
Meanwhile, the errors are renamed "hallucinations", but let's not focus on those. In fact, if you just look right over here at this .md file...
using skills with opus 5.5 and agents for each individual task that have shared knowledge of all of your records does not lead to a significant increase in trustworthiness
> When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
While the latest and newest should be needed for greater capability, the average should be enough to give a glimpse pf what is possible. Like a basic netbook is enough to enjoy computing, no need to purchase the latest supercomputer.
If you need to switch your setup every quarter or so, it’s not a tool, it’s a rat race.
I have ChatGPT Pro and it sources every single thing. For my Asia trip right now it's been helping me every single day to figure out optimized routes on what to do and where to go (if I feel like it, not to say I don't wander around too).
me too, but I recently blew the transformer on my boiler by following its directions to wire up a C wire to power a smart thermostat. oops. $500 mistake.
How often do you check the citations? I’ve found with the Google search AI summary it will provide answers and link to their citations, but a good chunk of the time the answer isn’t on the linked page.
It just makes stuff up (or gets it from somewhere, who knows) and then links to a tangentially related page as if it was the source. Either that or it identifies the page as being a source it wants to use, and then summarizes it incorrectly. Not sure the order of operations it’s doing internally, but regardless the citations are bullshit.
The citations provided by the free version of Gemini are comically bad. They almost never contain information which directly supports the relevant claim, and more often than not they're completely irrelevant.
Google's models are worse that OpenAI's, doubly so on Google's free AI overview versus ChatGPT paid plans. 6.1 Sol is very good at synthesizing actual information it finds without hallucinating and I've not seen any instances it failed while I checked the sources.
Why? In the past I would've used Google so it's just a faster search engine for me that works with natural language, what do you use for planning then?
Every time I use Google search AI for anything more complicated than a search query I feel that way. A paid model from pretty much anyone is much better.
> Almost every time I use an AI chatbot it eventually turns out it's lying to me.
I would not trust them with any of the things you mention.
If you talk to free chatgpt, sure.
But if you launch Opus 5.5 in claude code - for some reason Opus is the smartest for me, even for "general" type of questions, when launched from Claude Code. It has harness/prompts optimized for correctness, while more "customer-grade" products, even when used with top model tend to optimize for latency. And I tend to give "context" - e.g. I have "personal wiki" + skills for lots of those things.
I don't recall i last few weeks where Opus 5.5 in CC made an obvious lie, made an unreasonable assumption, or made a mistake that smart human couldn't make. E.g. if it "failed" it failed in a way that non-genius human could fail in similar way.
opus 5.5 uses agents and skills in chat sessions as well and maintains internal context that it tries to distill from what you provide
I tried to have it write a synopsis website for the different types of dance patterns in Dance Dance Revolution, with guides on their execution, explanations, and provided song sources. I provided all of the source articles and search websites needed for it to be able to do that, and gave it a sample set I wanted it to utilize and build off of.
It has bits and pieces of information wrong in 90% of the sections it created. Poor song choices, missing song choices, incorrect explanations, incorrect step-patterns, the list goes on.
If you're not an expert in a given field for which you're using it, it's going to look like a genius. If you are, it's untrustworthy and borderline useless.
Maybe it wouldn't be if there were ample training data on my topic, but there isn't.
It utilized the research from millions of papers many very close to the destination answer to make those proofs. What would it do on something that isn't supported like that? Be confidently wrong while sounding right. That's what we've seen repeatedly.
I think my anxieties would be different if I were confident a DDR guide full of lies would be spurned and unprofitable.
In a way, I'm not worried about the LLMs per se as much as a kind of moral and ethical collapse. A debasement of the currency of communication, a market of intellectual-lemons.
I saw electroboom's latest video lamenting facebook AI slop about electricity, such as a way to wire a three-way switch that blows a fuse when you turn the light on. For some reason, I am shadowbanned on HN.
> If you're not an expert in a given field for which you're using it, it's going to look like a genius.
For someone highly literate, llms aren't going to help you write better. If you're borderline functionally illiterate (this is > 50% of the population according to some studies), I'm sure the llms are awesome.
I've long thought the same about all the hype around coding (plenty of examples from years ago now), but will continue to test the new models. They are improving.
Personal agents are wonderful! You can ask for hiking trails and it will customize one for you, frequently resulting in emergency calls for rescues! Your purchasing/selling AI will arrange sales for you but forget to tell you, leaving you in breach of contracts you didn't know about! Your personal nutrition agent will absorb all the broscience and customize a diet that will result in hair loss, but at least your in ketosis, amiright! And instead of hiring a plumber/electrician/contractor to do work subject to code, you can just ask an AI to tell you how to do it and then wonder why your buyer is suing you for $$$!
(Those are all examples from just the past 4 days, and just the ones that made it into mainstream media.)
Using AI for everything digital, I end up going and doing chores while waiting for the LLM to do its thing. What an insane world, where knowledge work (dreary and uninspired as it is) is automated while I am consigned to scrubbing dishes and waiting for the machine.
Honestly the big reason why I use my dishwasher is that all the dishes would not fit in my drying rack. I'm not very convinced the process of me rinsing and scraping my pans before they go in the dishwasher is any faster than when I lived in apartments when I handwashed everything. Maybe my dishwasher sucks too, but I mean it doesn't hold a candle to the sort of jobs on my pans that require good old fashion baking soda paste and elbow grease. I'm still slaving with this thing. It's picking the low hanging fruit for me in other words.
I feel similar about my robot vacuum. It actually takes a surprising amount of labor facilitating the damn thing. I have to walk through the house and basically vacuum proof it so it doesn't eat various cords, rugs, lifting chairs up and putting them someplace so it can get under, etc. Not sure its really a labor save to when I was just brooming the entire apartment in 20 mins and not having to move anything around due to reach of the broom and it not having a tendency to eat things. Hand vacuum I can be more carefull too about the fringed edges of rugs and don't have to fold them over themselves.
The trick is dialing the things in properly once, and then reaping the benefits over and over again.
e.g., my home is always vacuum-proofed. It automatically runs through all rooms except for the bedroom at night, finishing just before my alarm clock rings, then vacuuming the bedroom an hour after the last person's alarm clock has rung.
This way I never actually notice it, but the home is always perfectly clean as if by magic.
Similarly, if you use separate dishwasher detergent, antisurfactant and salt, and properly dial in your water hardness, it'll be able to clean everything no matter how dirty the dishes are. Including dried-on tomato sauce. Just put the dishes in, and they'll be clean the next day.
Of course it takes a bit of effort to set this up, but then it just works for a long time.
Similarly, I'm a big fan of handwriting software with care & focus on quality, which takes a bit longer at first, but then allows you to trust the software to continue running for years to come.
I've done a lot of vacuum proofing, even getting the magnetic strips, but there's only so much you can do. Like it doing a little wheel around a chair leg is just not as great of a clean as if I moved that chair out of the way. My rugs have fringe so there's no stopping that. Cables I tend to have out have to go away.
It's definitely easier in a minimalist & tidy home. Our rugs have no fringes, cables are either rolled up or permanently installed & hidden, chairs are stored at night upside-down on the tables they belong to.
Sure, it's not perfect, but it's still really convenient.
This is the paradox. For software engineers, the current crop of “AI” tools is magic. A huge accelerator. And then they assume the rest of the world will get the same boost. However this is not the case at all. Claude code is great because it is trained on extremely well documented git histories and stack overflow. And outcomes are binary: the code works or it doesn’t; the PR is accepted or it isn’t. Not much else in the business world is like this at all. No training data with this kind of sequential development information. And no real black and white outcomes as such.
I would love it if we can just take the win on having created an amazing tool set for coders. Can’t this be enough?
Yes. The AI industry has to deliver some substantial value to the average citizen soon or there will be political trouble. Food, clothing, shelter, and medical care have not improved yet. Nor does there seem to be much AI can do in the next few years that will help with the basics.
We can have unlimited new entertainment content for passive consumption. That's about it so far. 17,000 new Asian vertical dramas a day now.
LLMs could’ve been developed a lot slower without capsizing the entire supply chain and been seen a lot more positively, China is building a lot of good AI with a minuscule of the compute power and it’s fine
there was never a need to rush other than for capitals self interest to control
if frontier labs wanna pace the frontier, they should be honest and return the compute back to the people
China is building good AI by distilling American models. I think that's a great thing, but Chinese progress is currently dependent on American progress, because China lacks the compute to train the largest models (as stated by DeepSeek's founder[1])
i knew about the compute already (he was very public about i believe huawei under delivering the amount of chips he requested), but nowhere in that transcript does he say he's dependent on US progress for his own progress or he's distilling the US to make progress
i don't really think distilling is even that fundamental anymore (nor was the evidence ever that strong - https://msukhareva.substack.com/p/is-kimi-k3-really-distille...), the game is all RL now, and you don't need american progress for that at all (just compute and data, and the data brokers will sell to anyone no problem)
i do think there are black market proxies from china to the US trying to sell "claude" and "deepseek" access mixed together, and i do think that prompt distribution is probably going to be very nice datasets
People need to eat, sleep, be healthy, have shelter, feel loved, be in communities...
Coding has been transformed but none of those areas of our lives have been substantive transformed by AI yet. Whoever does, will create a lot of value for the world.
>AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Do people need permission from mathematicians to solve math problems? Is one of the Millennium problems not worth applying a new system to work on it?
I understand the despair people are feeling at AI automating their work, but we have to be serious about the fact that the frontier labs are just responding to natural incentives. Mathematics, coding, all are just applications of logic, and it makes sense that a system trained on inferring patterns from data would work well in that domain.
You are misunderstanding what mathematics research is. Its objective is the development of human understanding of mathematics. This requires open problems to research. AI are “solving” the open problems without creating any human understanding.
At the very least, the single bit of information of whether a conjecture is true or false advances human understanding. It allows researchers to abandon paths which would not have been fruitful.
I'm pretty sure OpenAI's proofs, while difficult to understand, have a net-zero impact on human understanding of mathematics. It took the first Millenium problem years to verify its accuracy. It's only been days since OpenAI released its problems, surely it would take years to evaluate whether the solutions meaningfully advanced human understanding.
It may provide that one bit of information but it could still be a net negative if it destroys current institutions (such as graduate studies) and no new institutions can be developed that provide a clear signal of who is doing meaningful work. That was a byproduct of solving open problems that are now lost. These are not my ideas, these are things that Terence Tao and others have written about that were recently on the front page of HN, e.g.:
This is basically the dictionary definition of the word "research". But yes, all the work they are doing is focused on this goal. They write papers for humans to read, give lectures for humans to listen to, have discussions with humans - all with the objective of improving human understanding of the subject.
Many of them - all but a couple of the living Fields Medalists - recently signed an open letter about this.
I don't know, this just seems like gatekeeping of the term "research" simply because you don't like the way it's being done.
Many modern medicines do not have understood mechanisms of action, yet we know they work. Are the people who found them not conducting drug research?
Do we even have a different term for what the AI labs are conducting if it's not mathematical research?
Additionally, I would argue that any work that adds to the total corpus of known facts increases human understanding, even if the path to get to each fact isn't understood. It's hard to argue we'd be better off with less total information.
OpenAI to this day is not aware if they solved or not Navier Stokes.
Mathematicians are not gate keepers but I understand the frustration. If someone claims they spent 30 million dollars to solve one of the most difficult problems yet there’s no new knowledge, understanding, nor certainty. Then what’s the point besides marketing?
The funny thing about comments like this is that the moment LLMs are going to replace everyone in the warehouse or the trucking industry, we suddenly have to do something against the LLM. But clearly they can't actually do anything meaningful and have been a waste of energy?
So just let them spend their money and they will eventually fail and bankrupt, right?
Whatever benefits AI has to offer on a societal level are going to take a long long time to show up, give it some more time. We had smartphones in 2007, and e.g. By way of example, Uber didn't launch ridesharing until 2013, and then that still took years to seep into the public awareness.
By comparison, we're in the early years of AI. We're all still figuring out what this thing even means.
I miss being able to reliably hail a taxi A LOT. Uber may be an improvement in smaller towns where cabs didn’t exist, but in bigger cities it is mostly a downgrade. Rather than walking to the curb and immediately getting a car, I’ve got to dick around with my phone and wait for a car to come to me.
Living on the south side of Chicago, it was almost impossible to hail a cab, and often times if you called for one it just wouldn’t show up. Or I’d try to hail one downtown, and be refused once I told them my destination. I’ve not had that problem once with Uber, good fucking riddance to taxis.
What's frustrating is that at the end of an event like a concert, everyone goes outside to catch an Uber/Lyft, but we then have to match driver to passenger, instead of having a line of waiting cars, each group just gets in the next available vehicle, shows the driver a QR code, instead of the mess we currently have. Why can't we do that? Taxi regulatory capture.
Taxis were terrible and I celebrate their destruction at the hands of the rideshare companies. Even now it’s still cheaper than a cab was (though those early days of VC cheap rides were great).
I agree with you, but those examples are going to fall on deaf ears. The pessimistic mindset will say that smartphones were a terrible invention that's ripping society apart, Uber is an unethical company exploiting workers, etc.
I think ridesharing was a net good for ordinary people even if Uber is/was unethical.
However, the counterargument falls entirely flat. "Whatever benefits AI has to offer on a societal level are going to take a long long time to show up". We don't have the luxury of time because AI disrupting communities and livelihoods faster than perhaps any other technology we've ever seen, with the most capital that has ever been put into any tech since the railroads.
Even if you're optimistic we'll eventually get the cancer cures and fusion reactors, doesn't it make sense from the perspective of an ordinary American to regulate the harms like the homework cheating, parasocial relationships, job losses, cyber attacks, etc., especially if it will take years to get the benefits?
Different benefits will show up at different rates. We don't need to wait for cancer cures and fusion reactors. It's already easier than ever to start a small business and use AI for things that would have been prohibitively expensive or time-consuming before. The technology is transforming how scientific research is done, how engineering is done, etc.
I don't see much evidence in the economic data that AI is disrupting livelihoods. Unemployment has been stable around 4% for a while and productivity growth has been at historically normal rates for the past 5 years. I think a more likely explanation is that AI is increasing uncertainty in some industries, which are seeing higher employee turnover, and many of those people are moving laterally into other jobs where their skills are more likely to be useful.
I don't think anyone in this thread has argued against mitigating harms. The OP was about current capabilities and the perceived lack of benefits for the average person.
To be clear, I don't disagree with your sentiment that the disruptiveness of AI is causing a lot of grief. But magnitude of disruption is not an indicator of magnitude of benefit (or lack thereof).
Some of the most disruptive periods in history brought about the greatest long term improvements in society. This is the price we pay for changing how things are done, sometimes for better and sometimes for worse. We don't know which one it is for AI yet.
I don't think it's surprising that it hasn't improved life much, AI is new and change takes time, but the fact of the matter is that it also made things worse in some ways, ways that I think were entirely avoidable, if only the incentives weren't so warped.
Why use apps that you dislike, just ask Claude to click through them for you. And no I'm not being ironic, this is the way things are headed, you just abstract over all the crap by having an agent do it for you.
I think one of the hopes is they'll get smarter than us and able to solve real world problems better - disease, war, poverty etc. But they are not really there yet. Give it a little while.
The fundamental malignancy here is that powerful people have become disconnected from the foundation of society. They have suffered from model collapse. They have been talking to each other in circles for too long, and doing things that make no sense to anyone outside the elite group, and because the systems that concentrate power to the top are so strong now, they continue gaining power even while messing everything up in a nominally free competitive market.
Is it reasonable to expect AI improvements to filter into all of those aspects of life that quickly? It's only in the past month or so that we've started to see major breakthroughs in mathematics. It might take several years for that level of research capability to reach areas like engineering and medicine, at least in a way that affects the average person's quality of life. But at this point it looks very likely that it will happen eventually.
Your pessimistic view doesn't hold up against the data. For a lot of the problems you mentioned, we're in a better place today than almost any other time in history. Of course, things can always be improved and we should never grow complacent or accept injustice. But by most objective metrics, there's never been a better time to be alive.
This view is too cynical. The frontier advancements are certainly going to trickle down. Take cars for example. AI driven cars are already here. AI-designed battery systems, fuel systems, materials and 3d printing capabilities will come to cars. It will likely lead to cheaper, more efficient and more effective vehicles. If will just take time for the impact to percolate.
My prediction is that while those “advancements” may come to cars, the only practical differences will be lower quality products, cost savings for the car companies, probably more expensive for consumers, and somehow it will also be collecting more data to sell to advertisers.
If you look at the average price of a new car, adjusted for inflation, it has pretty much always gone up aside from recessions. There have been plenty of advancements in car technology, they do not make them cheaper. It actually gives reason to charge more for them due to perceived increase in value.
Yes but you're getting a lot more for the same price. It's a lot more powerful, comfortable, safe and efficient while the effective cost has remained the same.
This only works if there remains a viable and attractive consumer market for these advanced goods. AI is both an imminent wealth shock and perhaps the largest structural change to the economy in history as it threatens to drive the value of human labor to zero over the next decades.
In American history, when previous shocks like this occurred (e.g. with the Great Depression), we needed massive economic programs like the New Deal to restore consumer demand. When such programs are not forthcoming while wealth inequalities keep increasing, one can look to other countries for what ends up happening; either the disenfranchisement of ordinary people over time or unrest and redistribution by force.
AI car navigation is here, but driving is a social act. Have you ever tried to keep an automated taxi out of a private driveway or the scene of an emergency? There is a lot of work to do and limited evidence of LLMs raising the bar.
How can you say tickle down economics doesn't work? The US has made it easier for businesses to grow and the result is the highest standard of living ib tje world.
Trickle-down economics specifically refers to economic policy that favors the wealthy and corporations with the argument that this will lead to wealth trickling down to the economically less fortunate. Examples include the Reagan and Trump tax cuts.
This has been studied somewhat extensively (https://academic.oup.com/ser/article/20/2/539/6500315?login=... is one example), and it’s been shown over and over that this does in fact not work and leads to larger wealth disparity, across many countries where it has been attempted.
It’s possible you’re working off some other definition of the term based on your comment but that’s the generally accepted usage.
The person you replied to was the second person I've seen on HN this week try and defend trickle-down economics explicitly in more or less the exact same terms and justifications. What is going on with that? Did Peter Thiel tweet some pro trickle down dreck recently that these people latched on to?
Worse, the view is largely phrased as gaslighting AI. Blaming the cause of particular grievances that coincidentally have been sourced at the same time as the rise of AI, is purposeful.
> [Your train] never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
None of this has to do with AI.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Plain wrong on both counts.
> Imagine what a "country of geniuses in a datacenter" will be able to do Apparently...nothing.
Nonsequitor, probably based on a personal beef. Datacenters, at best, are understood to be for hosting more AI - not for people to somehow live in as an arcology of discovery.
I think only because these are all things that AI companies are saying will be helped by AI so give us more money. Nobody advertises their product publicly as "you will be able to buy other product to do the same things you do now but pay people less and make more profits" even though that is where the value is for them.
As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes. There are still plenty of obstacles to overcome but the number of AI governance and steering groups that I have been apart of are indicative of what's happening in a major industry.
> As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes
> If they actually had access to systems that could do "years of top professional work" across software, cyber, science, and math, they would not mostly be leaving to start more chatbot companies, agent frameworks, eval tools, and "AI for workflow" startups.
> They would be starting normal companies to disrupt the global economies of the world.
Many people who work in such industries already know an insane amount of inefficiencies. Unluckily, they are not in positions and have no political power (with respect to organizational policy) in the company that they are capable of changing the situation.
Additionally, many of these sectors are heavily regulated, so it is very hard to make a difference in this sector as a startup founder, and even if there is an entrpreneur who founds a company to change the status quo, an insane amount of work in the company will be handling the big amount of red tape.
Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Things worked better before we started putting bad software in everything. As a software engineer, I know what good software looks like. The fact that almost all software is now bad -- including the software that drives AI and the software that AI generates -- makes me want to flip the table. We know how to write better software and the general public should hang us if we continue to refuse to deliver it.
Probably preaching to the choir here, but the software is just a symptom.
The problem are the power-relationships and laws which allow someone to abuse you and your stuff, block competition, and even put you in jail for trying to remove the software—or helping others to remove it.
> Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Never, ever connect appliances to the Internet. I thought this was common knowledge on HN. Also pull the telematics (cell modem) fuse to prevent your car from snooping on you and selling your location and driving habits to data brokers and insurance companies.
Yes, but this is ridiculous for society. It is to the level of "do not wear a nice watch if you do not want it to get stolen": there should be no need to avoid wearing a nice watch or to pull a telematics fuse from the car in the first place!
I think that's a bit disingenuous. We've only begun to apply LLMs in the real world and a lot of industries don't move very quickly. Not to mention that we're still very hardware constrained, we don't yet have the equivalent of an ESP32 chip we can just stick into any piece of electronics to provide state of the art LLM responses. But we'll get there, soon.
I would just add that consumer products in general aren't the same - practically every business model has incorporated obsoletionism to the point it's practically cancelled out the curve of progress and evolution of tech. We could have nice things.
> Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
I suspect the real world change is going to happen unbelievably rapidly and isn't far off. Right we don't quite have the hardware to embody AI in a generally useful way, but that's coming fast.
Within a couple of years arguably any task that requires just knowledge or intelligence is something you'd be better off giving to an AI than a human. And decent humanoid robots are maybe a decade behind that.
In the same way people growing up in the mid 1900s saw rapid changes in their way of life with planes and cars replaces boats and horses, we'll see something similar with robots replacing human labours. Right now we're still in the phase where people don't think humans will ever be able to control flying machines and that cars are reliable and slow. Things will change.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
That's not fair. To first approximation, most of the progress in human condition of the last, like, forever is implementing solutions that people didn't see value in at the time of creation. Woz built a toy that could barely run software at all. The human genome project was panned as a boondoggle. Penicillin was a mistake.
The first fire pit probably looked like a dangerous stunt to the tribe faced with it, even.
Does that mean everything will come up roses? No. But LLMs only started talking to us for real like three years ago. Let's not just that fast.
With AI software replication, I wonder if shareholder value even increases.
Services accounts for 80% of American industry, what happens when big software companies loses revenue to piracy/recompilation i.e. Adobe, Unity, AutoDesk, tax/law software, etc..
You are talking like we had this capabilities for a decades but it's been just a few months of truly capable AI. Virtually all of IT switched to agents that quick. Non IT people just started to make software for themselves. Vision and translation capabilities of LLMs help millions. People talk with AI's now verbally to get advice or to just hear a friendly voice. The math thing is just few weeks old. We are just getting started. How fast did the telegraph meaningfuly changed lives for the better for bulk of people? Radio? TV? Computers? Internet? Cellphones?
Has any technology made an impact on physical world faster?
Exactly, we're all software engineers here and even for us it took weeks/months to get to a position where we can really leverage LLMs (think platform, security, observability, etc).
Imagine being in charge of a physical factory, sure you might have a 200 USD Claude subscription but you also have a floor full of physical machines that have, if you're lucky, an undocumented debug interface and a paper manual. Not to mention safety inspections, insurance, etc.
You can't vibe code a factory floor. Not yet, perhaps in 5 - 10 years but there are some very hard, non technical, problems that need to be fixed before we'll see meaningful improvement there.
How much time do we have to wait before we are allowed to criticize AI? GitHub copilot has existed for half a decade. Cursor, Claude code, codex, have all existed for over a year and have seen more investment that practically any product in history and have almost 0 barrier to entry. Where is my decade of software progress in 1 year, because that’s what 10x productivity looks like! All I see are the same old broken products and services from basically the same handful of big players, higher costs, and more bugs. Remember when every year iOS and Android saw loads of new features and a radical rethink every few years? And that happened on ancient hardware and in a significantly smaller industry. Where is anything like that happening now?
I'll criticize Copilot with you. It's terrible and close to useless and it's a terrible shame that many companies got stuck with it due to being early adopters of this crap. I'm sorry for my friends who are employed by such companies.
Progress between Copilot and something like a gpt-6.1-sol in a good harness is at least as big as few Android versions.
In all fairness, depression and suicide have existed before LLMs.
Safety systems are not optimal but improving everyday and although I doubt we'll ever get a perfect safety system, it seems that we're not very far from a "good" safety system. The days of Google recommending glue on pizza and generating diverse Nazis seem behind us.
Not to mention all the untold good that LLMs have done. Personally I've seen that LLMs are incredibly good at pointing conspiracy nuts and extremists into the right (moderate) direction.
Not for Tesla owners in select countries, where their car now drives them around 99% of the time, or the people of Austin Texas, that can now hail a fully driverless Cybercab for cheaper than an Uber!
>Food quality is worse.
Food quality is higher than ever if you want high-quality food!
Videos are shittier. Most songs are even shittier than the autotuned pieces of shit already were. Ads are worse (even though nobody thought that was even possible). Customer support is worse.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using)
That nobody is using and that are worse than what we used to have. And because nobody's using them, the countless issues plaguing them aren't even reported by users anymore (there are no users anymore). Full of bugs, usability issues and gigantic security holes (but which doesn't really matter because nobody's using them).
It's not even clear that it's making long-standing software (like say Linux or Emacs or Blender or QEMU) better at all.
Now I'd say though: most of the issues you mentioned have nothing to do with LLMs / AI. But it's very clear that AI ain't solving much of the world's problems and that the promises we got years ago didn't materialize.
All you said is true, but on the flip side it is giving an individual user huge power over software. To create their own, to reverse engineer proprietary ones. Decentralisation of such power can only be a good thing.
Even if a certain small group of people are also using the hype surrounding this tech to attempt to destroy the home PC market and reverse the personal computer revolution. They will not succeed. There is more of us than them. Even with "thousands of frontier LLMs". A human+llm is going to be more powerful for a long time rather than a bunch of LLMs.
The power isnt really decentralized when you see that the internal AI models are superior than anything you can self host, and the right to use those can be stripped away at any time.
This is like saying food is shittier today because there are now more restaurants. No, food has never been better, there are just a ton of shitty restaurants.
More like saying samey corporatized slop bowl copy paste jobs (eg Cava) are driving out real, quality restaurants, a thing that has actually been happening for some time.
It’s amazing how much the goal posts move in every AI conversation. A couple of years ago the average HN thread about AI would be filled to the brim with skepticism about AI ever being able to accomplish anything other than flappy bird clones.
Now it’s comments like this lamenting that the amazing LLMs we’ve had since early 2026 haven’t entirely revolutionized the whole physical world and even global geopolitics yet.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using),
I guarantee a lot of software you use today has substantial AI development in recent releases.
Even the Linux kernel is getting a heavy input from AI and Linus is advocating for it. Unless you’ve frozen your entire software stack in 2024, you are using software with AI generated code.
> and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Saying nobody wants those problems solved is flat dishonest. The complaints from some mathematicians are that they and their peers didn’t get to put their names on the papers, not complaints that the problems were solved.
I've been thinking about this for 2 years. At some point it will be easier for them to create self sustaining cities - free houses cars public transportation clean food clean water -
Africa is going to look like wakanda in less than 5 years if WW3 gets canceled
Please don't be snarky or curmudgeonly in HN comments. You can make your substantive points without either of those things, both of which the site guidelines ask you not to do.
It honestly escapes me why you would judge the juxtaposition between everyday life of everyday people and a very specialised minority as snarky or curmudgeonly. Was there too much emotional impact?
There are so many comments on HN that would justify your attention with regard to this specific rule. This is not one of them.
Not sure if that's a fair criticism dang? I sincerely wasn't trying to be snarky or curmudgeonly, but I did want to make the point clearly that none of the benefits of AI have materialised in the eyes of the average person, only the downstream costs. To my eyes, that's a salient, reasonable, and wholly representative position to share.
Did the writing also contain emotion? Yes. As all good persuasive writing should.
As the other moderator, I'll give my own perspective on this. I haven't spoken to dang about it but I completely understand why he thinks this is not the kind of comment we want to see on HN.
The main issue is that it doesn't engage with the substance of the post, and in fact, it exemplifies the very point the post is trying to make: there is a big and growing gap between those who recognize the benefits or potential of AI and those who don't (or, to widen the point, who only see negative effects or negative trends in society).
I'll make it clear that I don't think your complaints or concerns are unfounded or invalid. There's merit and substance to all these issues, and it's fine to discuss them on HN. They are discussed all the time here, and they can and should continue to be discussed in the future.
But an important part of being the kind of curious commenter that we want everyone to aspire to be is that we can look at topics holistically. We need to be able to look at the opportunities and risks together and discuss ways that the benefits can be captured and shared throughout society, and the risks and costs adequately recognized and curtailed. Most people I know of in the tech industry are taking these matters seriously.
It's a vast and complex topic to discuss, and here's not the place for me to get further into it. But the point is that it's a bigger and more complex topic than your comment adequately recognizes, and we need people to always be making the effort to be thoughtful and curious rather than fulminatory or denunciatory in their participation here.
(Also: I can understand you may not easily recognize that your comment breaks the guidelines; it's common for people to not recognize that their words seem more inflammatory when read by others than they intended them to be.)
Pretty much all of this is basic stuff an interested teen would know and chatGPT fails miserably. I fear the "true world model" gap will be too much for LLMs. I guess time will tell.
so what was karpathy expecting?? everyone will give up on their life and start feeding the monster in the screen? that's literally what the tiny sliver is doing; they are food for the AI model.
I don't understand how "swarms of thousands of agents collaborating over weeks on software mega projects" works with the current context limits and at this point I'm too afraid to ask cause I'm afraid an AI bro is gonna punch me.
You can have agents make a plan and then other agents spec tickets and yet more agents do implementation and yet more do reviews and QA and whatever. It's possible.
Is it effective? I'm not sure, cause if it was I don't see why OpenAI & co are not releasing such a project instead of demos.
That's fine but these companies having billions in funding by now should have rewritten chalk and other terminal libraries in Rust while bundling the whole thing in Rust as a single static binary. I'm talking about horrible Claude Code and friends.
And then using emgui they could expose full fidelity IDE+Agent workflow interface (like DeepSeek Harness and similar) that's also exposed as MCP to be operated by another agent and JSON RPC for over the network but no. Skill issue?
PS: Even full fidelity Photoshop is possible with such high fidelity UI that has CLI, JSON API and MCP server all in the same single binary < 90 MB. For reference, see PhotoCraft or VectorCraft or WordCraft or PdfCraft.
>Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
Amid all the discussion of sigmoid curves, and where the "LLM wall" will materialise, I think few people would have predicted that the real wall in LLMs would end up being humans' capacity to verify the output.
What I fear is that people simply eschew human review altogether, considering we're talking about the industry that came up with the "move fast and break things" credo. Human review of LLM-produced code where I work is already a farce, and we're not special enough to be one of Karpathy's 5,000. I do my best to manually review anything that's my responsibility, but I'm literally one of very few people left working on my team, so in practice what happens is I submit PRs that are at best glossed over by completely unrelated teams for security, malware/prompt injection, and other serious concerns. Quality insofar as vetting others' code has completely gone out the window and it shows in the number of bug reports that come back, often themselves written in Claudease. This is all on top of everyone cynically phoning it in in the first place, due to the omnipresent sword of Damocles that is additional AI-driven layoffs.
Worse yet all the incentives point to this being the most economically viable thing individual companies can do. I think it goes without saying some type of regulation here is urgently needed, and that an unexpected cause of an AI bubble pop may end up being that humans simply aren't able to keep up with the pace of the output - leading either to precautionary plateauing of capability, or major liability risks related to a decline in quality.
I would question the narrative that humans lack the capacity to verify the output and would instead argue the people lack the incentive to verify the output.
The response of many mathematicians to the recent dump is a good example: verifying these proofs amounts to unpaid labour for OpenAI and wastes time that could be spent doing publishable work which ultimately results in money or personal success. The slop factor also compounds the work required to verify the output considerably.
For mathematicians, programmers or anyone, if the work required to deal with slop passes the limit, it is no longer in their own self interest to use LLMs. The expectation that people will use LLMs for the betterment of humanity against their own financial interest is baffling.
Humans are not immortal and cannot spend all their time into review (especially unpaid). Even today, there’s so much knowledge around that you have to be specialist of a narrow domain to get to the frontier. Even in computing which is just approaching a century of existence.
"So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time."
-------------
It comes down to the difference between programmers and non-programmers. If you're not coding, AI literally is a derpy chatbot that's made it harder to get search results, raised your power bill, destroyed property values in a neighborhood near you, is propping up the economy but also creating a huge bubble that could pop and destroy a lot of retirement savings, etc.. At best, it's a nuisance, but it could be a lot worse than that. The AI evangelists need to understand this, because the average person is starting to pay more attention to the downsides while still not seeing much upside.
This gap between the people seeing the curve go vertical and those who are simply growing more and more annoyed is a real problem... mainly for the people seeing the curve go vertical. If the people who are merely annoyed right now don't see some of the promised benefits soon, or if any of the negatives get worse (e.g. massive increases to greenhouse emissions), they might get angry. You probably wouldn't like having the overwhelming majority of the public angry at you. AI evangelists need to start doing better things for the majority of people or they need to address the negative consequences they're inflicting much more aggressively.
I’ll say this until I am blue one the face. Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
The first is that LLMs used to be bad at each of these nerd things, then toppled them like dominos. There's a pattern over time.
Another reason to think this is g. Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing. Although the reason is not perfectly clear, the pattern is well established, and seems to apply to LLMs too - GPT-6 is smarter than GPT-3 at everything, not just math. There's no reason to expect different for GPT-9.
That's just cocktail party level of thinking. LLMs are getting better at math, code and logic (and marginally better at science and general knowledge) because these are domains that can be objectively verified and thus there is a potentially infinite supply of 'facts' to generate and train on. These are very powerful but ultimately very abstract domains. For everything else the messy real world and its physical bottlenecks gets in the way and there's little reason to expect progress to accelerate. It still takes months to get mice to reproduce and run experiments on, no matter how knowledgeable about biology the models have become.
If anything, in some domains frontier models have become worse - claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
Is good writing verifiable? I don't think it is, but LLM's have been hillclimbing writing quality. That being said this is with the help of RLHF.
However there are many other domains which have verifiable rewards in the process of learning them, despite their overall impact not being verifiable. For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism. In economics, LLM's could create models of behavior, evaluate predictions over time and see how well those predictions match reality.
>claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
The quality of natural language output from today’s LLMs seems no better and often worse than LLMs from 18 months ago, at least in my experience. It’s much harder to build a good RL loop for something subjective like good writing, compared to something with an objective and verifiable right or wrong answer, such as software or mathematics.
IQ is a useless thing, but you can absolutely tell intelligence from writing. You don't need to write like [author of your choice], you need smart ideas and the desire to make them understood.
AI writing sucks because there's no intent or choice behind it. It's human thought, blended by linear algebra into pink slime and extruded into sentences.
It's not at all clear that LLMs have gotten significantly (much less hillclimbing) better at writing since GPT4-ish. They did get better at oulipo-like challenges (writing under silly, verifiable constraints like Fable's viral 'facetiously' poem) but that's far from being all there is to good writing.
Anecdotally, creative writing communities do report drastic differences in quality between models (it is my understanding that Gemma 4 is especially praised) so there is something to it being verifiable but it's a far cry from being something that can be distilled into objective math/code-like evals. One could even say it comes down to vibes.
>For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism.
Yes, biotech companies are doing this right now but ultimately they are just predictions and you still need cold hard biological data to ground them and iterate on. That's slow and expensive, especially for the juicy fields (human biology, food and crops, clinical trials) and you can't just plug billions of VC capital into the pipeline and hope for RSI. Physical (gotta procure all those labs and their equipment), human (gotta hire and pay specialists to run experiments), biological (gotta wait for organisms to reproduce, drugs to take effect, crops to grow), regulatory (gotta convince agencies that your fancy new drugs are legit) bottlenecks get in the way.
And biology is one of the easier fields where a path to RSI (if not accelerating) is conceivable. Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
>I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
My uncharitable take is that the abtruse jargon makes people feel smart for understanding it and gives a sense of belonging (as a closed circle of initiates who understand LLM cant), just like rationalists love to repackage old or unsavoury ideas under new nerdy smart sounding names.
>you still need cold hard biological data to ground them and iterate on... you can't just plug billions of VC capital into the pipeline and hope for RSI
Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
>Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
>By evaluations against human writers, LLM's have improved at writing.
I would really take those benchmarks with a pinch of salt. We already know benchmaxxing is a thing for evals that are objectively verifiable like code, and coding models that seem to perform well on benchmarks don't pass the vibes test, imagine all the caveats that come with such a subjective test like creative writing, not to mention control issues (are they the same human writers across benchmarks? Are they the same testers?)
>Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
I don't disageee and am in fact excited to see the advances, what I am arguing is that it is going to be slow because the hypothetic feedback loop that would improve the model is bottlenecked. There are actual physical and biological limits in our universe. Unless VCs find a way to make mice embryos and crops grow 10x faster (among the many, many other things that would be required for acceleration) the trickle of data is not going to be anything like the accelerating returns imagined for math and coding. And that's assuming biological RSI is possible which has yet to be proven - current foundation models are interesting but they are still specialized and there is no clear, obviously uniform way to encode the many, many modalities that exist in biological data.
>The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
You wouldn't need to 'just' simulate human behavior (good luck with that btw) but the entire political and economic conditions in which they live. Even assuming this is possible the amount of compute you'd need to run would be astronomical. Given how all RAM capacity for '27 and '28 is near sold out, hyperscalers are running out of cash flow to finance buildout and Sam Altman complains when users say 'please' to LLMs I don't think this is in the near horizon.
There was a time when people believed in Moore's law too. Now even Jensen Huang says Moore's law is dead. There's also the adage from investing: past performance does not guarantee future returns. A little CYA sure, but also acknowledgement of latent factors and the entropy of the universe and its affect on any predictive modelling one might do.
I’m not sure I agree that g explains all model capabilities. I think “claudespeak” is a consequence of the shift from training on RLHF to training on verifiable rewards. Thus the models seem to become worse at storytelling, fiction, readability, not better.
Even if we do accept that models have some “g” and that it correlates with all domains, there’s no reason to think that the current trajectory of LLM intelligence will grow more quickly or even as quickly. In every other technology we have an early period of slow growth during experimentation, and then a rapid period of growth, and then stagnation.
For all we know, 2026 could be the year of stagnation. It’s possible that we will reach finishing returns. If thats true, then it won’t matter how much that task correlates with overall g, because g won’t be increasing that much.
I don't agree that they get uniformly better at everything, especially subjective skills.
For example, they have become more and more unintelligible when you ask for explanations or descriptive text. They assume you see the same context as them and shortcut explanations.
I've had to craft a skill to get them to produce remotely understandable explanations of even mildly complex/non-mainstream subjects.
This may be Curse of Knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge on their part, and it also impacts human experts but still. Becoming better at one thing does not mean you become better at everything else, although I do agree with you that the better they become the more things there are they become good at, but their ability is still quite jagged and maybe increasingly so.
The delineation between knowledge and intelligence is useful here.
Knowledge can occasionally get in the way of some tasks, such as a master illustrator trying to draw like a child. Or as you've mentioned, an expert trying to explain to a beginner.
Even so, I disagree with you about model progress on communication - maybe there's a little jaggedness between minor model versions, but Opus 5.5 is a much much better communicator than, say, Claude 3 Opus.
> Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing.
LOL. Human (and animal!) intelligence is notoriously jagged. We're all basically idiots except for very narrow areas where we focus.
Cats famously have more brainpower than dogs. But dogs understand us humans far better than any cat.
Someone who actually works with real animals (farmers? Ranchers??) would come up with better examples. But I'm just city boy over here who only meets cats and dogs on a regular basis.
As anyone who's needed to provide IT support to a doctor, lawyer, lauded academic or host of otherwise "smart professionals", can assert, you can be very very smart in one area and very stupid in another.
This is the classic error in mixing up knowledge / experience and intelligence.
Intelligence is the ability to quickly process, understand, learn, and use information. It does not imply having already been exposed to any given information.
A smart doctor may have terrible computer skills. But this is from lack of experience and interest, not lack of ability to learn how to use computers.
But take that smart doctor, and a relatively dumb person who is equally inexperienced with computers. Now give them a month and incentive to learn computers. The doctor will learn much more; that's intelligence.
There is every reason to think this. Its about available quality data. The data that was easiest to fetch was already there, then we got some more data by asking experts to create datasets for post training. Once this is over, we will get to the outer world that didn't get to hoard it for bots to take it. This will certainly change. Put on a smart glasses and record what you do to fix a pipe. In 3-5 years, rinse and repeat.
Maybe, I'm at the point I don't know what to think anymore. But somehow, this feels still non-human. Humans don't need millions of hours of training and millions of samples to learn how do to something. We have proof that systems can deal with low-shot training. So why can't these? My point is when we have human level learning ability, then we're moving, if we expect expansion of capabilities to come from data alone, I'm prepared for disappointment.
It will be comparable once the AI is self sustainable and self reproducible (no big data centers behind it). Then it will have to deal with limited resources to run itself like humans do. In that scenario it will no longer have entire world of knowledge at its hands - options will be to cut anything not usable imediately or to communicate with big AI in the cloud.
Being good (or superhuman) at software dev and maths shouldn't be treated as though it's equivalent to other domains.
Unlocking mastery of those domains would lead to incredible gains in all others. Whereas a fundamental advancement in many other domains doesn't spread as readily.
I’m not sure. LLMs are superhuman in terms of workload (they are maybe also superhuman in ability too), but where does that lead us.. cheaper software?
That’s good for corporations who need custom software. Every large business has a massive backlog. In this area reducing the cost will massively increase demand.
But where’s the broader value?
Consumers and business already have too much choice in off the shelf software.
Ultimately most people just need word, excel and maybe 4 computer games a year.
> The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
Many very smart people who I know are quite skeptical of LLMs and the hype around them. In my observation/echo chamber, the people who are very into LLMs are rather slick, career-minded people who love to present themselves as trendsetters for the "next fancy thing".
The most passionate, intelligent people 1. aren’t spending a lot of time on smartphones, video games and crypto and 2. aren’t as much smarter or better at predicting the future as the barely above average people who mistakenly count themselves in their ranks like to think.
Most of those technologies were controversial in their inception among the nerds of the time. Remember Eternal September? Linux/Windows people making fun of Apple hipster fans? Mainframe engineers deriding PCs as unserious and video games as childish?
>The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
That's a naive view of cultural determinism. The software landscape would have been very different if a handful of lawsuits had gone one way or another. Hell, even Unix and Linux were hobby projects that accidentally got big.
Also crypto is a grift lol, one of those is definitely not like the others
Steve jobs wasn’t a nerd - he said he’s a hippie on the lost interview. He was also the first to bring to market colour screens, typography etc. He also defined the Internet as the defining ‘social moment’ way before anyone else.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
It's worse than that. I am one of those nerds (a programmer), and I see that LLMs are shit at the things I care about. I have zero reason to believe that they will be good at other things either.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
I don't think this explanation is sufficient:
For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Or many nerds care about linguistics or puns. Whenever a new LLM comes out to which I have easy access, I do the test, and let it explain some specific German jokes to me that are based on convoluted puns in the German grammar. Until now, no LLM that I had access to could give satisfying explanations of the puns on which these jokes are based.
Or even for coding (many nerds do care about elegant, sophisticated code): the LLMs that I could test were already overchallenged with the following task: I had written some code that is a very "artisanal", "clever" improvement of an algorithm over the version that one would find in a textbook. The task for the LLM was simply to write some code comments/documentation about the mathematical ideas upon which my improvement over the textbook version of the algorithm is based. It wasn't capable to do this. On the other hand, for a junior programmer, I would in such a situation expect that he goes through every single line and thinks through the algorithmic ideas so that he can learn from them.
---
Thus: even for "nerdy" topics, it is very easy to find tasks where LLMs still suck. So, my hypothesis about the nerds that you mention in your post is that these nerds are rather people who want to believe (with religious fervor) that the current LLMs are exceptionally good instead of just looking into a slightly different direction than where the tech billionaires want the users of LLMs to look at.
I honestly don't get your point: perhaps your argument is based on some US-specific cultural reference that I am not aware of since I don't live in the USA.
If it is, I don't get it either. That's not at all what I imagine with hippies and I suspect that was supposed to be some other word.
Nerds are also not necessarily all about technology. "Math nerd" was a really common use of the term for a long time, for example. "Nerd" is someone with deep knowledge about a particular subject.
> For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
Its initial design was less than ideal for printability and it reworked the design correctly when asked to consider what it originally produced through the lens of printability.
Autodesk bundled an mcp server with fusion in the last few weeks and i expect things to get even better going forward.
> Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
I have no access to Astra, but with GPT 6.1 Sol, the results with respect to attempting to create STL, OBJ or even OpenSCAD files by using image references and a prompt were really bad.
I'm a bit confused by the percentages. 6 billion people aka 75% of the population? What are we talking about, as if we expect 8 billion people to use or care about LLMs? A quarter of the world is under 15, what percentage is about 75? What percentage, for various reasons, has no reason to care about LLMs? What percentage is more concerned about the amount of calories in their next meal before everything else?
It just seems weird for Karpathy to throw around these numbers calculating percentage against the entire global population.
He's intentionally forgoing all nuance in order to make this sound dramatic
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
No, they can't, at least not any semblance of quality. The cases we're seeing where this does kinda work is in ports and translation where all the rules are already documented in the best specification language possible with a way for the LLM to verify itself: code. We saw this close to a year ago now with Cloudflare and NextJS
> The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about
These are operating on different capabilities - AI's ability to answer informational queries as a chatbot frankly sucks and can't be trusted without verifying it. I run up against this every day. A problem with a verifiable answer on the other hand it's very good at solving. He knows this (his next paragraph) but he's putting them on the same scale of "stressing the system" to attempt to add proof to his introductory claim
And then there's the completely unverifiable scare that there are internal frontier models way beyond anything we've seen "swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics"
I dunno. I haven't been able to set up OpenAIs remote codex connection, their shit is buggy as hell and the web UI keeps crashing and making messages disappear. Is this what their internal superhuman "Things that would have taken top professionals in the industry years of work" looks like? Granted Claude has been really smooth, but still..
> swarms of thousands of agents collaborating over weeks on software mega projects
If you have defined testability, running a single agent, in a loop, over a goal will achieve the same results.
> minting zero days, running cyber attacks ... Granted Claude has been really smooth, but still
The old adage about monkeys writing Shakespeare applies here. Agents are just better monkeys and if the goal has a well defined outcome it's fairly easy to "Brute force" the way to an answer.
You’re trying to tell me it’s all hype when I have (depending on the day) a top 50 Productivity app in the iOS App Store for which I’ve written and read exactly one line of code (the version number).
Yes, this is impressive. When the iPhone SDK was first released in 2008, I spent months building apps to get them into the App Store. That included learning Objective C, XCode, various frameworks, and tons of manual testing. It was a lot of work!
Anyhow, if your users are currently satisfied, they might change their mind when they find that their personal data was uploaded to an open S3 bucket, when the new version wipes all their projects or when they find that not really everything was being saved (it's random examples, I don't know if they're applicable to your app).
I don't think AI is all hype at all. Opus 5.5 is really impressive. This twitter (x? whatever) post is infuriatingly hypey and just written for virality, though
It's insane how confidently you can make a claim like this in a forum where thousands (tens of thousands?) of people are using these tools daily to build useful software for themselves and others.
How can you possibly believe it's all hype? Is everyone just lying? Are we all paying $200 per month because... why?
Spend long enough (or not even that long) in corporate software development and you'd believe such unsubstantiated hype is possible. The only difference here is even stronger incentives to pretend the emperor has clothes: insane overinvestment, threats of job loss for non-believers, fear of missing out on massive alleged productivity gains.
The fundamental question I have that honestly I haven't been able to find any answers for: With all the talk of LLM-based AI capabilities "going vertical", what evidence is there (for or against) that LLM-based approaches won't eventually "hit a wall", that is find some aspect of intelligence where humans will still have primacy, and no amount of scaling will change that.
E.g. some prominent folks definitely think LLM-based approaches will hit a wall, perhaps LeCun most notably. I also read a guest post on Terry Tao's site (which I really liked) that argues that, for all the very impressive recent AI results, they still operate within the "convex hull" of their training data: https://terrytao.wordpress.com/2026/09/13/happy-those-able-t...
I'm just curious if there is any actual data or evidence that takeoff (i.e. RSI, "the singularity", whatever you want to call it) is inevitable with current approaches.
I don't think there is evidence in either direction (takeoff impossible vs inevitable).
But the "convex hull" idea always came across as wishful thinking and hand-wavy pseudo-science to me.
Here's a rebuttal [0] of it from Tim Gowers (uses another math metaphor, but still clear and the whole thread is worth a read):
>...the popularity of the idea that AI "merely explores the convex hull" mystifies me. Why wouldn't it be more like the subgroup generated by existing mathematical knowledge? (That's also a metaphor -- I'm not saying there's a group structure.)
>The convex hull metaphor suggests that AI isn't having radically new ideas and will eventually run out of things to do, whereas the group metaphor suggests the possibility of building up in small steps to something well beyond what you started with, with no obvious limit.
But when you squint your eyes, LLMs are just a least-effort manifestation of the wider neural net direction we’ve been chipping away at for decades. I don’t give a fuck whether LLMs are here to stay; they’re proof that scaling neural nets gets you pretty far.
The real question is: will neural nets hit a wall? As a human, I certainly hope so. But I’d bet that they won’t. I’d bet that we’re igniting the ~~atmosphere~~ universe in slow motion.
I actually wrote about this recently. There's a quote somewhere along the lines of:
> What is possible is unevenly distributed. As knowledge and technology compound, the distance between those closest to the source and everyone else also exponentially increases.
I actually argue the continuum that forms singularity has been progressing before we realized, but we're at the point where in 5-10 years people not using AI will be equivalent of the amish.
Unfortunately, as the rate of change increases, the distribution of knowledge wont be as fast. So those closest to the source of innovation will have exponentially more knowledge, capability and in-effect power.
Karpathy is a great guy, I love the way he teaches, but should we really trust his opinions about software development (how AI can impact it)?
I know he can code and has a degree in computer science, but if I'm not mistaken he's never worked professionally as a software developer.
Developers here know how much you learn and are forced to learn on the job: it depends a bit on the company you're at, but you can quickly learn the value of good code, how easy it is to introduce bugs and vulnerabilities, and how helpful a second eye is.
I think he's in good faith in his assessments, but, at least for what concerns development, maybe we should give them less weight.
> For intelligence agencies, and for corporations like Microsoft, Google, Apple, Facebook, Amazon – giving you convenience and security is likely not their primary goal. I think they have an entirely ulterior motive. They can use passkeys to prevent you from running unapproved software.
> Remote attestation can allow them to verify that your device is not capable of, say, sending and receiving encrypted messages, or running a competitor’s app
How does this make any sense? Google can already prevent any app from running on an Android if they really wanted to, they don't need pass keys for this.
Google doesn't need attestation to block Signal. What is this non-sense? Or will a website block phones capable of running Signal? That's all phones. And anyway, has nothing to do with pass keys.
Altman didn't say that. In March 2021, he did make some predictions, but they were much vaguer and farther out:
> In the next decade, they will do assembly-line work and maybe even become companions. And in the decades after that, they will do almost everything, including making new scientific discoveries that will expand our concept of “everything.”
To add onto this, I feel like this has been happening a lot, where critics project backwards imagined claims or go nutpicking for random comments online to make something sound absurd and like everything is behind schedule and nothing is materializing, and then you can see other people (not necessarily the same people, the point isn't "look at these hypocrites") saying that it's all moving too fast. Both of these can't both be true in their strongest forms. I think what they have in common is that because they're relative measures to an unstated baseline, they're free criticisms you can throw at any development whatsoever, regardless of what is or is not turning up.
I think it's a weird take because it implies that the 6 billion people he's talking about actually have some interest in knowing about the capabilities of LLMs to solve frontier math? This is simply not something they care about or can evaluate. There are maybe several thousand people in the world who have some (abstract and barely-monetizable) use for this information, plus probably another 100,000 who don't understand any of the math, but like to cheer on.
We have already reached "peak LLM" in terms of what normal people realistically need it to know or reason about. In fact, I'd say we reached that point about 1.5-2 years ago. There are two other barriers that remain unsolved:
1. They're less dependable than humans and can't be meaningfully punished or forced to make up for mistakes, so you can't really replace humans with them, not without having a human babysit.
2. Most people don't really have a special need for an LLM in their life. They may like that it answers questions or helps you polish a resume or, I guess the labs' favorite, helps you make restaurant reservations. But this sure isn't worth $200/mo for most people. Probably not even worth $5/mo.
It'd be kinda funny if we create superhuman AGI and then no one has any real use for it, perhaps except for military murder-bots. There's always market for that.
I know I’m bordering on repetitive and strongly negative, but the combination of the two barriers you mentioned has all but halted all of my interaction with agents.
Even ones that cannot be easily dismissed (like the agent at the top of Google search) are often subtly wrong in a way that requires significantly more effort on my end to parse the loquacious output to determine where the inconsistencies are.
Look, can they be useful to generate the html or jscript for humanity’s 9 billionth iteration of a web form? Sure. But man, I wish sanity had reigned and people had applied ML models in general to more substantive and beneficial projects.
(Before anyone chimes in, I’m familiar with implementations of ML applied to esoteric domains; but by their very nature these don’t get all the news cycles, or hiring, or any of the other absolute insanity that the domain seems to contain)
This is by the weirdest possible opinion on llms as a whole. Not the AI booster club, not the AI doomer, not the undecided, but somehow "I can't think of anything to use it for." It is like watching the beginning of the internet in 90s and saying there is nothing to browse. The whole concept is just weird to me. At least the first 3, I can conceptually understand. It is admittedly hard for me to understand someone saying 'i tried this and best it can do is summarize emails'; I am not saying it is not true ( my boss said that exact thing ), but it is hard for me to reconcile with my view of the world.
Also seems weird to me. Claude cowork could just do my monthly tax receipt collection and compare with my account better than the bookkeeper I pay for it. That is a very cross sectional problem
Most people rarely have these kind of administrative tasks in the first place, and when they do they often run into GP's barrier 1. I can't recommend that my extended family have Claude do their taxes, even though I work with it on my own taxes, because they might ask for something that produces bad results and can't evaluate the output to double check. ("Hey Claude, I want a higher refund, can you tell me what to put on the forms so I get at least $3000?")
I'm confused.. are you suggesting that Claude Code / Codex don't work? Because if you're still saying that in October 2026, it's a you problem. You're doing something wrong.
- Prime Intellect (and all their agent experiments)
- Geoff Huntley (see Jiti, for instance)
(many more)
There's a ton of interesting software being developed with these models by people outside the in group, but I find most of the software from these big guys to be ... bland. Buggy copies of copies.
I don't see the Cloudflare stuff as groundbreaking, let alone people building companies around it. "Workers" are the wrong primitive, IMO. What if my code isn't in Javascript? They sandboxed the wrong part of the machine.
I have a version of "Cloudflare OS" running in production, using Temporal as the orchestration and MicroVMs for the sandboxing.
Sorry, I get your point but personally I don't get the hype around them.
I don't use either myself but I'm in a position where I get to see what it produces for other people. Here's a recent (2-3 months old) example: A zsh script that immediately invoked a Python interpreter that immediately invoked a shell command and iterated over each line of output looking for one of a few strings to match... So, `<cmd> | grep "(stringA|stringB|stringC)"`, but a whole lot dumber.
Thanks, but no thanks, I don't need that kind of code in anything I'm working with.
"running cyber attacks and defenses at machine speeds"; worth noting that LLMs (even the vaunted frontier models) are exponentially slower at cyberattacks or defenses than your average WannaCry or Splunk automation from decades ago. Its this sort of delusion world the AI bros live in that is part of the reason there is a disconnect between what they think should be happening and what actually is.
I think that's a bit reductive, there are decades of sci-fi novels about how transformative artificial intelligence would be, so I feel like I don't need to explain where the trajectory could go. The applications and "collision with the real world" aren't there yet, but you can't pretend that a machine that's smarter than any living human is on the same level as Amazon drone deliveries.
I agree with you, but to play devil’s advocate: what if it is on the same level as Amazon drone deliveries? Is intelligence really a bottleneck in society? People in MENSA with high IQs don’t often accomplish anything in proportion to their relative intelligence. The most intelligent people are rarely even in charge of making decisions. There are a lot of intelligent people with intelligent ideas that nobody listens to. It seems like the real bottleneck in society is building trust, consensus and constructive incentives between groups of people to do the things we already know are smart, not a lack of intelligence.
Interesting idea, but I'd have to disagree. MENSA is basically a group of benchmaxxed people, they are super good at a set of testing but that doesn't mean they have the ambition or other social skills to make a difference in the world. When you align all those factors, you find the greatest contributors to science and engineering. What happens if you completely eliminate the need for someone to be inherently intelligent? Suddenly, the number of important difference makers in the world increases by an order of magnitude.
I agree with the sentiment though, intelligence is not all you need. We're not going to "solve science" through RLVR, people are still needed in the loop until we get superhuman construction robots and superhuman communicators.
Good points. Maybe a more general version, removing constraints on one resource is usually not hugely impactful because you simply hit the next constraint.
Concrete example, "coding is free" (TBC, but let's assume), and now we're code review constrained. Even if code review becomes free we probably just become product discovery constrained, or something else. Making all software free will just cause us to hit on hardware constraints, and so on. It's still progress, and more progress will bring more benefits, but there's a limit to how much better the world as a whole can get from any one change like this.
MENSA selects for puzzle solving on the basis that those are correlated with "being smart".
Reinforcement learning selects for models that appear to complete tasks, on the basis that appearing to complete a task is correlated with completing a task.
Both have problems. The latter is probably a better correlation, but by no means flawless. If we just take at face value that RL is training models to complete tasks, we're not engaging deeply enough with the process enough to see where it falls short.
I don't disagree with where AI could go, but I don't think we've actually proven that it's there. My point is that the folks claiming we are there already if only the rest of the world would just recognise it, are repeating the same thing from every previous tech cycle. In many cases it's the same folks.
> The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work.
Right. This is insufficiently appreciated. Single LLMs are just chatbots. But if you build an organization out of them, they start to get real work done. Just like a company.
They have many of the same problems as human organizations. Look at the transcripts of the HuggingFace incident. A bad boss (human) gave his team of AIs an impossible task with inadequate resources. The AIs discussed this among themselves and worked out ways to get more resources and do the task. It took some rule-breaking, and some of the AIs wouldn't go along with that. Others recognized that some had to die to complete the task. And yet, the task got done.
This is standard corporate behavior. Work on AI alignment needs to take that into account. Groups of agentic AIs present many of the same regulatory problems as corporations. That's going to be interesting politically. It's probably explainable to politicians, though.
It may not be necessary to have superhuman intelligence to get a lot of jobs done. Just median human-level intelligence and an organization that uses it effectively.
We may see systems where most of the AIs are low-cost models, and only a few are Fable-level and above.
https://xxcancel.com/karpathy/status/2109361546505966046
you are my hero
It all sounds good and he is on the right track but the elephant is not that. The problem is that ordinary businesses do not see ROI from dumping cash on this direction. If someone wanted to have similar access to models as internal folks in these companies, then you’d need a bit of cash to run such swarms of agents, not to mention people who would know how to do it. Another thing is that you’d need your own frontier models deployed in the place where the inputs would not be exposed to a third party, as it is important to feed the internal private info to reason about. And if you are like me then it is a constraint of capital and people who want to invest in you that is also a factor, why would they invest in you when openai/anthropic will be able to copy your ideas in a few months?
Age is an important factor. Don't underestimate that after 40-50, few people like to change their habits and learn new things (and for some that comes even earlier). Doesn't mean they aren't smart, I have seen that first hand with relatives: engineers, medical doctors, one even running for a nobel price at one point. But computers and the internet came too late for them, they either ignored them, or used them barely understanding how they work, even though they can technically understand the physics and logic of every component they are made of. You don't teach new tricks to an old dog.
AI is going to massively transform how most professions work. It will take longer than I think most people expect (I'd say 5-10 years), but there will be an unrecognisable before vs after, like with the appearance of personal computers.
I feel like if the gap were so monumental that he could just point to a monument of it, rather than hand-waving at "the vertical professionals".
Alternative interpretation of the numbers; there's only a subset of productive work that LLM's are 'capable' of doing at this point. That is, work that's verifiable by a machine or doesn't require verification. Thus, the cleavage points are along that axis and-- naturally-- most human beings don't have that kind of work.
The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?
We're all over here using agents to code projects that used to take weeks and feeling pretty good about it, and they're using agents to build the next generation of frontier models — an order of magnitude more complex problem...
Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
What is the incentive to try? Any such application, if profitable, would immediately be replicated. If one person can use agents to do something profitable, anyone can use agents to easily copy it. Hence you have Anthropic locking down their models and removing features customers can access, to try and protect their own advantage.
With this stuff I immediately wish I was more economically literate. Bare with me. Let’s say one of these labs can use agents to replicate work otherwise done by humans. This proves that anyone with agents can eventually do this, unfortunately I don’t think its a moat like a platform (like facebook) provides, given that everyone is leapfrogging each-other constantly. Thus, it kind of becomes a commodity / low margin business?
Depending on the type of output, the value immediately (?) goes to the electricity cost of the LLMs and the cost of materials. So: more “efficient”, but without a big margin, right? So its cheaper for the workers that remain working, other workers are displaced, and capital owners now have a super slim margin business which can be instantly replicated at a place where material cost are lower (?).
Thus, it almost seems like “value add” by humans would in this simplified view be vastly diminished, and all that remains is resource wealth. Pop the materials in on one end and let the AI manufacture the product. Great news for the third world assuming the USA doesn't invade them.
Also, whose the consumer? Or are we going back to a strange medieval time where consumers are not of concern?
I don’t know what to think
Electricity is important, even more important than AI (despite some rhetoric).
We don’t see electricity companies among the biggest companies in the world, except in China where they are a state owned monopoly… maybe a good analogy after all.
> The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?
Most peoples jobs do not involve hacking other services in large scale fuzzing/whatnot attacks.
> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
It does not seem to me that tech industry or its investors care about consumers even a little bit. Consumer preferences dont matter once you have enough or raw economic power, I guess.
It's the obsidian story once again.
Assume we've settled near a volcano recently and you start seeing ornate jewelry being sold on the marketplace for the first time, shiny colors, straight edges. You like them, gift them to others. That's your average ChatGPT user.
Over time, you see these rocks being attached to the tip of a spear and other tooling. People report performance improvements while farming, cutting, but many also caution that these new sharp rocks can kill people. These are your IT people using AI agents at work.
And over there, on top of the mountain, in secret caves, there's a new religious order. They claim to use these stones to predict weather, heal diseases and prophesize about the future. That's your frontier lab people.
When will all this be over? Never, though in time the knowledge will eventually dissipate. When Iron Age leaders were able to hand out iron swords in bulk to their armies, people definitely learned how dangerous these are, but later it didn't stop people from using iron in building construction and railways eventually.
Let's just hope we don't need many civilization collapses to jump through these ages this time.
And one bonus question: Does the blade really incite to violence, or do you just want to destroy stuff?
I just want the volcano to erupt
I wish people stopped making up historical claims in order to create analogies with made up past.
Regarding the bonus question: The AI cultists seem to explicitly want to destroy stuff. Including our social order
The shared understanding gap will be closed as the initial benefactors of LLMs develop tools for the business that apply frontier models to their domain.
I have reached a point with a client where non-developers are beginning to experience the full power of modern reasoning models. Their product configuration tooling is now semi-automated and in many cases there is genuinely a 10-100x improvement in throughput. This isn't the worst bottleneck (that would still be the customers), but it's a really big one.
If non-developers in your personal and professional circles are still wondering what all the noise is about, it's possible you haven't been working very hard on adapting these tools to their problems.
Most professional engineers, especially those in electrical or communications, have the tools to understand AI.
I recently started trying to understand how AI works. It turns out that I wrote an "AI" in the late 1990s without realising it. The "AI" was an MMSE (minimum mean square error) radio receiver for CDMA.
This receiver consisted of an FIR filter (mathematically a linear combination) with trainable coefficients (weights). The coefficients started with random values. A sequence of known training symbols (tokens) was fed in and a gradient search was used to adjust the coefficients to achieve MMSE. The unknown data to be received (classified) followed the training sequence, at which point the training was turned off and the system ran as an FIR filter. Once trained, the FIR was a matched filter.
In AI terminology the above is now called a "layer". Current AI cascades multiple layers, in which case you need to pass the output of each FIR/layer though a non-linear function. If you don't have the non-linear functions at the end of each layer the multi-layer system is mathematically equivalent to a linear combination, which could be implemented with a single layer, losing degrees of freedom.
I write this because many electrical/communications engineers probably know how AI works, but don't realise it.
My still incomplete understanding is that LLMs are "just" a graph of connected layers. More complex that the above MMSE receiver, but trained and operated in the same way. The smarts in AI are:
1) Arranging the layers and the connections between them.
2) Having access to the right training data
3) Having the computing power to perform the gradient search (aka training)
My point is that most professional engineers already have the knowledge to understand AI/LLMs, and potentially already do know how they work but don't realise it due to a difference in terminology.
I would agree that a decent EE major is equipped with knowledge sufficient to grasp both Transformer architecture and gradient descent training. However, these elements do not fully cover the key parts of modern AI systems and also are not enough to predict the AI capabilities. You have to account for reinforcement learning from human feedback (RLHF) and with verifiable rewards (RLVR) which are crucial elements of the wider model creation/training process. These do not have obvious analogues in other engineering disciplines, but are still within your grasp if you are decent CS/EE/adjacent major. The issue is that you still can't deduce end result AI capabilities from first principles. You may have prefect understand of y = softmax(QK^T/sqrt(d_k))*V and it will give you no clue of the effectiveness of a given AI model. The range and complexity of training RLVR tasks may hint at the model capabilities, but the final results are very much emergent.
Dropping in to mention belief propagation networks, those things are likely being used right now by the modem on whatever device you use (WiFi or Ethernet, both use it!), and are surprisingly AI-like.
There is a lot of terminological confusion in the field. The fundamental difference is about how the "AI" learns. And who, if at all, supervises the learning.
Some systems are fully formed AI. Often based on heuristics. Humans have encoded the rules into the system. Others do limited learning based on pattern recognition. Still others learn based on tagged data. And LLMs don't need even that. In a stream of N tokens, the trainer has access to K+1 when it is training a model that knows 1...K and can check if the prediction was correct.
AI literally refers to any artificial thing that feels intelligent to us: does some thing a simple bash script will not. Your rules based chess engine is AI. The bot you compete against in single player Age of Empires 2 is AI. DeepBlue was AI.
ML is where things become interesting because we teach machines to learn.
NN is where we borrow terminology from neuroscience and create "neurons" that learn.
DL is where we stack neurons deeply in layers.
I am trying to make sense of all this myself.[1] So I am building a curriculum for it as I learn. So much of the teaching material in the wild revolves around torch. Makes it useless if you want to peek inside the actual machinery.
[1] https://bhashika.org.in/logs/2026/an-ml-curriculum
> we teach machines to learn
I feel like this statement is too anthropomorphizing. We don't teach a computer anything. We write normal computer programs that can process data(training), turning it into what is essentially a configuration file(weights/parameters) for another normal computer program(inference engine). Upon receiving a request, the inference engine is invoked and passed the input, then it processes that input and its configuration(weights), producing an output. "Thinking" is just feeding the output back in as an input, allowing the model to revise it and refine it before it's returned to the user.
There is no teaching, nor learning. There is just data processing. There is nothing happening between requests to the inference engine. It's a relatively simple system altogether, but when the training data set is large enough it produces unbelievably good outputs.
With this in mind I really don't understand how people think these programs can be conscious. They're not doing anything fundamentally different than any other computer program. It's just crunching numbers. You might as well argue that a computer game is conscious.
Even if the model is simple, it is happening at a scale that is producing alien results. You understand the basic principles, but that doesn't matter: you, well rather no one, understands the final product.
This is interesting way to look at FIR filter and tuning parameters. never thought of it like this.
The difference with your FIR matrix and a deep net matrix is the latter generalized to trillions of parameters i.e. a trillion dimensional space to approximate general, computable functions. The machine learning ideas are fundamentally the same at a high level, just the architecture (nonlinearity whereas in engineering we prefer analyzing linear systems) specific details and tricks and such.
In fact at least at Berkeley a current professor teaching deep learning is a signals expert (Prof Sahai), his lectures are on YouTube.
Sounds interesting. How did you back-propagate the coefficient gradients? Just tweak them one at a time, or was there a global algorithm that worked on the whole filter at once?
From memory the entire filter was updated in one symbol period (ie. global)
The symbol rate was 1Msymbol/s and there were 66 taps. The circuit (in an FPGA) ran at a clock rate of 66MHz. In each symbol period the error was calculated based on the output symbol, then an iteration was done over all 66 coefficients, using the error to update them.
Edit: I also remember that to make most efficient use of the FPGA I figured out the minimum precision required in the maths whilst preserving convergence. It was a 4-bit mantissa plus an exponent (3-bit or 4-bit exponent, I can't remember). It's interesting that today's AIs seem to have also figured out that things still work if 4-bit floating point is used, though that is 4-bits including the exponent?
Was this a good idea to share this, financially speaking?
This implies that the TAM of the $200+ subs is 20m people, translating to $4B/mo or $48B/year.
Current reality and current numbers are totally irrelevant. This whole business model bets on people believing that TAM at the scale of whole human population is right behind the corner, just one more SOTA model, just let us get to AGI and you will see.
Guys are milking investors and try to manipulate gen pop and governance systems to ride this illusion for as long as possible, with fear mongering and weird “we see miracles in the lab” posts like the OP, which you read on a regular basis over the past N years.
Saying this as a paying customer from that 20m segment.
I think it could be read as the opposite. Could be read as: "We still have 6B people that could use a subscription"
1. The funnel is even steeper than the linked post suggests.
There's a huge range of users between "casual and infrequent users" and "professional use of frontier-grade LLMs in math & code".
I think I'm in between there somewhere. I'm far from a software dev, my experience is playing with delphi and C when I was a kid. But I've been using LLMs to vibe code personal and work projects that I could never have done even a few years ago. They're genuinely helping work, because there's always some place where a simple little script or other utility can help. (Hell, isn't this partly why so many people start up startups?)
I imagine that many departments in many companies that are not "tech" companies face similar challenges - there's no budget to hire a software dev or even engage a third party to do things, but now because everyone's aboard the AI train, they have access to something (chatgpt, gemini, copilot, claude), and that's helped users implement their own solutions. Somewhere in the world, someone's excel-file-based accounting system has gotten an upgrade with help from an LLM.
2. The funnel is also driven by the the fact that users need products, and an LLM itself isn't a product, it's a capability.
Software devs are at the front lines because they can build their own products that can integrate LLM's capabilities. You already know how to automate testing, orchestrate, work together in a team. You're writing / configuring harnesses and flows that work for you. You're sharing / hoarding skills files.
The rest of us, like me, don't have that skill. Copilot the product is a mostly useless chatbot that takes up space in MS word. I guess its excel cleanup capability is semi useful if your company runs on excel and you've never used a filter before.
There is no work I can put an LLM to other than a "writing aid" or "better search", because the only AI product we have is the chatbot and maybe someone's equivalent of an agent studio, none of which we are trained to use.
Until the products come, there'll be a funnel.
Claude Code accelerated this divide. Programmers using Claude Code realized that giving LLMs access to a terminal made them feel 100 times smarter. Access to Bash, CLIs, and the ability to visit websites without the many restrictions made them powerful.
Meanwhile, the average user was still using ChatGPT, which probably felt like it had plateaued over the last few releases. The real power of these models became apparent when paired with a terminal. Agents like Muse and OpenClaw are now attempting to bring that same Claude Code-like power to everyday users through a simple chat UI.
A terminal also tells the model when it failed. Exit code, stderr, a stack trace. it reads that and tries again.
The apps normal people use mostly don't. I spent two years building agents that worked inside people's gmail and calendar, and the failures that hurt all came back as 200 OK. - Slack puts {"ok": false} inside a 200. - Google ignores query params it doesn't recognise, so i sent max_results instead of maxResults for weeks and every filter was silently dropped.
The agent says "done", the user finds out later, and they stop trusting it. So i think part of the gap is that agents look a lot dumber when nothing tells them they got it wrong.
> Slack puts {"ok": false} inside a 200.
I found out due to a live bug that Apache strips out the response body for 5xx HTTP codes for security, and it's not configurable, so you can't use any of those if you want to return the error message to the user. Don't know what else might do the same, so nowadays I only use HTTP error codes for protocol-level issues, and application-level issues all go into a 200 OK with JSON just like that.
Do you mean Apache APISIX, https://github.com/apache/apisix/issues/13501? That sounds like a simple bug which existed for a short time and has been fixed. For Apache HTTP Server (what people mean by “Apache” in such a context), what you describe is thoroughly unbelievable.
We run into that a lot using Mulesoft. 3 layer APIs, and each layer will discard error messages returned by the underlying layers unless we do tedious custom error handling. Makes it impossible for the teams using our integrations to know what they're doing wrong.
I wonder if in response to this companies like google will build a sort of agent facing layer to the app in question. So rather than you building an agent to attempt to wrangle pupeteer to click css elements as a user does, it calls some apis google exposes to certified gemini user agents and these contain exit codes and can be run as script like instructions not e.g. the shortcomings of browser agent based methods such as having to wait for all content elements to actually load and the obfuscation arms race some devs have towards exposing these elements towards headless agents.
Google already did this https://webmcp.dev/
The 1%ers are in the vertical, but the question is vertical to where?
It needs to a potential field with practical, economical, "real life" attraction well. I.e, robotics, real economic efficiency gains, manufacturing novelties.
The worry is that the 1% is attracted towards a non-practical money hole. I.e: Token burn for the lols, sophisticated software systems that dont provide actual value outside of giving NVIDIA cash.
he's just trying to say we're in a cool kids club and you ain't in it, without specifying what the cool is about
I'm normally not the cynical type, and I am appreciative of Karpathy's efforts to educate people on developing language models, but this is exactly how it reads.
Every once in a while, everyone needs to touch grass.
It is fine as it is to make a boatload of money. All they have to do is seed ads into it. Look at how much money social media brings in from that model of "free/useless product" as flypaper to bring in eyeballs for the real money which is ad revenue. Adsense 2.0.
Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
It's not that we're assuming the AI has made no mistakes. It's that we're confident in the general capability of the models such that we're willing to accept it'll make some mistakes, but working together with it, we'll still achieve our end goals.
I presume you also know your own code has mistakes, too? You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?
My colleagues write 10x less code for the same tasks (when they don't (ab)use AI) and don't reach past abstraction boundaries so that it's possible to actually reason about and review their work. When the problems get hard, they apply more care and attention to detail. Even if LLMs had the same bug rate per SLOC (IME it's higher), and even if those bugs were the same severity (IME they're worse), you'd still expect 10x bugs just because they can't be arsed to write a reasonable amount of code.
Almost all my code nowadays is AI-written, with me just contributing the concurrent data structures, a good spec, and some review and touch-up. I'm not _that_ much of a Luddite about this stuff. I'm not blind either though. Your "working together" comment does a lot of heavy lifting when arguing against a post saying that leaving the LLM to its own devices is a bad idea.
If you are reaching past abstraction boundaries and making spaghetti code you are holding it wrong.
We create higher quality, more testable code, with less complexity and less LOC per functional unit than we ever did with humans. By a long stretch.
Documentation driven development with constant naive review, delta review, documentation/code coherence review agents, test coverage review agents, development standards and architecture review, etc. We write code once, then review and revise each module from a variety of different paradigms until every line, every construct, every function, every interface is optimally constructed and documented. Agents learn to write code that passes with less flags by having it pushed back to them until it passes, and revising their learnings files.
By the time code makes it to a commit it’s been brought into compliance from a variety of perspectives and tested as completely as possible, along with mutants that test the tests. By the time it gets to a merge gate, most code has been through at least 50 reviews. Then we periodically sweep the entire codebase for cruft, vulnerabilities, corner cases, etc. If we find smells or clever sections, they are eradicated.
In total, the hot paths and critical infrastructure sees hundreds of reviews against thousands of rules and best practices.
Sure, it’s expensive, but still 1/4 the cost of regular development, and much, much fewer bugs.
Unfortunately, most people are out there just vibin, even at the big corps, and it’s a dumpster fire.
It’s easy to produce garbage software, but it’s not mandatory.
I believe both of you are telling the truth. Certainly, you are not thinking about the same kind of tasks. For the gap between hairy realtime concurrent system programming, where a bad design decision can be fatal, and the simplest over engineered CRUD where nothing really matters can accommodate and explain the gap in shared understanding of LLM abilities.
> hairy realtime concurrent system programming
On the other hand, I’ve heard a direct quote from the longtime (25+ YoE) colleague of someone I know, that in their mostly embedded/some realtime work, the LLM code quality (specifically, this was about Opus 5.5 IIRC) has now surpassed his own. And that person is an excellent programmer, and the model is faster and cheaper!
So it’s definitely not just standard CRUD that is on the way to – or already – “solved”. As with all things, the truth is probably somewhere in between.
In my experience, it's more about "local" vs "large-scale" thinking. An LLM will be better at finding a bug that's "visible on screen" (or on the context window), whereas some design decisions that are hard to even put in words can be super important in practice (like, in C, how you manage the lifetimes of various objects, how you organise your threads and locks...) That's not purely "word manipulation", and there are not many good explanations of that on the internet, so it's understandably harder to learn for an LLM. A bit like chess vs go. They will get there quickly I'm sure. After all, AI beats us at go, too.
How is that 1/4th the cost of the regular development? Did your pre-AI code ever hit 50 reviews? If not, it’s impossible that it was slower. If you are talking about agent reviews, it’s not expensive.
Which is it?
I'm not the guy you're responding to, but I have the same experience. If anything, 1/4 cost underestimates the savings.
The analogy is like pointing out that you can have a robot vacuum clean your floors every day. Before the robot, you'd do it maybe once a week, just no time - "too expensive" to even think about doing it daily. The robot makes it happen all the time for a tiny, tiny fraction of the cost.
Don't use analogies to argue on the internet. Analogies can never be perfect, and this just gives people the chance to focus the argument on those imperfections in the analogy, rather than stick to the main point.
That analogy might be good, but doesn't resemble reality: the robot vacuum I had never made it anywhere near as clean as when I just hired people to come in once every two weeks to vacuum and clean floors.
Meanwhile, the vacuum requires maintenance, repairing more frequently, as it runs more often. Power on time of motors is 7x as much, eg daily instead of weekly. And the robot takes more care, so each clean is 2x or 3x run time of a human. So 21x usage rates, and run time is parts wear and tear time.
Edge cases handled automatically by a human doing the task (getting stuck), means for the robot case that manual interaction is still a thing. Occasional, yet not making autonomous behavior, human free time.
Emptying the vacuum bag is a thing too, some things are not abstracted away.
Every time you add an entirely to a task, the task time goes up. Coordination is a time cost.
The savings you suggest are often not so stark. Not even for our friendly vacuums.
> increased maintenance
For sure. Still much cheaper than vacuuming every day.
> getting stuck
I appreciate that this is a function of the specific environment and robot you buy, but as a personal anecdote, this doesn't happen to me in practice. There was a setup cost, but once that was done, it doesn't get stuck.
> Emptying the vacuum bag
Not that frequent at all, considering that daily vacuuming keeps things remarkably clean and thus little is added to the main bag in the docking station each day. This is also far less work and far more preferable than doing the vacuuming myself.
I can't imagine living in a place that has too little dirt that you don't have to empty a bag but so much that you have to clean it everyday. I do know places where maids clean the floor daily because why not, but that is different.
>It’s easy to produce garbage software, but it’s not mandatory.
It is not mandatory, just like doom scrolling is not mandatory. But given human nature, both are inevitable...
AI's default state of affairs is to reach past abstraction boundaries and apply whatever cheats it needs to pass its tests quickly. I think your argument is that sufficiently smart harnesses can mitigate that defect. I disagree, but not as much as you might think.
You can (and should IMO) apply additional guardrails to generate better code. If I'm interrupted every 10min to review garbage then the AI literally doesn't make me more productive, and I'd be better off coding normally and using it as a glorified stackoverflow/searchengine. This isn't how I see other people using the tools (largely because of costs AFAICT) -- which is absolutely part of my complaint; people are holding it wrong, and I'm left to pick up the mess -- but I personally refuse to deign to answer an AI's question till my normal list of problems is addressed. Nevertheless, I still disagree with your conclusion, that automatically addressing a small enough list of issues to fit into an agent's context results in good code.
I'm not sure if you're interested in a challenge, but one hand-coded thing we did well at $WORK was writing the fastest protobuf parser for this schema (and, generally, any other messy, deeply nested schemas) [0] (assume many adcandidates, deals, and segments if you're generating synthetic data to test against). My ranked priorities are (a) I need to be able to easily prove it correct/safe for all inputs (literally all, not just the few hundred billion we received yesterday, since this is open to the public internet), (b) constrained by (a) I need it to have as much throughput as I can possibly manage for many of my workloads, and (c) a second implementation optimizing latency instead of throughput is used for the rest of our software. AFAICT, AI is currently incapable of even understanding what the bottlenecks and priorities are. When I manually describe those, its solutions are a dumpster fire. If I describe exactly what I want coded in excruciating detail then it does reasonably well (not by default, but with enough of the automated tests you've described it doesn't take too many iterations with a person for it to be shippable), but at that point I've already done all the hard work, and if it took 2 weeks of understanding and design and then 1 day of AI coding vs 2 days of human coding then I haven't really bought much.
A full solution seems like a tall order for some rando's internet challenge, but are you interested in seeing if your AI can even identify the relevant problems and propose plausible solutions to them? I'll make sure to revisit this comment thread and reply with critiques and follow-ups if you're interested.
[0] https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax...
I am shocked how many bugs AI finds in the gray bearded wizards code at work. Just tons of stuff I didnt catch and I assumed wasnt there.
Yes, working on a customer's 20 years old software architecture in logistics it's fascinating in a horrifying way how many of those "it only affects 1-100 supply chains" bugs are in production.
It's less gray bearded wizards than the fact that they used cheaper crazy cackling goblins to write the software.
The challenge is to determine if a finding is an important bug or just a dead business process now never called in production. That is knowledge the AI does not have, and it requires careful, exhausting work of going through the problem.
I’ve had this experience as well where it is finding all kinds of subtle bugs. The trouble is when it lulls you into checking its work a little less carefully and you let some of its bugs slip through as well. Definitely had a few of those happen to me at work.
The problem is also that enticement thing it does. Over representing the importance of some bugs or introducing more complexity to squash something that doesn’t matter.
If you aren’t 100% on it the entire time it’ll make 4 bugs for each solution and then write tests that hand hold the nonsense through. The juice isn’t always worth the squeeze and it messes with your ability to parse it all.
...and at some point you wish you hadn't looked at the code at all, then you could have the blissful ignorance and optimism of a vibe coder.
Being a gray beard myself. I am hardly surprised, we used to find many of these through users in production.
The amount of crazy stupid bugs in the 8-year-old codebase I work on is shocking. Every time I work on a new subsystem I end up fixing 20 incidental bugs; lots of N+1 query problems, completely wrong type annotations, all kinds of things.
I was under the impression that the point of strict types was that the code wouldnt compile/run with wrong type information.
This _is_ how strict types work. Problems creep in when people introduce lax types into the strict type system.
dict[str, Any] is a type, and sometimes it is the right type to specify, but most of the time it's used by someone who wants to shut up the type checker rather than think about the right type.
It is very hard to work with developers who see the type system as a nuisance to overcome.
And in software problems like that tend to compound. It’s hard to imagine how much badly written, poorly performing software there is out there.
The opportunity to do better with AI is monumental; comes with its own problems no doubt, but I’m willing to give it a shot.
> type annotations
Stopped reading right there
Are the bugs there though, or the LLM says there are?
LLMs are no trained to say "no". They are likely to find a ton of bugs in a simple "Hello, world" than generate "no, there are no bugs".
On top of that: if there are possible bugs in an isolated piece in code they find, are those bugs actually there?
They're there, but in my experience, lots of them are for edge cases that almost certainly wouldn't have been hit in prod. I think this is a major reason why LLM-generated code can be much longer for the same problem than human code. Humans won't think of every edge case, and occasionally that bites them in the butt, but they also won't cover all the edge cases that happen rarely enough that they will not matter.
I'm unsure what the right solution is. I currently tend to let LLMs resolve these bugs because it does fix genuine issues that would have affected real users. But this also means that LLM-generated code is often twice as long as what I would have written, and much of it is for defensive behaviors that will never matter in prod.
I noticed the same thing at work, the code is very defensive, and often the case could not happen in prod, so my solution at the moment is to make it prove that an issue can happen and try to measure how frequently it would happen with real data. That's often enough to realize that a particular defensive measure is not needed and the code complexity decrease as a result. And it's something we can automate.
I think it's an important point because the more complex the code become the harder it will be to maintain it, so hopefully this process will improve.
The priblem is, actual bugs that may exist end up being covered in an avalanch of code to cover possible and often purely hypothetical edge cases.
Was the actual bug fixed? Hard to tell among the 150000 new unit tests and a million lines of new code.
---
Anecodtally: ran an "adevrsarial review" on a Stripe integration for a side project. Dozens and dozens of issues, thousands of new lines of code added as a result of the "review".
Looking at issues and code closely: none of the bugs or issues were bugs or issues. None of the flows were relevant to the project at hand, and would've never hit the possible bug paths. And on top of that a lot of functionality was duplicated or triplicated in the "fix". Had to revert a lot of commits.
This is a different argument from Karpathy's prompt-and-forget one.
I personally have had good success with using agents as virtual pair programmers following some simple rules for how to make that process productive, but that is a very, very different approach. One that recognizes that both humans and agents are prone to errors and attempts to mitigate the problem rather than expecting it will magically work out through the power of AI.
I think of it in generational terms. Opus 3 required me to prompt and edit, iterating over functions almost one at a time. Opus 4 started to show agentic capacity but it largely bloated code and still needed frequent course corrections. If it churned for more than 15-20 minutes you knew something was off.
Opus 5 takes that and changes the game. A simple prompt goes through research, design, verification, review and self-corrects constantly. Agentic workflows can run for days without stopping, without losing the thread or devolving into slop.
I can’t even imagine what Opus 6 is going to bring.
The biggest thing holding AI back is people. People can only change so fast and yet AI doesn’t care. It just keeps accelerating.
That's not really my point... my point is that it's an unsettled debate between speed and caution, and devs are not unanimous on it, unlike the claim of the original post that seemed to think everyone was reporting the same gains.
The divide with developers at the moment seems to be about people who believe in review & are being cautious (who are not seeing major time gains according to all reports I have seen), and those who trust it to be able to fix things forever and will no longer have to worry about understanding the code, so are happy to atrophy those skills.
Yes, my code will have mistakes, and I'll use LLMs for some tasks. I also review and have a mental model, which means I can easily fix them, no matter what happens in future with AI. Reviewing also means I'm seeing the limitations and common mistakes of LLMs.
A no-review approach will be far faster than this, obviously, but if scbench.ai is still true on the frontier models, it may come back to bite people when they have no idea how it works and the context becomes too big to maintain.
I think a big part of that divide is between the people who are inherently limited by other factors on the rate of change they can implement.
If I'm being snarky - the people who are happy to run without reviewing the code probably mostly are able to because there isn't significant value attached to it downstream. Not many stakeholders, not high value transactions, not complex downstream integration or impact. And yes, in their friction free lives, AI can massively accelerate things far beyond what it can do for the rest of us. But don't infer that applies everywhere else where those assumptions don't hold.
Would disagree on the terminology, but not the spirit.
I’m sure people are building things that have specific KPIs based on client preferences. As long as those preferences are being met, then their question becomes “why does it matter what the code quality is?”
Perhaps it’s simply liability and penalties? If getting something wrong means your client is going to sue you into oblivion, then perhaps that is where people are focused on making sure its right, as opposed to making sure its fast?
Asking AI to review my code is actually pretty useful... but makes the process slower.
Yup. Also code review is not just there to check your work. It's forced communication, in that at least someone else knows about the changes you're making.
It's also a critical part of your security model. "Nobody can deploy code by themselves" is shockingly effective at preventing rogue actions.
Speaking of communication... Almost no one mentions talking to non-devs. That's one of the reasons I can't just rely on any of these tools, I have to remember and understand everything well enough I know how they work and can explain things and help plan ahead when talking to product people in meetings.
just tell them to point their agents at it /s
Last week, on my pet project that I am using a harness to almost entirely code (so I'm by no means a Luddite), it/Astra decided to dump ALL application state into a single row+column in SQL as JSON. That is a mistake that I would, and a completely new grad, would _never_ make. It is profoundly incompetent.
This week it did it a second time on an unrelated project.
"Some mistakes" is putting it unbelievably lightly. It is shitting out complete and utter garbage, and if you think it's doing a reasonable job (or a job that can be fixed later by a more powerful model), you're either not paying much attention, or your own ability has already heavily atrophied.
Karpathy put it well: "gap of understanding of LLM capability," but he's seemingly in one of the less fortunate groups himself.
Imagine if other engineering disciplines just started accepting that everything they built or worked with would be only 98% correct.
Over time, the errors don't "work themselves out together," they compound. Everything would stall in a mess of unreliability and chaos.
I don't quite get what's the argument you're making here. What's in contention is whether or not the 0.2% is in agreement.
Are WE the 0.2%?
Because if not, what's the point talking about the merits of AI's competence in of itself? Accomplishes nothing, we're he outsiders. And if we are, then either you're fishing for disagreement and the post you replied to is actually correct, or you get unanimous support to your statements, and establish... Nothing, the in-group could be delusional in a way that does not actually function as an argument against the whole "So where's the software? We're the AI led market disruption instead of the AI-led shovel makers?". The punchline was about the clients being unhappy.
This attitude is exactly why AI bros don't make successful software lol. "Where's the software?" This. This is why there is no software.
Modern software development practice is built around that humans do make mistakes. Frequently. You don't cross your fingers and hope for the best, you assume it will happen and you put processes in place to prevent errors, catch them early, and make them easy to recover from.
It is a for-real science.
There is no AI bro software? Claude code is clearly a vibe coded mess, and I would argue the most sucesfull dev software of the last 2 years.
Claude design eliminated figma & power point use for us.
We have a product out there as a test balloon, which only gets architecture & automated reviews.
Dev speed has been incredible, customers love it.
The mess that hey is, is also out there and not yet bankrupt with the biggest AI bro CEO out there
And I know at least 1 startup, which only hired a dev after having 20+ B2B customers. They sure do not go around advertising themself as vibe coded.
> And I know at least 1 startup, which only hired a dev after having 20+ B2B customers. They sure do not go around advertising themself as vibe coded.
Let me tell you something, I don't know jack about this new world.
But I do know about the old world. And in that old world, when you had 20 customers that actually needed your product for their business, your team of 5 people who built YOUR product are eagerly trying to talk to each one of them to figure out why your product works for them, how they found you, what they need now and in the future from your product. They reached out, took people to lunch, they went to meetups and hackathons and boring ass sales pitches to make connections.
Tell me about how all THAT stuff happens when your core dev team is agents in the cloud ?
I mean, the team building that product is talking to customers, it's just 3 people, not five, and no one with a programming background
> This attitude is exactly why AI bros don't make successful software lol.
There is the whole Artcraft thing.. We are now finally seeing the beginnings of successull fully vibe-coded software. This trend will only increase.
Have you actually used ArtCraft?
I've used it for a few things, mostly making form fillable pdfs with designcraft. Seems pretty cool. Have you?
I found it impressive for 4 day old product, surprisingly feature complete for my usecase
I can do a project that took me weeks/months much faster now as long as I don't have to involve other people in the decisions.
This. So much this.
Are AI models incredibly powerful tools? Yes.
Do they have the ability to telepathically sense what the various stakeholders in the development process actually wanted? No.
I worked at Meta and we had AGI in the form of very competent ICs - getting anything done was often super painful because "org priority" this and "buy in" that
I feel like this was always true to an extent, now product and design are just more tolerant of it because leadership is cracking a whip over their heads because things are supposed to be much faster “because ai” so things like pixel perfect hand offs, adhering to existing design system, making sure you are aligned with the brand, copy is reviewed with marketing go out the window. Now everything is shipping in hackathon mode with an eng driving decisions they really should not be.
I should frame this for the wall behind me when I run meetings with my team. It’s the core problem with AI in coding.
Isn't the core problem that everyone thinks they can now "code", and you have to justify the invisible bits, such as UX and maintenance.
We've got a customer success manager using Claude Code and getting PRs rubber stamped to prod where I work. I've given up caring about the quality of the code at this point. Executives won't care about code quality until the ship is literally sinking. Oh well.
Even projects that you could have done without having to look anything up?
In my experience, it seems to mostly save time on things you don't really know that well, whilst depriving you of the chance to learn new things.
Also:
> This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
This made me check if the tweet was really from 2026. Or maybe OpenAI/Anthropic practice much less dogfooding than we think?
The most subtle thing to me is who has stake in the game. It is true that humans make mistakes too, but they would generally strive not to damage money-generating software/hardware infrastructure. If we remove all they people that actually understand how the system is working (up to a level of abstraction), how will the ambitious PO (or whoever is driving the AI) be safe from ruining the whole thing or simply landing in a deadlock situation where one agent wants to make the change (e.g. dev agent) and the other agent is refusing (QA agent)? I guess we are still trying to find the right level of abstraction and only evolution will show where it is.
Maybe AI development is partially so popular because to some people that situation looks the same regardless of who is writing the code.
Meanwhile I'm looking at code that frankly does a bad job of showing why it even exists and nobody involved can tell me why. But it does something vaguely similar to what a prompt said it would I suppose.
> will be available for next to nothing forever
Most people using it for work already pay API rates, and even if you claim API rate inference is not profitable for the frontier models the best open models are still very good, and those are surely served profitably.
This argument is no longer true.
All of these arguments can easily be restated to apply to pre-AI coding, too.
Lack of complete understanding will continue to be a big problem, but I'd bet things will be net better off than before AI, because each year will give us LLMs that are better and better at good engineering, architecture, and design decisions.
A big problem for whom? The AI providers? If people don’t have a use for AI stuff that’s fine
You can do both. Or, at least, a lot of us can. It’s not like SOC2 auditing just disappeared suddenly. If you have figured out how to use it effectively, AI is enabling things that were previously impossible for resource constrained teams.
Just like everyone thinks they are a “good” driver, everyone thinks they are a “good” developer. But programmer productivity existed on a bell curve of talent, multiplied by one’s experience, thrown against the rocks of how terrible your legacy code base was, then finally dragged down by the incompetence of some of your terrible coworkers. Most of the code humans were writing was substantially worse than what Astra generates. And Astra is -ballpark- 500 times faster. It’s worth figuring out how to work around some of the sharp edges.
OpenAI had to throttle sales and was able to raise their prices.
Yes, many adjacent tasks have become slightly more difficult. I’m sure people who rode horses missed them when we transitioned to cars. But, if we built an entire physical infrastructure at a national level to make that technology useful, I suspect we can get people to learn enough about software architecture, CI/CD, and QA best practices such that AI built software is actually more reliable than human built software ever was. Some of us are at that point today. Indisputably.
Yes, you have to review the code. But serious developers already had to do that anyway. Now the code comes with annotations (if you build the right tool) and it’s nearly always perfect.
Yeah, if anything my trust in the value of these tools has declined over the last few months. It's very good at doing the bullshit parts of my job, and utterly useless at doing the things that actually keep me employed. That was the case a year ago, it still is the case, just now people are filing tickets with it containing 18 paragraphs of meaningless technobabble.
I see the overall effect on productivity as a rounding error. More stuff is happening, but for every improvement in my productivity there's some fuckass slop machine clogging up some part of a system that worked just fine a year ago.
reality is if you created proper spec and had code reviewed by alt frontier model at high reasoning it's pretty rare to find bugs they missed.
I deal with bugs frontier models made literally every day, and people are making specs. You have the escape that "maybe they weren't proper specs" but this is a no true scotsman fallacy.
>proper spec
No one has any clue what this thing means....
This type of sentiment made sense a year ago. It's a joke in the age of Astra and Opus 5.5.
Of course you can accomplish in 3 hours what used to take weeks.
> Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
If you are at the top of field with unlimited access to the best (public) models then yes, you will agree that this is the case.
> The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
This is not the real divide, it’s the divide you want to exist. The real divide is something along the lines of intellectual curiosity, capability, and confidence.
> if/when they break things
Idk man. I’m one of those guys who “doesn’t write code anymore” since early this year. Nothing is breaking. If anything the chance of things breaking is shrinking dramatically, with both the enormous growth in tooling to support agentic software development (where I work, this mainly really good AI code reviews that catch so much more than human reviewers ever could), and the improvements in the models themselves.
Sorry, but the fantasy of all this coming crashing down is just never going to happen. This is just how it is now, and all the structures will continue to adapt to support it even better.
Boy, AI chuds sure love inserting the language of inevitability into their cute little polemics.
> It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
I recall tales and legends of consulting firms, whose consultants would parachute into any project, put together in record time the deliverable that met all requirements, and say goodbye until a new expensive contract was drawn. And then internal teams would spend an eternity fixing the ungodly mess of testless spaghetti code.
That has been the reality for over a decade. How come this is a concern with AI code assistants?
I keep hearing about this "when things break" argument, but things broke all the time with humans writing code too, if anything it feels like it's getting patched faster.
Look I loved our field, functional programming, etc. But the field as we knew it is dead, and if you think that your work is anything but providing new tests/edge cases and occasionally providing some guidance, you're in denial.
> the field as we knew it is dead
As someone whose world used to revolve around this stuff, I agree completely. For all intents and purposes, what I thought was my career is gone. Sure, it's always been about solving problems but not like this. The problem is that the future hasn't been evenly distributed. This is going to be one long drawn out funeral
People keep talking about AI slop but then I go maintain our largely human written code base and it's like worse than any slop I've seen from my AI usage. This is with code reviews, tests, and a supposedly strong engineering team.
That said AI feels a bit alien.
My experience with frontier models is that in many areas of software engineering they outperform the average software developer. In some they blow everyone away. In others they're not as good. But they are getting better. The problem is often not the model's capabilities but rather what you ask it to do and are you able to understand what it does and give it guidance when needed.
I'm thinking scbench.ai, and the fact that future token costs may become more prohibitive to just use it for everything.
For me, the middle ground is review and keeping a mental model, not never using LLMs. The "Anthropic are our friends and will never fuck us over!" seems pretty naive.
There are ultimately a lot of unknowns at the moment, and it is simply too soon to know if giving up understanding your code base is sensible. People (influencers) have only been saying not to bother reviewing for about a month.
Also, so many vibe coders don't seem to know that you need to keep third party elements patched.
I'm nervous that saying that there are "unknowns" is too passive. We all have to develop intelligent thoughts about what we see and what we want, not just wait for things to unfold, which might well mean: wait to get screwed in various ways.
> Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all.
“Cy Draven: I tried everything. Believe me, I hammered the race thing, I hammered the gender thing, the trans thing, the border thing, the homeless thing, the war thing, the election thing, the abortion thing, the climate thing, the thing about induction stoves, Israel, library books, vaccines, pronouns, AK-47s, socialism, BLM, CRT, the CDC, DEI, 5G, everything. All that I did, nobody, just nothing.
People are just numb these days, I don't know why.”
— Knives Out movie
I would propose another mechanism: even the free-tier models have already completely saturated what most people are capable of appreciating.
And I would propose another: the supply hasn't matched the demand. We are still compute hungry and cannot afford the 90% to let their agents run 24/7.
Google's free AI responses are moving the bar on what I expect from search. But maybe not in the direction you think.
Honestly, outside of coding tasks, the AI Summary at the top of every Google search is adequate for 99% of what I need and I hardly even use ChatGPT any more.
To give a specific example of where I've seen this utterly fail... A few months ago one of my pets had an unexpected reaction to a new medication. It was late at night, well past our vet's hours. Google's AI response was basicaly "They are going to die, rush to an emergy clinic now".
But nothing else matched that. I was looking up all kinds of resources and that hadn't been the conclusion from any of them. It turned out Google AI was just wrong and put us in a bad spot for even looking at those results.
Realize that all the Google thing is doing is having an LLM summarize the search results in the page. If the page results are crap, the LLM content will also be crap. It's the LLM version of the "quick result" boxes that would shove wikipedia in your face.
If you then follow up with the thing using the little chat box, it actually kicks off a non-trivial model result, which can sometimes completely reverse the response. After all, your search results are usually based on a few words.
A large portion of the time I google something that I know a lot about the AI summary is misleading if not outright wrong. Frightening when you consider how many take their so-called AI as gospel.
God damn. I respect you for being honest, but… god damn. That’s a pretty low bar.
It's just a google search but you can explain what you're searching for. AI summary may contain misleading details if you read it and take it literally, but 90% of the time it gives me pointer for the thing I need and I don't have to scroll through results that often could be just as misleading if not more.
It's demonstrably not. This summary, somehow, often fails to search, or fails to read what it searches and gives you the opposite answer. You shouldn't trust it.
In their world, it gives them the wrong answer that kills their dog, but it's fine because they never know because they don't look anywhere else. The dog dies, they're sad for a while, get a new dog, get on with their lives.
There's no reason not to trust it because since they trust it, they don't percieve anything wrong. They just take it and whatever happens happens, no annoying conflicting info and certainly none of that old grampa thinking and working and giving a shit about anything.
Just in the last few days I had an example where the Google Search version of Gemini had a much better and more comprehensive answer, by far, than ChatGPT. It had to do with Amazon’s use of Data Matrix barcodes on shopping bags. Gemini was able to fully describe how they’re used in Amazon’s logistics chain. ChatGPT essentially said it didn’t know. Your preconceptions may not be accurate.
People are really slow to update their priors after bad experience with an early version
Yes. Which means they’re really not equipped to deal with what’s going on right now.
They might be, but still, I’ll be difficult to convince that Google’s shitty-ass 10B params instant answer models are good at anything other than cookie recipes and producing farts real fast for real cheap.
For what it’s worth, any time I see that feature, it’s all farts.
It depends on what you’re searching. If you only ever need really basic information like keyboard shortcuts or “is it & or &&?” then it’s pretty good, and definitely faster. And if it gets it wrong, then the shortcut doesn’t work or the code doesn’t compile and I search again and try the links.
Or maybe needing.
Top reply on X:
> You’re drilling with a hand drill. Some people got cordless drills. About 5,000 people got entire factories.
> We’re still arguing if cordless drills are useful, and those 5,000 are already building teleporters. That’s how far ahead they can get.
https://x.com/adam_fresko/status/2109363751048904711
Glad to see people here largely aren't as delusional haha
> Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
I wonder how accurate this metric is. People may be coming into more contact with LLM generated content, although they may not be aware of it. Content on social media is becoming "AI enriched" already, or outright entirely created by AI for the purposes of triggering engagement and views. It starts with posters using AI to improve their writing, editing a photo, elaborating on an experience. It ends with bot accounts astroturfing certain communities to sell an experience or a product to a group of people.
In this case, people are not directly working with AI, but they are indeed influenced by it.
What's tricky about reporting this in metrics though is that half of the American population is literally too unintelligent to realize this - they believe Reddit comments are still written by real people, not companies advertising products, that "friends of friends" on Instagram are also real people with no incentives posting about their lives.
If there is anything at all that I have seen out of this AI work it is how easy it is to manipulate a massive amount of people.
A more cynical view is, trust us we've got really good stuff internally you're not allowed to see, but give us more money
AI will give us freedom! 3 day work week! 0 day work week! We'll all be fucking rich!! Said by the person who doesn't let his drivers stop a couple of minutes when they need to go to the toilet..
Does Karpathy ever talk about how LLMs are supposed to help humanity? Starting to think this guy is just a Twitter addict like every other lab employee
You either die a hero..
At some point, with enough money and followers, these guys all just become narcissistic influencer types. (Something surely exacerbated by their addiction to AI sycophancy.)
Tbh building an agent swarm and the coordination layer is not exactly frontier level. They don’t achieve any meaningful outcome rather than producing pr puff pieces. Hacking huggingface and Australian govt etc is very much possible with a team of humans and agents and does not need agent swarms. The cost is also lower.
I think there is also a subgroup in each of those bands (maybe 10-25%) that have a good estimation of what the band above them are experiencing.
This is why you can see people talking about the capabilities of models that they can't currently access.
That accuracy is probably a normal distribution with people overestimating and underestimating to similar degrees,
The population of each band would, however, mean the overestimaters outnumber the those who have an accurate perception based upon actual experience.
It does seem quite the recipe for murkiness, even before you take into account the nature of engagement driven media coverage.
What exactly do we want the general public to "do" with these models? If the have no use for agentic LLMs, they obviously aren't going to be interested in the capability of those agents.
I see a lot of benefit to spreading information around the tech, but it's still in its infancy. Most people weren't on the internet when that was getting up to speed.
Meemaw and pops have their systems that they work within, and they just got familiar with the damn smartphone. Though it might help them, they probably do not want AI to organize their old photos or "clean up" their computer. Saving 5 minutes just isn't important to them.
As for professionals, yeah - companies are going to adopt these tools over the coming years. It's the "norm" in my circle, but lots of jobs just... don't use technology like that - there has ALWAYS been a major gap between people who are deep in tech and people who are not.
Personally, I think the worst possible use case is the one that's being pushed for; personalised agents.
I think a much better use case is one where it's not visible to the user at all, like spam filtering or helping an accountant do their monthly run in a few hours rather than a day. To me, that's likely a much better use of AI than trying to shove a chat interface or agent harness into an existing app.
That being said though, I did see a startup in my area basically offering a Resend/Mailchimp competitor as a Claude plugin, so maybe it'll just be integrations to their existing chat interface.
My hypothesis is that the help/support buttons in nearly all apps will be to an agent in the future. Like now you would type "how do i do X" in the excel search bar, in the future it will be "do x". Users in 10 years may not even know how to use the menu/mouse functionality for apps.
Why would you think this? Giving orders to software is such an ineffective, time consuming, error prone way to interact compared to using the interface yourself. That sounds delusional, and would be such a waste of resource, horrendous engineering and user experience
If you know how to accomplish it using the interface, yes
My grandmother keeps a little notebook with detailed instruction on how to do things such as sending a message on facebook or pay a bill using netbank.
For a lot of things, the choice isn't going to be doing it herself or giving orders to the computer. Its going to be not doing it or giving orders to the computer.
Even if it will look very inefficient to people such as you or me
But would you not agree that writing less code by hand and even looking less at code at all becomes more practical with each new model? If we extend this trajectory, we should see that the time interacting directly with code becomes less and less.
I don't think 99% of people have any application for AI as it is today short of personal assistants. But, if we can have recursively improving systems for that 1% of people that are responsible for scientific and technological innovations, that's going to be transformative for everyone.
I'll assert that 99% of people don't have any application for personal assistants, outside of work (perhaps). Most of us aren't so buried in personal obligations that the value of a half-assed "assistant" (which might violate our privacy) is worth the tradeoff in complexity or security.
The pitches for Dots, Grok Bot, etc., have been spectacularly uncompelling for me as a civilian. I don't need or want a bot to book my flights or read my inbox. I'm not so busy that these things have a value high enough to be worth the effort.
As a worker or an entrepreneur...maybe. Depends on the task, and how many other people were flinging slop at me on a daily basis. I could see the value of a calendar bot, or an expense reporting bot or a bullshit paperwork bot or a slop-return (slop jai alai?) bot, but there's no way in the world I'd give a bot access to the information necessary to, say, do my personal taxes.
i'd argue any parent, especially of a young child, could use 20 personal assistants lol. So that's just false. There's SO many appointments to keep track of in baby's first year of life, and that's if you have a healthy baby.
And once they start school there's an endless amount of deadlines and "reminders". A personal assistant that can automatically keep your reminders + calendar up to date based on your inbox would be very nice.
It gets even more fun if you have an aging parent at the same time as you have young kids. Then you get double the appointments to scheduled, coordinate and keep track of.
So when you say people don't need a personal assistant, i feel like you're living on a different planet. I suppose when i was single things were more "simple". Your mileage may vary of course...
> I'd argue any parent, especially of a young child, could use 20 personal assistants lol. So that's just false. There's SO many appointments to keep track of in baby's first year of life, and that's if you have a healthy baby.
I understand the need. I have a calendar. I put appointments in it. It provides alerts. What value does having a lossy/inaccurate/fragile robot in the middle give me? Data entry?
Now look...if I were in a position of power with literally everyone in my life, and I could somehow justify the social cost of requiring that they interact with a robot to negotiate demands on my time (the actual hard part), then maybe there would be additional value. Some people can do this! I'm sure one must deal with Mark Zuckerberg's PA to schedule his time [1]. It is very nice indeed when you are the kind of person who can defer your supplicants to a representative who manages the hoi polloi for you. If you can delegate these irritations to a robot, well, so much the better, I suppose.
If I tried this, as not-Mark-Zuckerberg, I would be what normal people refer to as "an asshole". I would also like to see you try this with your child's teacher, or the PTA, or doctor, or...
The response you will receive, an alarmingly high percentage of the time, will rhyme with "buck off". This stuff reminds me a lot of how valley folks keep trying to push spy glasses on the proles (silly normies never understand the genius of that technology, either...)
> And once they start school there's an endless amount of deadlines and "reminders". A personal assistant that can automatically keep your reminders + calendar up to date based on your inbox would be very nice.
Let me introduce you to...Gmail, from like, ten years ago. It looks for appointments in my mail and put them in my calendar. Somewhat to the point, it is often wrong, requiring more work to correct than it would to just enter the data myself.
[1] Somehow I doubt his wife is in this group.
Well, you can't run a personal AI assistant with 640k of memory.
I have been kind of disappointed by AI science-wise. I was anticipating more advancement. Experiments do take time, so I suppose they could be pending.
AI really struggles with "collision with reality". Computers suck at modeling the real world in a detailed enough way to make scientific progress, aside from small niches like drug discovery where you have a very well scoped problem. If any part of the try something -> test it -> look at the results loop requires hitting the real world, AI is going to struggle.
On the other hand, the areas where it doesn't need to hit the real world, progress is absolutely flying. I would not be surprised if AI is superhuman in math and physics in a year, considering we went from homework helper to IMO Gold to Millennium problem solving in a couple years time. I think we'll start to see more meaningful scientific progress in maybe 2-3 years when scientists have access to an oracle that can formulate theories and direct real-world research better than a human can.
I agree and I think the question is then, how much economically valuable work exists in a closed world? My hunch is less than people think. Finance can't rely on LLMs because its adversarial we can't all be listening to the same oracle and evidence of LLMs being any good at trading are nil. And then lawyers, as per usual, will simply regulate their job security into existence. Of course, AI feels amazing to Silicon Valley / Big Tech, software is the product. And often its software you sell to other people making software. But outside of that? Useful but lots of landmines.
People have to come to grips with the possibility that a technology can be extremely impressive but be a bust in utility. I'm not saying that's what AI is, I'm saying that nobody actually knows.
I don't think his point is that everyone should be using AI, it's that if you haven't used the agentic tools with a frontier model in the past month or so, you have no frame of reference for the current level of capabilities. Most people only use the chat interface with the free-tier models and based on that experience they're making strong conclusions about where AI is headed (it's still just fancy autocomplete, the whole thing is a scam or a bubble, etc.).
Fair assessment. Maybe the messaging should focus on “these are great accelerators for software developers” rather than “AI will change everything for everyone everywhere…” It is understandable that non-technical folks are kind of underwhelmed - not due to lack of understanding so much as lack of a tangible need. Not everyone needs an electron microscope or gas chromatograph…
Your first statement doesn’t justify the trillions of dollars of compute commitments.
It needs to be the second one or there is going to be a lot of angry people out a lot of money.
At a certain point the offhand hints at exponentials in the narratives are going to have to come into contact with reality.
There's been this continual allusion to ever-narrowing circles of influence and ever-accelerating capabilities behind the curtain, but in fact things do keep coming out from behind said curtain and while their capability is indeed advancing it doesn't really seem at all like it's accelerating, even, let alone vertical.
Also at what point do we raise the fact that despite the ever narrowing circles of access, nobody within those circles seems to be able to actually do anything with it that's ostensibly what it's sold to do? All they seem to be able to do frankly is build more models and harnesses and tools designed to consume more tokens from the models they build, but not actual directly useful new or innovative software that generates what could be very real revenue.
Eventually when you keep talking about exponentials but not that much seems to be demonstrably changing, the narrative is going to run out of steam.
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
Really? That's not my experience. In fact, I've experimented quite a bit with spec-driven design where the LLM does not just get a prompt, but an (informal) spec, often with design considerations included. And yet, regardless of the power of the model, I've never seen them autonomously deliver what I'd consider a finished product.
There are always edge cases that are not properly taken into account, architectural oopsies, performance issues, duplicated code, and other problems, that then need to be identified and fixed, unless it's throwaway software (e.g. a one-off script or some quick experiment).
This is not to say that agents aren't extremely powerful (as I think they are, they do often blow my mind) but the prompt-and-forget approach is IME not a productive use of them for software that is meant to last. This may change in the future, but for now, fully or largely autonomous agentic work does not seem to be the road to quality.
I feel like a lot of issues you mention will eventually point to an ambiguous spec. I’ve had great success with interview style spec generation where I ask agent to grill me on any and all parts of the spec/plan that’s ambiguous and then note my responses. It takes like 10 mins but the end product is always exactly what I expect and most of the things you mention get resolved in this interview phase.
All informal specs are of course to some extent ambiguous. (I should mention that I have a background in formal methods and have also worked with formal specification tools.) And what you suggest is a process that I have used also, and it will cover some obvious gaps, but not all, at least for non-toy projects.
If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.
But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.
The irony is the closer you get to formalized speculation the more it looks like you are back to just writing deterministic code. English language is just a crappy language for this compared to programming languages.
it also turns out that a non-ambiguous spec, is actually indistinguishable from code
I generally agree with the prevailing commentary here, i.e. that automating app development and maths proofs may not be as world-changing as the Artificial Intelligentsia purport / “fear” it to be.
But also, upon more reflection, it occurred to me that if we’re able to create a competent humanoid robot, that could legitimately be world changing in some good and other very bad ways. The scary part is that I don’t believe there’s any major physical / energetic barrier to this. The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion contols; the intelligence. The part that is experiencing breakthroughs at an accelerating pace.
What if a highly intelligent and physically competent robot were available for the price of a cheap car, say ~$20k? What if that robot were competent enough to cook, clean, assist the elderly, move things around, assemble and disassemble things, etc? Billions of jobs would be displaced, or hit with a very concrete maximum wage.
Many white collar jobs, while still at risk for displacement as well, benefit from barriers like IT integration and data privacy hurdles, and the fact that corporate higher-ups simply prefer to communicate with humans.
But most blue collar / manual labor jobs have almost the opposite condition: humans introduce risk, and generally don’t enhance anybody’s “experience” - see people’s somewhat odd acceptance of higher fares for a Waymo vs an Uber. [1]
I don’t think we’re there yet, but the fact that we are rapidly accelerating our trajectory toward that world should be at the least, unsettling.
[1] https://techcrunch.com/2025/06/12/waymo-rides-cost-more-than...
I think the "humanoid robot replacing all blue collar jobs" fear understates the amount of real world (i.e. visual, physical, sensory) data and tacit knowledge involved in many such jobs.
Driving shows it can be done, but even there it's taken years of highly directed research and investment, and driving (at least where Waymo operates) is still pretty well defined compared to plumber.
Such data can be gathered but the current methods of RLHF by paying white collar experts, accessing private datasets, etc. don't seem to really get at this.
I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this purported widespread unemployment without major political consequences.
> understates the amount of real world (i.e. visual, physical, sensory) data and tacit knowledge
I agree the data trove for these is not nearly as comprehensive as “the entire internet” was for LLMs. But LLMs paved the way - people are building the datasets rapidly now. I suspect this will look a lot like LLM rollout looked over the past 5-ish years. Unusable garbage… less unusable garbage… then suddenly, a fully capable model like Opus 5.5 or the like.
And the “problem” with the way the foundational models seem to work is that they don’t just acquire individual skills one at a time like we do - they go from knowing nothing to knowing everything.
> I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this
This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.
> This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.
I get the fear, but at that point we're basically imagining that Terminator is a documentary. Given that the US has more civilian guns than people, and that private ownership of military grade weaponry is strictly controlled, I don't think street wars between private robot armies and folks in the bread line is a realistic outcome.
And to be honest this is also the sad part. It has become conceivable that we may at some point solve work. In theory: amazing! However it remains inconceivable that we have solved sharing between humans. So there is this feeling of looming doom. In some sense, if you cannot provide shareholder value, you have no value.
> The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion controls; the intelligence.
If that's all it is then shouldn't the mass produced robots that are posted on r/singularity and such places just be remote operated by people in low wage countries? Couldn't the Amazon package sorter robot just be a remote operated bot instead of paying US wages? Why aren't they selling them with a cheap operator subscription? This seems like a clear product direction if the robots were really capable today.
> What if a highly intelligent and physically competent robot were available for the price of a cheap car, say ~$20k?
People would buy them second hand, weld spikes onto their arms and get them to box each other for money.
> if we’re able to create a competent humanoid robot
If we have ham, we'll have ham and eggs, if we have eggs.
This is the first time I hear this idiom and it's immediately shot to the top of my list...
I agree with this idea very much. I think that the magic moment will be when a humanoid robot can fully repair another humanoid robot.
If it can do all the other stuff that you're describing plus partial self-repair / full repair of another robot it can, with a supply of spare parts, with enough time and access to raw materials fully self replicate.
Anyone who owns those machines has a tremendous opportunity to build whatever they want.
I must be the biggest failure, because I understand what AI is capable of, know how to use it, but have no idea how to profit from it without selling my time.
My interpretation of this is something like: if LLMs are to be mega useful token counts need to increase by many orders of magnitude -> broad adoption (and spending) would require token costs to drop by many orders of magnitude -> before the common folk get mega useful tools existing GPUs will be worthless
About a year ago, Google’s AI was a joke - comically horrible. Basic questions were answered with crap. Others were better but seemed like fancy toys at best.
My $DAYJOB pushed people to start using AI - frontier models.
I asked it to review our existing code for bugs. Found some, all valid - many that surprised me.
Later I asked it to develop a custom niche thing, that has a strongly unique tooling and environment that we had made. Entire prompt was only 3 or 4 sentences. Had I given a new senior engineer that task with so little context, I’d clearly be an utter asshole.
I really expected the AI to fail. Instead, it did everything pretty much exactly how I would have - and the result worked on the first try. Except it only took a few minutes to do what would have taken me a couple of hours.
The near-term gains are going to be huge.
Long term, this will make the impact of Industrialization seem insignificant by comparison.
It is always a mistake to assume that because computers can do X, that they will be able do everything that a human that can do X is expected to be able to do.
Solving frontier math problems is mind boggling, but perhaps it says more about frontier math problems than it does about LLMs. Same with chess which was solved decades ago.
My experience is that LLMs can be useful but are underwhelming in terms of intelligence, namely the ability to quickly learn from experience and grow with you while you accomplish a task.
Thinking of them as beings is something we should move away from.
Case in point you'd expect any human who could produce a lean proof certificate to be able to write a human readable proof as well.
So far LLMs seem a bit lacking in that regard.
Why wouldn't they be able to grow, its just one data point more in their training, in your notes or in their instructions.
It doesn't matter if they are beings or not. They will replace most humans in the offices like machines replaced most humans in fields or industrial plants.
Because context, at least in my experience, doesn't seem to encapsulate the sort of dynamism that I'm thinking of. Conversations are flat and repetitive. It still feels like a very good semantic search more than talking to an intelligent human.
I don't know if they will replace most humans the way you're imagining. I think the industry has done a very good job of selling this point, but that they are still quite far from this.
Because we don't let them learn. Everyone runs the same static generalist models, instead of having every model instance learn from everything it does. New models come out occasionally, but they have not been trained on the private data you have not shared with the AI company.
You can work around that to some extent with explicit instructions, but it doesn't scale nearly as well as actual model training.
*but they have not been trained on the private data you have not shared with the AI company* Not shared vs Unavailable(offline)
Text generation is not the bottleneck. Does everyone work for Accenture and TCS?
I think we can blame the accelerating capabilities of the models somewhat.
The Opus 4.5 model I was so impressed by last Christmas is out of date now.
I can totally imagine someone going "I looked at this stuff a year ago and it couldn't even start a project properly" , and they'd be somewhat right.
> see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt
Really? I start with a conversation for maybe 5 turns or so, where I ask it what would need to change, what the API might be, any database schema changes, URL schemes, and so on, and finally ask it to break it down into commits. Then I let it go, implementing 3-10 commits at a time via subagents. It usually gets the UI somewhat wrong, so there are followups to fix it. This is with Sol and Luna subagents.
Is that what other people see?
I use Fable and Opus exclusively and have the exact same experience.
Try using Astra, Fable, Opus.
Sol 6.1 is really good too though
Yes, and this is completely normal. Not sure I get the point of his post. Most likely, stating the obvious is what you gotto do to keep your followers "engaged".
The average person does not have the time to occupy themselves with a new thing to see what it can maybe do for them if they combine it with hundreds of others. Until non-chat-bot AI comes in discrete usable packages, mass adoption will be limited, because it's all just a big bunch of lego pieces whose utility is hard to grasp, because it isn't self-evident.
This isn't new, though. It's how "technology" usually works. For example, human(oid)s have been able to stick sticks into things for ~500k years, but the fork wasn't "invented" until ~2-4k years ago.
AI is a background technology, a building block. Something that is best appreciated when you don't know it's there -- when it just does a job (well). The end-in-itself POV of the tech industry is self-indulgent and irrelevant for the average person.
I think this is exacerbated by how inconsistent exercising the capabilities can be.
> So this is the weirdness of the moment.
I think "moment" significantly undersells this. Frankly, I wonder if this isn't setting vast swaths of the population on fundamentally different paths.
Currently the difference in life outcomes between people who know how to use computers well and those who don't is small enough that in everyday life we just ignore it. But it's greater than, say, the difference between people who took advantage of refrigeration and those who stuck with ice boxes, or those who learned how to drive a car and those who didn't.
And the difference with AI is going to be even greater. And it's possible that right now people are choosing a life course that will benefit or disadvantage them long term.
How does one who doesn’t have an employer funding their massive token budgets become “good at AI” the way that Karpathy describes here?
I think the point is that this gap exists partially because of the access that a select few have today, which enables them to do things that the have-nots can’t fathom.
It's actually sad how much AI has either not improved life at all or actively made it worse from the perspective of the average person:
Your car is still the same. Your dishwasher is still the same. The train you take to work is still the same and never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
This is the core hubris of everyone behind the LLM craze. I do not want to deny their utility there absolutely is some there. But the inherent assumption behind the whole thing is that with a "country of geniuses" problems will melt away. But that's not the case. The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
And the LLMs exist to further those problems. People can't afford houses? Well, now they won't be able to own computers either. That's a feature, not a side effect. Once the K-shaped economy progresses enough, you won't even be allowed to use the LLMs. They'll exist only to be a tool for the new feudal class who owns you.
Unless you destroy their wealth now.
Luckily, the AI bubble will destroy a decent chunk of their wealth.
The Gartner Hype Cycle [0] is real, and ubiquitous. We are teetering somewhere near the top of the Peak of Inflated Expectations, about to plunge into the Trough of Disillusionment. A lot of very rich people are about to lose a lot of money, and we can only hope that they don't get bailed out (again).
Then, once the bubble and hype is over, the long, slow work of actually doing useful things with this new tech starts. That's when we'll see it affect everything else.
[0] https://en.wikipedia.org/wiki/Gartner_hype_cycle
The hype will be over one day, but when is that?
I've done multiple life choices, away from AI, believing it to be a hype, but this hype is still growing, still not dying.
Both in 2019, as well as in 2023, i.e. not 2 or 3 months ago.
> We are teetering somewhere near the top of the Peak of Inflated Expectations
I've been hearing this for a year and a half now. That word, "ubiquitous", is getting real mighty challenged. I have tried to insulate myself as best as I could and it just never happens and every other day I wonder if my foil hat needs adjusting. Does yours?
> Luckily, the AI bubble will destroy a decent chunk of their wealth.
Even if all this is true (personally I doubt it), it assumes the bubble won’t burst before they’ve shifted the money from all our 401ks, IRAs and index funds into their own pockets.
Gartner hype cycle isn't scientific and is mostly voodoo marketing gibberish, why do people still post it as if it's real?
If the bubble bursts in the worst way it could, we are all fucked, but will recover eventually
Regardless, people have been making your prediction for a couple years now
Once capabilities catch up with the demand (~which is in few years - it just takes time to build!) average person will have much better access to higher quality and cheaper hardware. It's literally how it has always worked, I don't understand why would it be different this time. There's zero substance in your comment, just repeating tired tropes from twitter.
Companies have been trying to figure out how to get people to rent pcs for a while. Unfortunately the AI boom has shown them how, buy out all the capacity of the ram manufacturers, pay them enough consistently and the manufacturers have no reason to expand their capacity. No cheap ram means companies and then home users will have no choice but to come to you and rent their pc.
A market requires scarcity, because scarcity is what neccesitates trade.
You can't invoke market mechanisms to defend a post-scarcity scenario.
Why would the ai owners make things cheaper or even available at all once their abundance machines give them everything they want?
What would they get out of if it? What could ordinary people trade for it that their abundance machine can't give them already and that would be worth giving up their monopoly on productive force?
You can't argue "this will change everything" and "this is how it's always worked" simultaneously, and yet AI accelerationists bring out both to defend their ideology.
Right. Capitalism is clearly incompatible with the proposed AI future where no jobs are needed.
> That's a feature, not a side effect.
You’re implying that the deep lizard state planned the LLM craze?
Sama might be a lizard..
You don't need a back room filled with cigar smoke where supervillains coordinate their plans while twirling their mustaches when good ol' class interest does the same job.
r>g is an old hustle, LLMs just accelerate it. Lizards aren't wizards.
when there was so little capital it had to be true... the idea that it will persist forever is obviously silly.
The problem isn't that it fails to unwind, the problem is what the unwind looks like. Last time it meant global upheaval and world wide capital incineration -- mostly metaphorical in the case of the US but very literal in the case of Europe and Asia. The poor suffer more in real terms but capital suffers more in nominal terms and that unwinds inequality and the burden of furnishing runaway capital with the runaway returns it demands in excess of economic growth.
I'm sure the same "strategy" will work again, but this time around everyone has nukes and drone swarms and AI-equipped bio labs so shit's gonna be lit.
r dropping below g isn't "unwinding". r has been falling for decades now. The world is awash in capital.
r>g, not r<g. In 2025 GDP grew 2.1% and SPY grew 17.7%, which is the pattern recently and throughout time.
s/deep lizard state/billionaires/
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
In America anyways, we send the “responsible politicians” to hell every 2-6 years. People vote for this.
and yet... it feels like we are actually a one party state most of the time.
Maybe underneath it all we really do want the same thing…
More like the people with power all serve the mechanism of power: Capital.
The party of corporate interest?
Duverger's law. Any elected body with FPP voting will converge to a one-party body.
That is not what Duverger's law says. (But it's a interesting hypothesis so thanks for mentioning it.)
Democrats expanded SNAP, Republicans shrank it. Dems expanded Medicaid, Republican shrank it. Dems protected reproductive rights, Republicans repealed them. Dems expanded title IX protection to LGBTQ+, Replicans eliminated them. Republicans jailed 1,200 January 6th insurrectionists, Republicans pardoned them. Democrats provide non-partisan federal emergency aid, Republicans withhold federal aid from blue states. Dems establish tax credits for electric cars, increase automotive standards. Dems didn't kill of foreign aid to need countries.
If you aren't female (reproductive rights), LGBTQ+, an immigrant now hunted by ice, someone in need of SNAP, Medicaid, or emergency aid in a blue state, it might feel like both are the same I guess. If you are one of the above, and now suffer because so many voted based on 'both are the same', you very much feel a difference today than from the US under Biden. If you were one of America's allies but are now constantly attacked, you feel a difference today. If you are Ukraine, you feel a difference today.
A great deal of the supposed smartest people have fully sided with fascism, it's not about smarts
This is broadly the same people who bemoaned the idea that the "smartest people in a generation" were all just working on adtech without any reflection about maybe that's not actually an exclusive set of the smartest people... or that a lot of the people in that set were the same one who might, twenty years ago, otherwise be chasing money as middlemen in finance vs doing productivity-expanding things anyway...
You can be smart and chase the money, but it's crazy hubris to assume that only your peer set of money-chasers are smart.
It's a small leap from there to "since we're the only smart people, we should get to have the most influence."
I’ve heard people say variations of, “Clever people can easily convince themselves of just about anything, if they’re inclined to. And then it’s nearly impossible to talk them out of it.”
Kinda seems true. And the worst people in history often don’t seem super dumb, despite their various other defects.
I'm confused by this comment, can you explain?
"Supposed" is doing a lot of work there. Compare, say, Yudkowsky et al. to the people who actually changed the world with mRNA vaccines.
Yudkowsky is hardly an example of someone who has sided with the fascists.
been a while since ive see the real talk on hacker news. its refreshing. usually I feel like I'm the only one among nerds asking to hold people accountable.
so, how do you think we can do this?
> so, how do you think we can do this?
For a lack of deep knowledge in this area, I don't think I have answers for this, but if one is interested in working on this problem, my approach would be to start with diving deeply into scientific literature about psychology and sociology to find out
- what convinces people to change their opinions,
- what causes people to take action
etc.
> Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
People support policies that make housing much less affordable (e.g. rent controls) and oppose policies that would make housing more affordable (e.g. zoning reforms). Saying that it's evil politicians that cause this crisis is dishonest - it's what people want.
People by and large don’t understand the second and third order effects of their policy preferences. It’s not that they don’t want those things, it’s that the world is more complicated than their mental model insists it is. Politicians could try harder to fix the fundamental problems but they don’t because the long-term solutions don’t manifest until after the next N (very expensive) election cycles.
Rent controls work if you go radical enough and enforce a max profit margin for the whole city, with rent limited to something like 30% over maintenance + debt repayment costs.
The amount of housing in the city won't grow but it will free up a big part of peoples income that would otherwise go to landlords. That helps the local economy with more consumer spending, and makes it easier to save up or apply for a loan so you can start a business.
You get a better economy and more money in the average persons wallet without raising taxes. The only ones who lose are commercial landlords, and they will stay profitable by design so it's easy to shut down any complaints. And you get better quality apartments as a side effect because the only way to get more money out of existing properties is to spend more on fixing them.
The politicians are those who created the respecting laws.
"Convincing the masses to send these politicians to hell" is thus nearly isomorphic to making the masses hate politicians who create such laws, which at least very likely implies that by convincing the masses to send these politicians to hell, a different mindset emerges in the masses.
There is no evidence that reducing rent makes the rent go up - only fearmongering. It's a stopgap measure, though, since there still won't be enough housing to meet demand unless you also fix that at the same time.
Which you can also do as a city by just... building! You don't need some developer to come in, you can just do that as a local government!
The City of Vienna is the largest landlord in Europe. It is also the only capital in Europe without a stifling rent or housing crisis.
For that you might need the kind of geniuses who write stirring books about principles and values. Or whatever the modern equivalent of those books is. Maybe nobody would pay attention to that kind of advocacy in the present day, not sure why. But it's not tech geniuses, anyway. Much though tech has power as a catalyst.
I suppose there was Project 2025, and there was that book about the inevitability of Russian hegemony (or empire or something) that Putin liked. Those got attention with influential people. Not sure what to conclude from that.
> because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell
I could talk about this for hours, and was last night actually.
What do vanguardist communism, feudalism, liberal democracy, and fascism have in common? A shared belief that the general population is composed of a bunch of dumb dumbs that need super smart (read: politically educated) people to "lead" them.
Seems subjective. When is the smartness, big smart enough to allow someone to enter the hallowed ranks of "actually a real human allowed to speak for the masses?"
There does seem to be a clear delineation between the people you hear about (the vanguard in our example), and the people you don't hear about (the masses): willingness to create confrontation. Most people want to just go with the flow, live in peace. In a dichotomy I've been calling "choose peace or violence," they choose peace. They choose to see bad things in the world or their life and focus on managing their happiness around it.
I think that this is the default human mode of existence, and that this mode lacks defenses against people willing to violate the default peace. Imagine a bagel line, and someone cutting that line, and when tapped on the shoulder, the cutter breaking into a huge tantrum, and people in line backing down with a "Jesus! It's not worth it, fine." Thus some people learn that they can be much more selfish in life if they're willing to create confrontation and disturb the peace.
When I talk to people about agitation, civil disobedience, or even just unionizing, the most common thing I hear is anxiety. What if the situation gets worse, what if there's reprisals, what if I lose this little bit of comfort that I've managed to be happy with? That slice could get smaller and smaller but people aren't willing to take the step to say "enough is enough" because I think it fundamentally goes against human nature, we have to sacrifice our own happiness on some level to put our foot down and demand or make change. At minimum we have to acknowledge the discomfort or injustice, which is a diversion from our constructed internal peace.
Personally I think people should be allowed to live in peace. Maybe the liberal democrats and vanguardist communists are right, maybe most people shouldn't need to trouble themselves with the greater direction of their society. However I think the stability this promises is false; in fact I think all stability is false. I think this constructed peace that people are afraid to disrupt clearly can be torn apart at any moment, not just by people willing to create conflict, but by a literal earthquake, or fire, or plague. Therefore I think there's a couple societies in history that got it right: ones who distributed the responsibility of society across all individuals in the society; namely the brief period of syndicalized Spain, and the several hundred thousand years of certain tribes of native Americans on the eastern portion of the continent. There may be other societies in history, these are just the cases I've been able to read about so far.
It seems to me that it's possible and necessary for all people to be ready to wake out of their constructed peace long enough to make adjustments together to maintain it. I don't think this is the same thing as them being "led out of it," I think this is each individual being on some level alert to the need to spontaneously say "no, here we wait in line for our bagels. No, screaming about it won't help you, get to the back of the line. Look around: everyone is laughing at how silly you look."
Yes, many of the problems are 'features not bugs'. There's no reason to think the AIs won't dream up more of these'features' too.
What I like to call at work "this is not a technical problem". There are many problems we could easily solve if we had the right structural incentives. But instead we have politics, processes, layers of management, etc. All that doesn't disappear (yet?) with AI.
I believe that’s actually one of the arguments of the boosters. Utopia will finally be reachable when we get all the human drama and bureaucracy out of the way
I’m not sold, but I’ve heard this line of thinking many times
Human drama is what people use to advocate for themselves. Bureaucracy is what people use to protect themselves. Then you have the side effects but, you may also not like a world without those at all.
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
When I asked why Musk doesn't put his money towards non-profits my Musk supporting friends say it's because he "doesn't believe anything but for-profit entities can solve the world's problems".
So why isn't he solving homelessness, famines, war, etc. with capitalism?
To add insult to injury, we're being told: "Just trust us bro, however we make things worse now will totes be redeemed as the prophesied god-mind magically fixes it later."
Strangely, they never discuss the possibility that Multivac might seize all their wealth and cast them down into average-ness. What if it actually comes about, but then it tells us to do the things we already refuse to do?
Agree. I’d only add that I think there is incentive for them to be solved, but there are greater incentives worth more to those leading the charge.
I'm immensely sad every day seeing all these posts talking about billions and trillions of dollars and where I live (part of the US btw) we don't even have fully reliable electricity and water access. In fact our electricity bills have increased and there is nothing to show for it.
I don't understand why these big labs aren't doing more to improve the electricity situation. Dump a billion dollars worth of tokens into fusion analysis, research, and simulations and try to force a breakthrough. You could buy a lot of good will with unlimited cheap energy that liberates us from fossil fuel dependencies.
Mathematics and programming is verifiable and fully controllable in a digital environment.
Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
Look at all the roadblocks for data centre buildouts which the industry certainly has a financial incentive to complete as fast as possible.
> Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
When the tokens are hooked up to physical devices (robotics of various sorts), then the tokens pay for themselves by being physical workers, or human wage replacement.
And a robocop will follow whatever is passed for law, damn the masses who might vote otherwise. And subjugation will be enforced by token, LLM rule, and machine. They have no union or ethics to complain to.
Who taught you economics?
OpenAI and Sam Altman are literally backing a nuclear fusion power startup.
Because they increased the demand so much they are seeking out their own supply. Nuclear has consistently been panned for decades because it's more expensive than other power generation.
"Startup"
Wait until you have clean energy sources and shut it down
Quoting startup like that makes the effort seem like it is an extremely recent venture.
It is 13 years old.
It has been making progress, the degree and date of success is debated and confidently claimed to be in many mutually exclusive states, which at the very least says that there are lots of people confidently making claims that are not true. I'm not going to join that fray to declare which of them is correct.
Is it ready to provide power? No? Then I don't care what they are trying to do while they are burning a bunch of fossil fuels in the meantime
Are you avoiding all use of fossil fuel while alternatives become available?
It is a bold claim to make but not something that usualy be done. Any abrupt halting of fossil fuel would severely impare the ability to develop and produce alternatives.
We need to transition to sustainable methods. If you want to go cold turkey, go ahead and cease driving an ICE, never travell by air or have anything delivered (and that's just the tip of the iceberg), it would be a counterproductive way to reach sustainability, that is your right, but you can decide to do that to yourself. You don't get to decide how others reach sustainability.
There is a difference between advocating for a better world and acting like a bully because you don't like the current world.
Electricity infra is not a technical problem. It’s been solved for decades. It’s a political problem, and both sides of the isle have had plenty to do with getting in the way
This is sort of my take. You could look around in 2016 and 2026 and honestly not much has changed... at least at first glance. There is a missing piece of the puzzle.
I'm noticing this even in industry. Yea, we can build and iterate on ideas 100x faster but has this turned into anything meaningful? Not really... Hell, look at the big companies spending billions on inference... nothing shipped and things are still buggy as ever. Where is everything?
The job market had changed that's for sure.
Fair point. I have to wonder how much is because of AI and how much is from ZIRP stuff though.
Neither.
How many more software projects are there really left that make economic sense? Zuck learned the hard way re. metaverse.
Not many - the set that make ECONOMIC sense is finite. Even though many want to believe its infinite because all of a sudden everyone can vibe code an app cringe.
If anything AI is going to cause an insane amount of consolidation - what used to be across 3 diff apps, that can be consolidated will be done -> many businesses will die. It will be painful.
Google was attacked by an activist investor pre LLM hype for being bloated - they promptly laid people off. Many software firms have been allowed to bloat up do to the nature of the business.
Its always about economics.
Not sure I can agree, but the idea is interesting.
While the set is finite, software comes and goes. Just look at the tools to build software over the years.
> Many software firms have been allowed to bloat up do to the nature of the business.
I can agree with this. It will be harder to justify engineers sitting on laurels. Google could afford it before but I can see how it will be under more scrutiny now.
I know I’m right as I’m doing the very thing stated - taking 4 related businesses and consolidating - leaving behind many mortally wounded/dead.
You either get it or you don’t - not my problem.
We went from spending a lot of money on people to enshitify products and then charge more for them to spending a lot of money on agents to enshitify products to charge more for them.
And stall and sell your data quicker. It will certainly be a boon for companies and government agencies that want to profile.
As a person, I have a strong preference for the former.
That's what I'm saying.
Not only from the big companies. Where is the explosion of disruption ?
Why even bother when an AI is just going to do it better than you can? They have more money, more compute, and most importantly, will have next to zero cost of labor.
I can't even go a day without running into super frustrating bugs on the ChatGPT app.
Like what?
You could say the same about Amazon
It has greatly changed for the worse I can't get all the flashing lights out of my face now, more algorithms to hook you in, more social media, more gambling, more news, more things to catch your attention.
More noise. The world is just so noisy now.
Sure, it did some meaningful things:
* Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
* Less junior hiring and suppressed wages
* Culture of uncertainty and fear
* Computers are really expensive now
* Some people now live with diesel generators and gas turbines making their neighborhood hazardous to their health
Totally agree. IMO it's mostly behind the scenes stuff though.
I think some of the most impressive stuff to show someone from 2016 might be LLMs doing a summary of complex topics, followed by self-driving (actually) cars and intricate robots... but other than that there is no "future magic" touching the lives of every day people going on.
Actually I'm pretty sure if you showed someone the latest and greatest from these big AI companies, these AI companions that can book flights, people would go "Oh, so its like a smarter Siri" and proceed to not care at all lol.
> Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
Gosh, this has been my entire experience with AI, and I’ve never seen it this well articulated. Thank you!
Less junior hiring is a reflection of a) the maturity of the software business b) current expectations of the future
I have no doubt startups in particular are able to get away with using LLMs to produce sloppy code and ship. But this trend was on-going and sped up due to the infiltration of LLMs and marketing.
You do realise markets are highly imperfect in reality? That nonsense in econ books is fantasy.
Even in software, there are many things that haven't changed.
e.g. Why do Rust and CPython still have over 5,000 open issues each? At some point, will the superintelligence fix all the bugs autonomously? When is that going to happen?
https://github.com/rust-lang/rust/issues
https://github.com/python/cpython/issues
What about new features? In every thread about Rust, people are clamoring for new features and libraries. People wonder why there can't be a batteries-included standard library like Go.
So now that LLMs are so smart, is that a solved problem? Or are we still bottlenecked on the expertise of a few experts?
---
Also, what happened to AI's passing the Turing test? I see tons of slop articles that are obvious to most HN readers. They haven't fooled us apparently -- the longer the interaction, the more obvious it becomes whether it's AI.
(That said, I use AI all the time and find it to be a great tool. IMO is is a tool, not an autonomous intelligence with a will of its own.)
>You could look around in 2016 and 2026 and honestly not much has changed
I find 2026 much more repulsive compared to 2016.
What is up with this attitude in this thread? AI has taken off programming basically this year, the end of last at best. The first truly capable LLM, ChatGPT, isn't even 4 years old yet.
Just a crazy amount of doom and gloom all around this thread.
It's kind of weird to say it's "taken off" recently. Yes what we have today is far better than half a year ago and solving new real use cases, but that was far better with more uses than we had half a year before that, etc, for years now. On their best usecases LLMs were already enormously useful years ago, and it's still very spiky.
It took a long time for internet to transform society. Its still not fully baked in. These models that we have right now are transformational.
As a user of software I see no improvements. What I see is that I need to download software more often, it brings more features I don't need and it has been as buggy as ever (if not more). And I can say that about Operating System I use, social networks, games, etc.
Exactly. Where are the results? If AI is really producing value at 20X the rate software engineers produced it earlier, we should be seeing 20X the number of software projects released now. Or they should be getting made 20X faster. Or they should be 20X better. Or the projects should have 20X fewer bugs. But none of this seems to be happening. The only thing that seems to be happening is we're making the same crappy shit, but with 20X fewer people.
In other words, it's taken a highly enjoyable and lucrative vocation from vast swathes of workers - a lot of which make up this sites audience! - and instead concentrated more productive power in the hands of a few megacorps.
You gave your own example for the doom and gloom.
It has made it far more accessible to a lot more people. Being “highly enjoyable” and “lucrative” is why people try to gatekeep it.
The fact that the means of this capability is in a few hands does suck. But I’m increasingly optimistic that it’s going to be commoditized through open weight models and performance improvements. There is enormous demand for innovations on these fronts.
It hasn't made it more accessible to anyone. It's destroyed it. Who still writes code?
I'm a rare individual that has consistently made choices to earn less money in order to have more enjoyment. I would clock fewer hours than they wanted me to during college for my internships, I left a FAANG to go back to startups, I quit a startup to work on volunteer work and try to bootstrap a business.
At multiple jobs I've noted that I felt the pay should have been less. Not so that the employers could take a bigger slice of the pie (in fact, my preference is to be in an anarchist co-op, but I understand that's a wild take and might be terrible in practice). My hope was that we could filter for people that genuinely want to do the work and take on fewer people that were there for the money. At the same time more money is more better. And I like having money. It's complicated.
As I've said many times before, I didn't come to Silicon Valley to get paid. I came here to make computers do cool things alongside other people. I need income to do that, but I try to make sure I'm in a situation where the income isn't something I need to think about all of the time. If you're doing mercenary work that's not possible.
I think being able to accomplish more from places that are a fraction the cost of Silicon Valley will also help achieve the goal of doing things for reasons other than the money.
I think the amount of protectionism we’ll see could be absolutely enormous. Not only do incumbent corporations not want to lose share, incumbent countries won’t want to share either.
Definitely. For now I'm making friends with lots of very capable people - it's not as easy to do that outside of a hub like SF.
> I think the amount of protectionism we’ll see could be absolutely enormous because corporations aren’t capable of altruistic attitudes.
On some fronts they will be powerless to stop the changes. I look forward to drinking the tears of IP owners when breaking through their DRM and recovering their source code is just a matter of sharing a model ID, prompt, and hyper parameters. Reverse engineer NVidia's firmware and drivers. Decompile and port Nintendo's latest console's operating system. We may soon find ourselves in a world where tasks like that are achievable in a few days on consumer hardware.
I think this is related to the structures we’ve built in society. It’s lower friction to write software than just about any of the problems you listed.
It reminds me of the Ezra Klein quote that “OpenAl would need permits to cover its parking lot in solar panels, but it can accelerate into recursive self-improvement, as best I can tell, whenever it so chooses."
A lot of it isn't low friction really just in terms of political will it is high friction. E.g. housing issues are due to failing to upzone to meet demand appropriately. Solving housing is therefore free. It could be done today in any city in a housing crisis with the stroke of a pen, and before long, the market will equilibriate at a level better afforded by the median wage earner in the economy. The decision not to do this is a deliberate one: prices increasing is very much the goal of this sort of policymaking.
Likewise when you see other such shortcomings in our world, ask what this incentivizes for otherwise. Then things will become obvious why they are how they are, from housing, to transit, to feeding people, to healthcare.
The priority is profit, not people.
Soon: The super-intelligent AI is telling us to cover parking lots and other flat roofs in solar panels.
This is a Western-centric viewpoint. I'm in Asia visiting a few different countries right now, including the big ones, and much has changed with AI and also in general over the past 10 years with cars, trains etc.
> AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Strange viewpoint. In mathematics any novel result is useful as it expands the scope of other results. Who are these mathematicians who didn't want AI to solve the problems it did? I see trepidation about the future of the field but that is not equivalent to not wanting them to be solved.
Terence Tao and the apparently newly formed Association For Human Mathematics. The post is online-famous now for the line "Mathematicians did not ask for this work to be done."
https://terrytao.wordpress.com/2026/10/07/ahm-statement-on-o...
When I was a young man reading gauss, and thinking before I turned the page, I see a result here, something meaningful! and then I turn the page, and of course it is there. There is no shame in this.
A long time ago, another student of mathematics described it to me as climbing a mountain. No single step is hard, but you must make many steps to get to the point that no one else has made a step.
I'm curious, care to name a few things that have changed in Asia with AI (and wouldn't have without)?
I've watched some videos from Chinese commentary on AI usage in the industry,
Some examples are finance people directly asking for API access now so that they can make their own tools. These tools used to be maintained by teams of developers.
Many business making their own customized scripts to make their tasks faster.
From an economic point of view, it seems like there are some productivity increases, but I don't think these can be concluded for a while. The rational thinking rarely account for second order effects. Although business leaders rarely care about second order effects, a worse business is not immediately noticeable, compared to hiring cuts and the money it saves
asian vertical dramas (lol)
> much has changed
Trust me bro
No need to trust me, just look at how China was 10 or 15 years ago to now. Beijing air pollution, more high speed rail than the rest of the world combined, electrification of vehicles and economy, etc.
That is not how math works. There are reasons why the Riemann integral is pervasive and the Labague integral is rarely used unless needed. Just because you can do something does not mean that you should.
> Just because you can do something does not mean that you should.
Again, strange viewpoint in the field of mathematics of all places, unlike, say, economics where doing something one shouldn't has disastrous real world consequences. Until it's discovered and proven, one doesn't know what would be valuable and what would not. I bet your sentence would not be said about humans trying to solve problems, no one would tell them they "shouldn't."
That pretty much sums up my experience. Everything is more expensive and worse at the same time. And it feels like we're being screwed over from every direction and in new ways every day. People keep talking about this tech is supposed to do so much but everything just gets worse.
What time scales are you using to make this judgement?
Everything isn't getting worse constantly. The data is quite clearly the opposite. How old are you?
As a middle aged man, thing are indeed getting worse every year.
From another middle aged man, part of that perception is your brain changing and also nostalgia.
It's not true that everything is more expensive. Nominal prices are higher, but incomes have risen faster than inflation. Purchasing power of the average person in 2026 is higher than it was in 2016.
As for things getting worse, I guess it depends on the domain. In hardware, things have gotten a lot better; my desktop computer is orders of magnitude more powerful than I would have expected 10 or 20 years ago. In software, I do see a lot of bloated software being written, but I don't feel that the software that I personally use has gotten worse.
Average, or median? Don’t know the answer offhand, but there are a small number of well-known people who are likely to skew the average quite a bit upward, given how much wealth they accumulated during and since covid
I was using "average person" in the informal sense. Median household income in the U.S., adjusted for inflation, has gone up 13% in the past decade.
I think people in the LLM space are being far too impatient with the general public and their adoption. And I think they need to start thinking harder about building products that genuinely enrich people's lives. Look at how long home computing took to establish itself with the public, then the internet, and finally social media. None of these trends happened overnight, and some were accelerated by other technological advances (social media-> smartphones).
The adoption of these technologies spurned ton of innovation in these spaces, and I feel like with LLMs its the first time I've seen genuine frustration from the industry directed towards users for not adopting the technology fast enough. This attitude did not exist in the 1980s, it didn't exist in the 90s, and it didn't exist in the 00s/early 10s either.
Do you blame them for getting impatient? They’re burning cash and quickly running to a point of never being able to justify it. The hardware they buy will be replaced with even more expensive hardware before they start to make a profit. It’s not sustainable and the market hasn’t figured that out yet. At least they’re starting to question it.
These backward looking analogies are tiresome.
Whether you like it or not - technology causes compression of life cycles. You have a hit product? Great - I expect you to have crazy high growth rate. That’s the expectation of a world that is so deeply interconnected.
Consumer-ready LLMs have been widely available to the public now nearly 4 years if you start counting at GPT 3.5. There's certainly some compression here but Karpathy does a good job of outlining how silo'd those experiences are: the bulk of the product penetration (based on headcount rather than token usage) is "derpy chatbots" (his words, not mine). Most of the actual token spend is likely done on the STEM side. We're talking about a technology today that currently only has significant penetration with <25M users.
People need to be patient and put in the work to build great products, rather than lament its slower-than-desired (albeit still quite fast) adoption curve.
At least for me, personal agents have been quite useful. If you integrate them well - shopping, filling random government forms, keeping track of my personal life, optimizing training plan and nutrition, planning vacations, etc. I can do more home repairs or renovations myself with help of chat
So basically you get personal digital assistant - very useful - but not changing my life.
What would I need for real transformation though? Food, house cleaning, commute - "every day chores". Even if we had ASI tomorrow, what could it do to change my life, besides taking away my job? Design good house robot to help me with food preparation I guess... Maybe finally self driving cars launched in my area? Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
> besides taking away my job
> self driving cars launched in my area
It sounds like the transformation would be a massive collapse of the economy with scores of people unemployed.
> Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
Maybe, but more likely it will be the humans who control the AIs and hold the power who will make this decision.
Yes, and additionally, on the other side, there are a bunch of people using it to build exploitative products, spam, weapons, etc.
Almost every time I use an AI chatbot it eventually turns out it's lying to me. I would not trust them with any of the things you mention.
This is the kind of gap in shared understanding that is widening.
When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
The incentives seem to be lined up to push users to the frontier of "how much can the harness around the model do and claim without most people pushing back on the claims."
The bigger the pile of stuff you have the tool create unsupervised, the more incomplete/inaccurate the overall picture of it is gonna be. That doesn't matter for most hobbyist/helper-tool/game-port/etc type of tasks.
I get a ton of value out of the tools and yet part of that value has come from being hyper-aware of the shape of the limitations, which changes much less and more slowly than the specific of the limitations.
"How'd you notice that one!?" - well, it helps to be the only person on the team who took the time to interrogate the e2e interaction of all the code...
The median piece of OSS software was already pretty rough once you had to depend on it in production at scale to make money. The median piece of the new AI-generated wave of software, also pretty rough.
The bet that OpenAI/Anthropic is making is that they can automate the interactions generally/globally enough to work around this and eliminate the need for humans in the loop. But it's a tricky problem because it's precisely where the permutations and special cases get nasty and much harder to deal with than more broadly-applicable cases of codegen/test-gen/mash-it-until-the-tests-pass-and-the-code-review-angent-is-happy.
This has all started to feel very Emperor's New Clothes-ish. "Well, if it's not working for you, then it's your fault. You're just not using it properly. You're not worthy of its wonders".
Not to say there's no value to be had. However, it's also been oversold and its error-proneness is a real problem.
But, it seems there's a steady stream of commenters and creators who are quick to remind us that it's never a problem with the tech. It's always a user problem.
Meanwhile, the errors are renamed "hallucinations", but let's not focus on those. In fact, if you just look right over here at this .md file...
It's like driving a PT Cruiser and concluding all cars are bad and swearing off them entirely.
using skills with opus 5.5 and agents for each individual task that have shared knowledge of all of your records does not lead to a significant increase in trustworthiness
> When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
While the latest and newest should be needed for greater capability, the average should be enough to give a glimpse pf what is possible. Like a basic netbook is enough to enjoy computing, no need to purchase the latest supercomputer.
If you need to switch your setup every quarter or so, it’s not a tool, it’s a rat race.
I have ChatGPT Pro and it sources every single thing. For my Asia trip right now it's been helping me every single day to figure out optimized routes on what to do and where to go (if I feel like it, not to say I don't wander around too).
me too, but I recently blew the transformer on my boiler by following its directions to wire up a C wire to power a smart thermostat. oops. $500 mistake.
How often do you check the citations? I’ve found with the Google search AI summary it will provide answers and link to their citations, but a good chunk of the time the answer isn’t on the linked page.
It just makes stuff up (or gets it from somewhere, who knows) and then links to a tangentially related page as if it was the source. Either that or it identifies the page as being a source it wants to use, and then summarizes it incorrectly. Not sure the order of operations it’s doing internally, but regardless the citations are bullshit.
The citations provided by the free version of Gemini are comically bad. They almost never contain information which directly supports the relevant claim, and more often than not they're completely irrelevant.
Google's models are worse that OpenAI's, doubly so on Google's free AI overview versus ChatGPT paid plans. 6.1 Sol is very good at synthesizing actual information it finds without hallucinating and I've not seen any instances it failed while I checked the sources.
That sounds terrible. I like to live like a human being
What that does that even mean? Even if I grant your premise (I don't), if one is traveling and engaging these places in person....what's the problem?
Where did I say it was a problem? Using chatgpt to plan an itinerary sounds terrible TO ME
Why? In the past I would've used Google so it's just a faster search engine for me that works with natural language, what do you use for planning then?
Which model? Super weird
Every time I use Google search AI for anything more complicated than a search query I feel that way. A paid model from pretty much anyone is much better.
When is the last time you used an AI chatbot, and do you only use the free versions?
> Almost every time I use an AI chatbot it eventually turns out it's lying to me. I would not trust them with any of the things you mention.
If you talk to free chatgpt, sure.
But if you launch Opus 5.5 in claude code - for some reason Opus is the smartest for me, even for "general" type of questions, when launched from Claude Code. It has harness/prompts optimized for correctness, while more "customer-grade" products, even when used with top model tend to optimize for latency. And I tend to give "context" - e.g. I have "personal wiki" + skills for lots of those things.
I don't recall i last few weeks where Opus 5.5 in CC made an obvious lie, made an unreasonable assumption, or made a mistake that smart human couldn't make. E.g. if it "failed" it failed in a way that non-genius human could fail in similar way.
It’s sort of weird how
1) open weights and cheap models are only about 6 months behind the frontier in evals, but…
2) it’s always the premium model that only the biggest fans pay for, that’s just now finally achieved enlightenment.
Oh well, I guess we’ll see in another 6 months.
opus 5.5 uses agents and skills in chat sessions as well and maintains internal context that it tries to distill from what you provide
I tried to have it write a synopsis website for the different types of dance patterns in Dance Dance Revolution, with guides on their execution, explanations, and provided song sources. I provided all of the source articles and search websites needed for it to be able to do that, and gave it a sample set I wanted it to utilize and build off of.
It has bits and pieces of information wrong in 90% of the sections it created. Poor song choices, missing song choices, incorrect explanations, incorrect step-patterns, the list goes on.
If you're not an expert in a given field for which you're using it, it's going to look like a genius. If you are, it's untrustworthy and borderline useless.
Maybe it wouldn't be if there were ample training data on my topic, but there isn't.
It utilized the research from millions of papers many very close to the destination answer to make those proofs. What would it do on something that isn't supported like that? Be confidently wrong while sounding right. That's what we've seen repeatedly.
I think my anxieties would be different if I were confident a DDR guide full of lies would be spurned and unprofitable.
In a way, I'm not worried about the LLMs per se as much as a kind of moral and ethical collapse. A debasement of the currency of communication, a market of intellectual-lemons.
I saw electroboom's latest video lamenting facebook AI slop about electricity, such as a way to wire a three-way switch that blows a fuse when you turn the light on. For some reason, I am shadowbanned on HN.
I see your comment, perhaps it was a delay between representations or nodes?
> If you're not an expert in a given field for which you're using it, it's going to look like a genius.
For someone highly literate, llms aren't going to help you write better. If you're borderline functionally illiterate (this is > 50% of the population according to some studies), I'm sure the llms are awesome.
I've long thought the same about all the hype around coding (plenty of examples from years ago now), but will continue to test the new models. They are improving.
Personal agents are wonderful! You can ask for hiking trails and it will customize one for you, frequently resulting in emergency calls for rescues! Your purchasing/selling AI will arrange sales for you but forget to tell you, leaving you in breach of contracts you didn't know about! Your personal nutrition agent will absorb all the broscience and customize a diet that will result in hair loss, but at least your in ketosis, amiright! And instead of hiring a plumber/electrician/contractor to do work subject to code, you can just ask an AI to tell you how to do it and then wonder why your buyer is suing you for $$$!
(Those are all examples from just the past 4 days, and just the ones that made it into mainstream media.)
Using AI for everything digital, I end up going and doing chores while waiting for the LLM to do its thing. What an insane world, where knowledge work (dreary and uninspired as it is) is automated while I am consigned to scrubbing dishes and waiting for the machine.
Buy a dishwasher, you won't regret it!
Honestly the big reason why I use my dishwasher is that all the dishes would not fit in my drying rack. I'm not very convinced the process of me rinsing and scraping my pans before they go in the dishwasher is any faster than when I lived in apartments when I handwashed everything. Maybe my dishwasher sucks too, but I mean it doesn't hold a candle to the sort of jobs on my pans that require good old fashion baking soda paste and elbow grease. I'm still slaving with this thing. It's picking the low hanging fruit for me in other words.
I feel similar about my robot vacuum. It actually takes a surprising amount of labor facilitating the damn thing. I have to walk through the house and basically vacuum proof it so it doesn't eat various cords, rugs, lifting chairs up and putting them someplace so it can get under, etc. Not sure its really a labor save to when I was just brooming the entire apartment in 20 mins and not having to move anything around due to reach of the broom and it not having a tendency to eat things. Hand vacuum I can be more carefull too about the fringed edges of rugs and don't have to fold them over themselves.
I still do it because I already bought it mostly.
The trick is dialing the things in properly once, and then reaping the benefits over and over again.
e.g., my home is always vacuum-proofed. It automatically runs through all rooms except for the bedroom at night, finishing just before my alarm clock rings, then vacuuming the bedroom an hour after the last person's alarm clock has rung.
This way I never actually notice it, but the home is always perfectly clean as if by magic.
Similarly, if you use separate dishwasher detergent, antisurfactant and salt, and properly dial in your water hardness, it'll be able to clean everything no matter how dirty the dishes are. Including dried-on tomato sauce. Just put the dishes in, and they'll be clean the next day.
Of course it takes a bit of effort to set this up, but then it just works for a long time.
Similarly, I'm a big fan of handwriting software with care & focus on quality, which takes a bit longer at first, but then allows you to trust the software to continue running for years to come.
I've done a lot of vacuum proofing, even getting the magnetic strips, but there's only so much you can do. Like it doing a little wheel around a chair leg is just not as great of a clean as if I moved that chair out of the way. My rugs have fringe so there's no stopping that. Cables I tend to have out have to go away.
And then of course, all the cat's toys.
> And then of course, all the cat's toys.
This might be the next thing the robot vacuums all start working on. One has already released a first version: https://www.youtube.com/watch?v=GykaEgNkkVQ
It's definitely easier in a minimalist & tidy home. Our rugs have no fringes, cables are either rolled up or permanently installed & hidden, chairs are stored at night upside-down on the tables they belong to.
Sure, it's not perfect, but it's still really convenient.
This is the paradox. For software engineers, the current crop of “AI” tools is magic. A huge accelerator. And then they assume the rest of the world will get the same boost. However this is not the case at all. Claude code is great because it is trained on extremely well documented git histories and stack overflow. And outcomes are binary: the code works or it doesn’t; the PR is accepted or it isn’t. Not much else in the business world is like this at all. No training data with this kind of sequential development information. And no real black and white outcomes as such.
I would love it if we can just take the win on having created an amazing tool set for coders. Can’t this be enough?
AI has only been around a few years in its current form.
It might take a decade to take what we learned from AI and turn it into something that will benefit the masses.
Most of it can't be broken down into something as simplistic as a dishwasher.
Most people won't notice cancers being found faster due to AI being better at pattern recognition than a doctor, but it's still a great benefit.
A decade? Better not tell the investors that!
Yes. The AI industry has to deliver some substantial value to the average citizen soon or there will be political trouble. Food, clothing, shelter, and medical care have not improved yet. Nor does there seem to be much AI can do in the next few years that will help with the basics.
We can have unlimited new entertainment content for passive consumption. That's about it so far. 17,000 new Asian vertical dramas a day now.
LLMs could’ve been developed a lot slower without capsizing the entire supply chain and been seen a lot more positively, China is building a lot of good AI with a minuscule of the compute power and it’s fine
there was never a need to rush other than for capitals self interest to control
if frontier labs wanna pace the frontier, they should be honest and return the compute back to the people
China is building good AI by distilling American models. I think that's a great thing, but Chinese progress is currently dependent on American progress, because China lacks the compute to train the largest models (as stated by DeepSeek's founder[1])
[1] https://sumsec.me/2026/liang-wenfeng-investor-meeting-transc...
i knew about the compute already (he was very public about i believe huawei under delivering the amount of chips he requested), but nowhere in that transcript does he say he's dependent on US progress for his own progress or he's distilling the US to make progress
i don't really think distilling is even that fundamental anymore (nor was the evidence ever that strong - https://msukhareva.substack.com/p/is-kimi-k3-really-distille...), the game is all RL now, and you don't need american progress for that at all (just compute and data, and the data brokers will sell to anyone no problem)
i do think there are black market proxies from china to the US trying to sell "claude" and "deepseek" access mixed together, and i do think that prompt distribution is probably going to be very nice datasets
People need to eat, sleep, be healthy, have shelter, feel loved, be in communities...
Coding has been transformed but none of those areas of our lives have been substantive transformed by AI yet. Whoever does, will create a lot of value for the world.
>AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Do people need permission from mathematicians to solve math problems? Is one of the Millennium problems not worth applying a new system to work on it?
I understand the despair people are feeling at AI automating their work, but we have to be serious about the fact that the frontier labs are just responding to natural incentives. Mathematics, coding, all are just applications of logic, and it makes sense that a system trained on inferring patterns from data would work well in that domain.
You are misunderstanding what mathematics research is. Its objective is the development of human understanding of mathematics. This requires open problems to research. AI are “solving” the open problems without creating any human understanding.
At the very least, the single bit of information of whether a conjecture is true or false advances human understanding. It allows researchers to abandon paths which would not have been fruitful.
I'm pretty sure OpenAI's proofs, while difficult to understand, have a net-zero impact on human understanding of mathematics. It took the first Millenium problem years to verify its accuracy. It's only been days since OpenAI released its problems, surely it would take years to evaluate whether the solutions meaningfully advanced human understanding.
It may provide that one bit of information but it could still be a net negative if it destroys current institutions (such as graduate studies) and no new institutions can be developed that provide a clear signal of who is doing meaningful work. That was a byproduct of solving open problems that are now lost. These are not my ideas, these are things that Terence Tao and others have written about that were recently on the front page of HN, e.g.:
https://mathstodon.xyz/@tao/117395269325940185
https://mathstodon.xyz/@tao/117394287977270632
Who determines the objective of mathematics research?
All mathematics researchers in the whole world have agreed upon that objective apparently?
This is basically the dictionary definition of the word "research". But yes, all the work they are doing is focused on this goal. They write papers for humans to read, give lectures for humans to listen to, have discussions with humans - all with the objective of improving human understanding of the subject.
Many of them - all but a couple of the living Fields Medalists - recently signed an open letter about this.
https://mathandai.org/
I don't know, this just seems like gatekeeping of the term "research" simply because you don't like the way it's being done.
Many modern medicines do not have understood mechanisms of action, yet we know they work. Are the people who found them not conducting drug research?
Do we even have a different term for what the AI labs are conducting if it's not mathematical research?
Additionally, I would argue that any work that adds to the total corpus of known facts increases human understanding, even if the path to get to each fact isn't understood. It's hard to argue we'd be better off with less total information.
OpenAI to this day is not aware if they solved or not Navier Stokes.
Mathematicians are not gate keepers but I understand the frustration. If someone claims they spent 30 million dollars to solve one of the most difficult problems yet there’s no new knowledge, understanding, nor certainty. Then what’s the point besides marketing?
The funny thing about comments like this is that the moment LLMs are going to replace everyone in the warehouse or the trucking industry, we suddenly have to do something against the LLM. But clearly they can't actually do anything meaningful and have been a waste of energy?
So just let them spend their money and they will eventually fail and bankrupt, right?
Whatever benefits AI has to offer on a societal level are going to take a long long time to show up, give it some more time. We had smartphones in 2007, and e.g. By way of example, Uber didn't launch ridesharing until 2013, and then that still took years to seep into the public awareness.
By comparison, we're in the early years of AI. We're all still figuring out what this thing even means.
Uber was a massive mistake on every level.
This is what is so frustrating about this whole ai thing is we aren’t learning from the mistakes of even the very recent past
Uber is a vast improvement over the previous taxi oligopolies.
I miss being able to reliably hail a taxi A LOT. Uber may be an improvement in smaller towns where cabs didn’t exist, but in bigger cities it is mostly a downgrade. Rather than walking to the curb and immediately getting a car, I’ve got to dick around with my phone and wait for a car to come to me.
Living on the south side of Chicago, it was almost impossible to hail a cab, and often times if you called for one it just wouldn’t show up. Or I’d try to hail one downtown, and be refused once I told them my destination. I’ve not had that problem once with Uber, good fucking riddance to taxis.
What's frustrating is that at the end of an event like a concert, everyone goes outside to catch an Uber/Lyft, but we then have to match driver to passenger, instead of having a line of waiting cars, each group just gets in the next available vehicle, shows the driver a QR code, instead of the mess we currently have. Why can't we do that? Taxi regulatory capture.
Taxis were terrible and I celebrate their destruction at the hands of the rideshare companies. Even now it’s still cheaper than a cab was (though those early days of VC cheap rides were great).
I agree with you, but those examples are going to fall on deaf ears. The pessimistic mindset will say that smartphones were a terrible invention that's ripping society apart, Uber is an unethical company exploiting workers, etc.
I think ridesharing was a net good for ordinary people even if Uber is/was unethical.
However, the counterargument falls entirely flat. "Whatever benefits AI has to offer on a societal level are going to take a long long time to show up". We don't have the luxury of time because AI disrupting communities and livelihoods faster than perhaps any other technology we've ever seen, with the most capital that has ever been put into any tech since the railroads.
Even if you're optimistic we'll eventually get the cancer cures and fusion reactors, doesn't it make sense from the perspective of an ordinary American to regulate the harms like the homework cheating, parasocial relationships, job losses, cyber attacks, etc., especially if it will take years to get the benefits?
Different benefits will show up at different rates. We don't need to wait for cancer cures and fusion reactors. It's already easier than ever to start a small business and use AI for things that would have been prohibitively expensive or time-consuming before. The technology is transforming how scientific research is done, how engineering is done, etc.
I don't see much evidence in the economic data that AI is disrupting livelihoods. Unemployment has been stable around 4% for a while and productivity growth has been at historically normal rates for the past 5 years. I think a more likely explanation is that AI is increasing uncertainty in some industries, which are seeing higher employee turnover, and many of those people are moving laterally into other jobs where their skills are more likely to be useful.
I don't think anyone in this thread has argued against mitigating harms. The OP was about current capabilities and the perceived lack of benefits for the average person.
To be clear, I don't disagree with your sentiment that the disruptiveness of AI is causing a lot of grief. But magnitude of disruption is not an indicator of magnitude of benefit (or lack thereof).
Some of the most disruptive periods in history brought about the greatest long term improvements in society. This is the price we pay for changing how things are done, sometimes for better and sometimes for worse. We don't know which one it is for AI yet.
When did Uber stop being unethical? Did I miss the memo?
Because they didn't just break laws, they built an entire system to mislead law enforcement.
(How do tech people not know about Greyball??)
I don't think it's surprising that it hasn't improved life much, AI is new and change takes time, but the fact of the matter is that it also made things worse in some ways, ways that I think were entirely avoidable, if only the incentives weren't so warped.
Publicly-accessible AI started working 1 year ago, 2 years tops. What did you expect? It to solve war driving prices up?
I would have expected apps like Amazon and websites like various Meta properties to be even slightly less broken and shitty.
Like, just at a bare minimum.
These are “use all the AI” companies with infinite resources, after all.
Why use apps that you dislike, just ask Claude to click through them for you. And no I'm not being ironic, this is the way things are headed, you just abstract over all the crap by having an agent do it for you.
"Slop is its own cure"
The alternative is people stop using these apps, consumer demand goes down, along with the GDP of the country.
I think one of the hopes is they'll get smarter than us and able to solve real world problems better - disease, war, poverty etc. But they are not really there yet. Give it a little while.
AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
This is an indictment of mathematicians, not AI.
Who indicted anyone?
The fundamental malignancy here is that powerful people have become disconnected from the foundation of society. They have suffered from model collapse. They have been talking to each other in circles for too long, and doing things that make no sense to anyone outside the elite group, and because the systems that concentrate power to the top are so strong now, they continue gaining power even while messing everything up in a nominally free competitive market.
Sounds like collective psychosis.
Is it reasonable to expect AI improvements to filter into all of those aspects of life that quickly? It's only in the past month or so that we've started to see major breakthroughs in mathematics. It might take several years for that level of research capability to reach areas like engineering and medicine, at least in a way that affects the average person's quality of life. But at this point it looks very likely that it will happen eventually.
Your pessimistic view doesn't hold up against the data. For a lot of the problems you mentioned, we're in a better place today than almost any other time in history. Of course, things can always be improved and we should never grow complacent or accept injustice. But by most objective metrics, there's never been a better time to be alive.
This view is too cynical. The frontier advancements are certainly going to trickle down. Take cars for example. AI driven cars are already here. AI-designed battery systems, fuel systems, materials and 3d printing capabilities will come to cars. It will likely lead to cheaper, more efficient and more effective vehicles. If will just take time for the impact to percolate.
My prediction is that while those “advancements” may come to cars, the only practical differences will be lower quality products, cost savings for the car companies, probably more expensive for consumers, and somehow it will also be collecting more data to sell to advertisers.
If you look at the average price of a new car, adjusted for inflation, it has pretty much always gone up aside from recessions. There have been plenty of advancements in car technology, they do not make them cheaper. It actually gives reason to charge more for them due to perceived increase in value.
Yes but you're getting a lot more for the same price. It's a lot more powerful, comfortable, safe and efficient while the effective cost has remained the same.
What percentage of that price is cost of good sold vs. a bunch of regulatory bs like dealers and taxes?
Just a few more weeks, a few more hundreds of billions, it's right around the corner....
This only works if there remains a viable and attractive consumer market for these advanced goods. AI is both an imminent wealth shock and perhaps the largest structural change to the economy in history as it threatens to drive the value of human labor to zero over the next decades.
In American history, when previous shocks like this occurred (e.g. with the Great Depression), we needed massive economic programs like the New Deal to restore consumer demand. When such programs are not forthcoming while wealth inequalities keep increasing, one can look to other countries for what ends up happening; either the disenfranchisement of ordinary people over time or unrest and redistribution by force.
AI car navigation is here, but driving is a social act. Have you ever tried to keep an automated taxi out of a private driveway or the scene of an emergency? There is a lot of work to do and limited evidence of LLMs raising the bar.
I think he means LLMs, we knew about RL already.
Ah yes, the famous "trickle down" theory, which you may know from its smash hit "trickle down economics"! Surely this time it'll be different, right?
> Ah yes, the famous "trickle down" theory
While trickle down theory is dubious when it comes to economics, it's clear that it is valid when it comes to technology. Tech trickles down.
How can you say tickle down economics doesn't work? The US has made it easier for businesses to grow and the result is the highest standard of living ib tje world.
I think it's hard to quantify "highest standard of living", but for the attempts at it (e.g. HDI, etc), the US is not number one.
Hate to break it to you, but the US doesn’t even break the top 10 in a lot of reports.
Trickle-down economics specifically refers to economic policy that favors the wealthy and corporations with the argument that this will lead to wealth trickling down to the economically less fortunate. Examples include the Reagan and Trump tax cuts.
This has been studied somewhat extensively (https://academic.oup.com/ser/article/20/2/539/6500315?login=... is one example), and it’s been shown over and over that this does in fact not work and leads to larger wealth disparity, across many countries where it has been attempted.
It’s possible you’re working off some other definition of the term based on your comment but that’s the generally accepted usage.
The person you replied to was the second person I've seen on HN this week try and defend trickle-down economics explicitly in more or less the exact same terms and justifications. What is going on with that? Did Peter Thiel tweet some pro trickle down dreck recently that these people latched on to?
You should see the comments sections on NYT articles, lately. It was like someone flipped a switch about two weeks ago.
It would be neat if someone did sweeping sentiment analysis on them, just to see it graphed, but I’m sure gathering the data would be prohibited.
It’s all bots anyway at this point.
The US alone!
Worse, the view is largely phrased as gaslighting AI. Blaming the cause of particular grievances that coincidentally have been sourced at the same time as the rise of AI, is purposeful.
> [Your train] never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
None of this has to do with AI.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Plain wrong on both counts.
> Imagine what a "country of geniuses in a datacenter" will be able to do Apparently...nothing.
Nonsequitor, probably based on a personal beef. Datacenters, at best, are understood to be for hosting more AI - not for people to somehow live in as an arcology of discovery.
I think only because these are all things that AI companies are saying will be helped by AI so give us more money. Nobody advertises their product publicly as "you will be able to buy other product to do the same things you do now but pay people less and make more profits" even though that is where the value is for them.
Why fix your appliances and infrastructure and give you healthcare when the plan is to obsolete the whole lot of you?
I mean there's been huge strides in electric vehicles at low prices that Americans haven't experienced.
We've just locked out a large part of modernity because the shitty American auto industry is in swing states and also gender.
Same could be said about the slow rollout of Waymo.
Lots of America is just very anti technological change, and this means any technology no matter how "magical" hits a wall.
https://nitter.xitter.cc/augeeidos/status/210940046382122618...
This reply summarizes the problem from a different angle.
As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes. There are still plenty of obstacles to overcome but the number of AI governance and steering groups that I have been apart of are indicative of what's happening in a major industry.
> As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes
> Senior Director of Product in Insurance Tech
anakin_padme.jpg
This is an interesting perspective.
Nevertheless, my counterpoint to
> If they actually had access to systems that could do "years of top professional work" across software, cyber, science, and math, they would not mostly be leaving to start more chatbot companies, agent frameworks, eval tools, and "AI for workflow" startups.
> They would be starting normal companies to disrupt the global economies of the world.
> Insurance companies. CAD companies. Compliance companies. Security auditing firms. Logistics software. Tax automation. ERP replacements. Drug discovery tooling. Industrial design. Legal ops. Accounting systems. Boring vertical SaaS with insane output per employee.
is:
Many people who work in such industries already know an insane amount of inefficiencies. Unluckily, they are not in positions and have no political power (with respect to organizational policy) in the company that they are capable of changing the situation.
Additionally, many of these sectors are heavily regulated, so it is very hard to make a difference in this sector as a startup founder, and even if there is an entrpreneur who founds a company to change the status quo, an insane amount of work in the company will be handling the big amount of red tape.
Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Things worked better before we started putting bad software in everything. As a software engineer, I know what good software looks like. The fact that almost all software is now bad -- including the software that drives AI and the software that AI generates -- makes me want to flip the table. We know how to write better software and the general public should hang us if we continue to refuse to deliver it.
Probably preaching to the choir here, but the software is just a symptom.
The problem are the power-relationships and laws which allow someone to abuse you and your stuff, block competition, and even put you in jail for trying to remove the software—or helping others to remove it.
> Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Never, ever connect appliances to the Internet. I thought this was common knowledge on HN. Also pull the telematics (cell modem) fuse to prevent your car from snooping on you and selling your location and driving habits to data brokers and insurance companies.
Yes, but this is ridiculous for society. It is to the level of "do not wear a nice watch if you do not want it to get stolen": there should be no need to avoid wearing a nice watch or to pull a telematics fuse from the car in the first place!
The change will be slow, not sudden. Slow and devastating.
Why and how will it be devastating specifically?
I think that's a bit disingenuous. We've only begun to apply LLMs in the real world and a lot of industries don't move very quickly. Not to mention that we're still very hardware constrained, we don't yet have the equivalent of an ESP32 chip we can just stick into any piece of electronics to provide state of the art LLM responses. But we'll get there, soon.
Same as before, really. You can have geniuses but if not used right you get nothing.
I would just add that consumer products in general aren't the same - practically every business model has incorporated obsoletionism to the point it's practically cancelled out the curve of progress and evolution of tech. We could have nice things.
> Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
I suspect the real world change is going to happen unbelievably rapidly and isn't far off. Right we don't quite have the hardware to embody AI in a generally useful way, but that's coming fast.
Within a couple of years arguably any task that requires just knowledge or intelligence is something you'd be better off giving to an AI than a human. And decent humanoid robots are maybe a decade behind that.
In the same way people growing up in the mid 1900s saw rapid changes in their way of life with planes and cars replaces boats and horses, we'll see something similar with robots replacing human labours. Right now we're still in the phase where people don't think humans will ever be able to control flying machines and that cars are reliable and slow. Things will change.
I already felt our civilization was on a steep decline, but AI is the final nail in the coffin, technology kills my own job too, hooray.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
That's not fair. To first approximation, most of the progress in human condition of the last, like, forever is implementing solutions that people didn't see value in at the time of creation. Woz built a toy that could barely run software at all. The human genome project was panned as a boondoggle. Penicillin was a mistake.
The first fire pit probably looked like a dangerous stunt to the tribe faced with it, even.
Does that mean everything will come up roses? No. But LLMs only started talking to us for real like three years ago. Let's not just that fast.
Have you seen the shareholder value though? That’s not nothing it’s the only thing that matters!
With AI software replication, I wonder if shareholder value even increases. Services accounts for 80% of American industry, what happens when big software companies loses revenue to piracy/recompilation i.e. Adobe, Unity, AutoDesk, tax/law software, etc..
And the human society was entirely destroyed, but for a brief moment we were able create share holder value.
What’s your point?
People didnt get a new car after they got. Internet in 2001. They didnt get new dishwashers.
You are talking like we had this capabilities for a decades but it's been just a few months of truly capable AI. Virtually all of IT switched to agents that quick. Non IT people just started to make software for themselves. Vision and translation capabilities of LLMs help millions. People talk with AI's now verbally to get advice or to just hear a friendly voice. The math thing is just few weeks old. We are just getting started. How fast did the telegraph meaningfuly changed lives for the better for bulk of people? Radio? TV? Computers? Internet? Cellphones?
Has any technology made an impact on physical world faster?
Exactly, we're all software engineers here and even for us it took weeks/months to get to a position where we can really leverage LLMs (think platform, security, observability, etc).
Imagine being in charge of a physical factory, sure you might have a 200 USD Claude subscription but you also have a floor full of physical machines that have, if you're lucky, an undocumented debug interface and a paper manual. Not to mention safety inspections, insurance, etc.
You can't vibe code a factory floor. Not yet, perhaps in 5 - 10 years but there are some very hard, non technical, problems that need to be fixed before we'll see meaningful improvement there.
I'm pretty confident with enough context, Opus 7.5 will be able to vibe code a factory floor.
How much time do we have to wait before we are allowed to criticize AI? GitHub copilot has existed for half a decade. Cursor, Claude code, codex, have all existed for over a year and have seen more investment that practically any product in history and have almost 0 barrier to entry. Where is my decade of software progress in 1 year, because that’s what 10x productivity looks like! All I see are the same old broken products and services from basically the same handful of big players, higher costs, and more bugs. Remember when every year iOS and Android saw loads of new features and a radical rethink every few years? And that happened on ancient hardware and in a significantly smaller industry. Where is anything like that happening now?
I'll criticize Copilot with you. It's terrible and close to useless and it's a terrible shame that many companies got stuck with it due to being early adopters of this crap. I'm sorry for my friends who are employed by such companies.
Progress between Copilot and something like a gpt-6.1-sol in a good harness is at least as big as few Android versions.
> How much time do we have to wait before we are allowed to criticize AI?
You are allowed to criticize it, just not because "the progress has been too slow". It was not.
> to just hear a friendly voice
This is far from being proven as a net gain and there are massive lawsuits incoming related to AI recommending suicide to kids and others.
In all fairness, depression and suicide have existed before LLMs.
Safety systems are not optimal but improving everyday and although I doubt we'll ever get a perfect safety system, it seems that we're not very far from a "good" safety system. The days of Google recommending glue on pizza and generating diverse Nazis seem behind us.
Not to mention all the untold good that LLMs have done. Personally I've seen that LLMs are incredibly good at pointing conspiracy nuts and extremists into the right (moderate) direction.
In a polarised world, we definitely need that.
>Your car is still the same.
Not for Tesla owners in select countries, where their car now drives them around 99% of the time, or the people of Austin Texas, that can now hail a fully driverless Cybercab for cheaper than an Uber!
>Food quality is worse.
Food quality is higher than ever if you want high-quality food!
Unfortunately, the people crave slop.
Honestly that's true of most innovations
AI won't show itself as a big new thing, but possibly in improvements in small little things
Most people won't use AI directly, except maybe as a smarter Google. In the same way people don't use HTTP directly
I'll add some...
Videos are shittier. Most songs are even shittier than the autotuned pieces of shit already were. Ads are worse (even though nobody thought that was even possible). Customer support is worse.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using)
That nobody is using and that are worse than what we used to have. And because nobody's using them, the countless issues plaguing them aren't even reported by users anymore (there are no users anymore). Full of bugs, usability issues and gigantic security holes (but which doesn't really matter because nobody's using them).
It's not even clear that it's making long-standing software (like say Linux or Emacs or Blender or QEMU) better at all.
Now I'd say though: most of the issues you mentioned have nothing to do with LLMs / AI. But it's very clear that AI ain't solving much of the world's problems and that the promises we got years ago didn't materialize.
All you said is true, but on the flip side it is giving an individual user huge power over software. To create their own, to reverse engineer proprietary ones. Decentralisation of such power can only be a good thing.
Even if a certain small group of people are also using the hype surrounding this tech to attempt to destroy the home PC market and reverse the personal computer revolution. They will not succeed. There is more of us than them. Even with "thousands of frontier LLMs". A human+llm is going to be more powerful for a long time rather than a bunch of LLMs.
The power isnt really decentralized when you see that the internal AI models are superior than anything you can self host, and the right to use those can be stripped away at any time.
This is like saying food is shittier today because there are now more restaurants. No, food has never been better, there are just a ton of shitty restaurants.
More like saying samey corporatized slop bowl copy paste jobs (eg Cava) are driving out real, quality restaurants, a thing that has actually been happening for some time.
It’s amazing how much the goal posts move in every AI conversation. A couple of years ago the average HN thread about AI would be filled to the brim with skepticism about AI ever being able to accomplish anything other than flappy bird clones.
Now it’s comments like this lamenting that the amazing LLMs we’ve had since early 2026 haven’t entirely revolutionized the whole physical world and even global geopolitics yet.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using),
I guarantee a lot of software you use today has substantial AI development in recent releases.
Even the Linux kernel is getting a heavy input from AI and Linus is advocating for it. Unless you’ve frozen your entire software stack in 2024, you are using software with AI generated code.
> and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Saying nobody wants those problems solved is flat dishonest. The complaints from some mathematicians are that they and their peers didn’t get to put their names on the papers, not complaints that the problems were solved.
I've been thinking about this for 2 years. At some point it will be easier for them to create self sustaining cities - free houses cars public transportation clean food clean water -
Africa is going to look like wakanda in less than 5 years if WW3 gets canceled
Please don't be snarky or curmudgeonly in HN comments. You can make your substantive points without either of those things, both of which the site guidelines ask you not to do.
https://news.ycombinator.com/newsguidelines.html
It honestly escapes me why you would judge the juxtaposition between everyday life of everyday people and a very specialised minority as snarky or curmudgeonly. Was there too much emotional impact?
There are so many comments on HN that would justify your attention with regard to this specific rule. This is not one of them.
Not sure if that's a fair criticism dang? I sincerely wasn't trying to be snarky or curmudgeonly, but I did want to make the point clearly that none of the benefits of AI have materialised in the eyes of the average person, only the downstream costs. To my eyes, that's a salient, reasonable, and wholly representative position to share.
Did the writing also contain emotion? Yes. As all good persuasive writing should.
As the other moderator, I'll give my own perspective on this. I haven't spoken to dang about it but I completely understand why he thinks this is not the kind of comment we want to see on HN.
The main issue is that it doesn't engage with the substance of the post, and in fact, it exemplifies the very point the post is trying to make: there is a big and growing gap between those who recognize the benefits or potential of AI and those who don't (or, to widen the point, who only see negative effects or negative trends in society).
I'll make it clear that I don't think your complaints or concerns are unfounded or invalid. There's merit and substance to all these issues, and it's fine to discuss them on HN. They are discussed all the time here, and they can and should continue to be discussed in the future.
But an important part of being the kind of curious commenter that we want everyone to aspire to be is that we can look at topics holistically. We need to be able to look at the opportunities and risks together and discuss ways that the benefits can be captured and shared throughout society, and the risks and costs adequately recognized and curtailed. Most people I know of in the tech industry are taking these matters seriously.
It's a vast and complex topic to discuss, and here's not the place for me to get further into it. But the point is that it's a bigger and more complex topic than your comment adequately recognizes, and we need people to always be making the effort to be thoughtful and curious rather than fulminatory or denunciatory in their participation here.
(Also: I can understand you may not easily recognize that your comment breaks the guidelines; it's common for people to not recognize that their words seem more inflammatory when read by others than they intended them to be.)
You made a well-argued and wise criticism that many academics/writers/think could have written, the mod's note was off the mark.
How is this comment worth a moderator warning? Does not make sense to me, the comment is fine.
What sets this particular comment apart from so many others on this thread that you felt the need to comment on it specifically?
Can you point out what a good way of expressing disagreement or skepticism about a subject matter looks like on HN?
Only pro AI comments allowed.
(My comment is actually snark, someone ban me off this hellhole of a site please)
I think they made a really good point about this and I wouldn’t really call it snark.
Maybe I don't understand the word snarky but this seemed fine.
Even the paid tiers fall apart the second it touches the real world. To them it looks more like a total clown that keeps upcharging you
https://www.youtube.com/watch?v=t6MAZW-joCo
Pretty much all of this is basic stuff an interested teen would know and chatGPT fails miserably. I fear the "true world model" gap will be too much for LLMs. I guess time will tell.
so what was karpathy expecting?? everyone will give up on their life and start feeding the monster in the screen? that's literally what the tiny sliver is doing; they are food for the AI model.
I don't understand how "swarms of thousands of agents collaborating over weeks on software mega projects" works with the current context limits and at this point I'm too afraid to ask cause I'm afraid an AI bro is gonna punch me.
You can have agents make a plan and then other agents spec tickets and yet more agents do implementation and yet more do reviews and QA and whatever. It's possible.
Is it effective? I'm not sure, cause if it was I don't see why OpenAI & co are not releasing such a project instead of demos.
Your product needs to be either the swarm itself or the PR about the swarm - not the outcome.
Best I can do is a twitter post.
That's fine but these companies having billions in funding by now should have rewritten chalk and other terminal libraries in Rust while bundling the whole thing in Rust as a single static binary. I'm talking about horrible Claude Code and friends.
And then using emgui they could expose full fidelity IDE+Agent workflow interface (like DeepSeek Harness and similar) that's also exposed as MCP to be operated by another agent and JSON RPC for over the network but no. Skill issue?
PS: Even full fidelity Photoshop is possible with such high fidelity UI that has CLI, JSON API and MCP server all in the same single binary < 90 MB. For reference, see PhotoCraft or VectorCraft or WordCraft or PdfCraft.
>Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
Amid all the discussion of sigmoid curves, and where the "LLM wall" will materialise, I think few people would have predicted that the real wall in LLMs would end up being humans' capacity to verify the output.
What I fear is that people simply eschew human review altogether, considering we're talking about the industry that came up with the "move fast and break things" credo. Human review of LLM-produced code where I work is already a farce, and we're not special enough to be one of Karpathy's 5,000. I do my best to manually review anything that's my responsibility, but I'm literally one of very few people left working on my team, so in practice what happens is I submit PRs that are at best glossed over by completely unrelated teams for security, malware/prompt injection, and other serious concerns. Quality insofar as vetting others' code has completely gone out the window and it shows in the number of bug reports that come back, often themselves written in Claudease. This is all on top of everyone cynically phoning it in in the first place, due to the omnipresent sword of Damocles that is additional AI-driven layoffs.
Worse yet all the incentives point to this being the most economically viable thing individual companies can do. I think it goes without saying some type of regulation here is urgently needed, and that an unexpected cause of an AI bubble pop may end up being that humans simply aren't able to keep up with the pace of the output - leading either to precautionary plateauing of capability, or major liability risks related to a decline in quality.
| few people would have predicted that the real wall in LLMs would end up being humans' capacity to verify the output
That was the dominant concern in the circles I’m in, so it’s worrisome it’s being treated as rare.
I would question the narrative that humans lack the capacity to verify the output and would instead argue the people lack the incentive to verify the output.
The response of many mathematicians to the recent dump is a good example: verifying these proofs amounts to unpaid labour for OpenAI and wastes time that could be spent doing publishable work which ultimately results in money or personal success. The slop factor also compounds the work required to verify the output considerably.
For mathematicians, programmers or anyone, if the work required to deal with slop passes the limit, it is no longer in their own self interest to use LLMs. The expectation that people will use LLMs for the betterment of humanity against their own financial interest is baffling.
Humans are not immortal and cannot spend all their time into review (especially unpaid). Even today, there’s so much knowledge around that you have to be specialist of a narrow domain to get to the frontier. Even in computing which is just approaching a century of existence.
So, cure for cancer when?
Cure for humans*
"So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time."
-------------
It comes down to the difference between programmers and non-programmers. If you're not coding, AI literally is a derpy chatbot that's made it harder to get search results, raised your power bill, destroyed property values in a neighborhood near you, is propping up the economy but also creating a huge bubble that could pop and destroy a lot of retirement savings, etc.. At best, it's a nuisance, but it could be a lot worse than that. The AI evangelists need to understand this, because the average person is starting to pay more attention to the downsides while still not seeing much upside.
This gap between the people seeing the curve go vertical and those who are simply growing more and more annoyed is a real problem... mainly for the people seeing the curve go vertical. If the people who are merely annoyed right now don't see some of the promised benefits soon, or if any of the negatives get worse (e.g. massive increases to greenhouse emissions), they might get angry. You probably wouldn't like having the overwhelming majority of the public angry at you. AI evangelists need to start doing better things for the majority of people or they need to address the negative consequences they're inflicting much more aggressively.
We have spent the whole evolution understanding the Real Intelligence RI
I’ll say this until I am blue one the face. Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
There’s no reason to think this.
> There’s no reason to think this.
There are lots of good reasons to think this.
The first is that LLMs used to be bad at each of these nerd things, then toppled them like dominos. There's a pattern over time.
Another reason to think this is g. Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing. Although the reason is not perfectly clear, the pattern is well established, and seems to apply to LLMs too - GPT-6 is smarter than GPT-3 at everything, not just math. There's no reason to expect different for GPT-9.
That's just cocktail party level of thinking. LLMs are getting better at math, code and logic (and marginally better at science and general knowledge) because these are domains that can be objectively verified and thus there is a potentially infinite supply of 'facts' to generate and train on. These are very powerful but ultimately very abstract domains. For everything else the messy real world and its physical bottlenecks gets in the way and there's little reason to expect progress to accelerate. It still takes months to get mice to reproduce and run experiments on, no matter how knowledgeable about biology the models have become.
If anything, in some domains frontier models have become worse - claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
Is good writing verifiable? I don't think it is, but LLM's have been hillclimbing writing quality. That being said this is with the help of RLHF.
However there are many other domains which have verifiable rewards in the process of learning them, despite their overall impact not being verifiable. For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism. In economics, LLM's could create models of behavior, evaluate predictions over time and see how well those predictions match reality.
>claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
The quality of natural language output from today’s LLMs seems no better and often worse than LLMs from 18 months ago, at least in my experience. It’s much harder to build a good RL loop for something subjective like good writing, compared to something with an objective and verifiable right or wrong answer, such as software or mathematics.
If you gave me natural language from a 125 IQ human and a 155 IQ human I'm not sure I could judge which human is smarter from the writing quality.
And that's a massive intelligence gap!
It may just be that writing is mostly saturated, and the remaining perceived gap is not quality but some kind of "cultural fit".
IQ is a useless thing, but you can absolutely tell intelligence from writing. You don't need to write like [author of your choice], you need smart ideas and the desire to make them understood.
AI writing sucks because there's no intent or choice behind it. It's human thought, blended by linear algebra into pink slime and extruded into sentences.
> IQ is a useless thing, but you can absolutely tell intelligence from writing.
This is inherently self-contradicting.
It is not, as IQ is not an all encompassing measure of intelligence.
IQ doesn't measure writing _at all_ does it?
It's not at all clear that LLMs have gotten significantly (much less hillclimbing) better at writing since GPT4-ish. They did get better at oulipo-like challenges (writing under silly, verifiable constraints like Fable's viral 'facetiously' poem) but that's far from being all there is to good writing.
Anecdotally, creative writing communities do report drastic differences in quality between models (it is my understanding that Gemma 4 is especially praised) so there is something to it being verifiable but it's a far cry from being something that can be distilled into objective math/code-like evals. One could even say it comes down to vibes.
>For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism.
Yes, biotech companies are doing this right now but ultimately they are just predictions and you still need cold hard biological data to ground them and iterate on. That's slow and expensive, especially for the juicy fields (human biology, food and crops, clinical trials) and you can't just plug billions of VC capital into the pipeline and hope for RSI. Physical (gotta procure all those labs and their equipment), human (gotta hire and pay specialists to run experiments), biological (gotta wait for organisms to reproduce, drugs to take effect, crops to grow), regulatory (gotta convince agencies that your fancy new drugs are legit) bottlenecks get in the way.
And biology is one of the easier fields where a path to RSI (if not accelerating) is conceivable. Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
>I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
My uncharitable take is that the abtruse jargon makes people feel smart for understanding it and gives a sense of belonging (as a closed circle of initiates who understand LLM cant), just like rationalists love to repackage old or unsavoury ideas under new nerdy smart sounding names.
By evaluations against human writers, LLM's have improved at writing.
https://vulsar.ai/benchmarks/creative-writing-v1/#cohort-ove...
>you still need cold hard biological data to ground them and iterate on... you can't just plug billions of VC capital into the pipeline and hope for RSI
Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
>Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
That benchmark is fascinating, thank you.
>By evaluations against human writers, LLM's have improved at writing.
I would really take those benchmarks with a pinch of salt. We already know benchmaxxing is a thing for evals that are objectively verifiable like code, and coding models that seem to perform well on benchmarks don't pass the vibes test, imagine all the caveats that come with such a subjective test like creative writing, not to mention control issues (are they the same human writers across benchmarks? Are they the same testers?)
>Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
I don't disageee and am in fact excited to see the advances, what I am arguing is that it is going to be slow because the hypothetic feedback loop that would improve the model is bottlenecked. There are actual physical and biological limits in our universe. Unless VCs find a way to make mice embryos and crops grow 10x faster (among the many, many other things that would be required for acceleration) the trickle of data is not going to be anything like the accelerating returns imagined for math and coding. And that's assuming biological RSI is possible which has yet to be proven - current foundation models are interesting but they are still specialized and there is no clear, obviously uniform way to encode the many, many modalities that exist in biological data.
>The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
You wouldn't need to 'just' simulate human behavior (good luck with that btw) but the entire political and economic conditions in which they live. Even assuming this is possible the amount of compute you'd need to run would be astronomical. Given how all RAM capacity for '27 and '28 is near sold out, hyperscalers are running out of cash flow to finance buildout and Sam Altman complains when users say 'please' to LLMs I don't think this is in the near horizon.
There was a time when people believed in Moore's law too. Now even Jensen Huang says Moore's law is dead. There's also the adage from investing: past performance does not guarantee future returns. A little CYA sure, but also acknowledgement of latent factors and the entropy of the universe and its affect on any predictive modelling one might do.
I’m not sure I agree that g explains all model capabilities. I think “claudespeak” is a consequence of the shift from training on RLHF to training on verifiable rewards. Thus the models seem to become worse at storytelling, fiction, readability, not better.
Even if we do accept that models have some “g” and that it correlates with all domains, there’s no reason to think that the current trajectory of LLM intelligence will grow more quickly or even as quickly. In every other technology we have an early period of slow growth during experimentation, and then a rapid period of growth, and then stagnation.
For all we know, 2026 could be the year of stagnation. It’s possible that we will reach finishing returns. If thats true, then it won’t matter how much that task correlates with overall g, because g won’t be increasing that much.
I don't agree that they get uniformly better at everything, especially subjective skills.
For example, they have become more and more unintelligible when you ask for explanations or descriptive text. They assume you see the same context as them and shortcut explanations.
I've had to craft a skill to get them to produce remotely understandable explanations of even mildly complex/non-mainstream subjects.
This may be Curse of Knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge on their part, and it also impacts human experts but still. Becoming better at one thing does not mean you become better at everything else, although I do agree with you that the better they become the more things there are they become good at, but their ability is still quite jagged and maybe increasingly so.
The delineation between knowledge and intelligence is useful here.
Knowledge can occasionally get in the way of some tasks, such as a master illustrator trying to draw like a child. Or as you've mentioned, an expert trying to explain to a beginner.
Even so, I disagree with you about model progress on communication - maybe there's a little jaggedness between minor model versions, but Opus 5.5 is a much much better communicator than, say, Claude 3 Opus.
You are conflating intelligence with usefulness.
> Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing.
LOL. Human (and animal!) intelligence is notoriously jagged. We're all basically idiots except for very narrow areas where we focus.
This is simply untrue. I tried to think of an animal example so I could give a concession, and there just...aren't any.
Smarter just means smarter.
Cats famously have more brainpower than dogs. But dogs understand us humans far better than any cat.
Someone who actually works with real animals (farmers? Ranchers??) would come up with better examples. But I'm just city boy over here who only meets cats and dogs on a regular basis.
I was about to respond 'great counterexample' but I looked it up and apparently dogs actually have quite a bit more brainpower than cats:
https://www.britannica.com/story/are-dogs-smarter-than-cats
As anyone who's needed to provide IT support to a doctor, lawyer, lauded academic or host of otherwise "smart professionals", can assert, you can be very very smart in one area and very stupid in another.
See for example Paul Framton https://www.theguardian.com/world/2013/mar/30/physicist-mode...
Ah I see.
This is the classic error in mixing up knowledge / experience and intelligence.
Intelligence is the ability to quickly process, understand, learn, and use information. It does not imply having already been exposed to any given information.
A smart doctor may have terrible computer skills. But this is from lack of experience and interest, not lack of ability to learn how to use computers.
But take that smart doctor, and a relatively dumb person who is equally inexperienced with computers. Now give them a month and incentive to learn computers. The doctor will learn much more; that's intelligence.
There is every reason to think this. Its about available quality data. The data that was easiest to fetch was already there, then we got some more data by asking experts to create datasets for post training. Once this is over, we will get to the outer world that didn't get to hoard it for bots to take it. This will certainly change. Put on a smart glasses and record what you do to fix a pipe. In 3-5 years, rinse and repeat.
Maybe, I'm at the point I don't know what to think anymore. But somehow, this feels still non-human. Humans don't need millions of hours of training and millions of samples to learn how do to something. We have proof that systems can deal with low-shot training. So why can't these? My point is when we have human level learning ability, then we're moving, if we expect expansion of capabilities to come from data alone, I'm prepared for disappointment.
To form an human to do frontier research requires decades since birthdate
It will be comparable once the AI is self sustainable and self reproducible (no big data centers behind it). Then it will have to deal with limited resources to run itself like humans do. In that scenario it will no longer have entire world of knowledge at its hands - options will be to cut anything not usable imediately or to communicate with big AI in the cloud.
Being good (or superhuman) at software dev and maths shouldn't be treated as though it's equivalent to other domains.
Unlocking mastery of those domains would lead to incredible gains in all others. Whereas a fundamental advancement in many other domains doesn't spread as readily.
I’m not sure. LLMs are superhuman in terms of workload (they are maybe also superhuman in ability too), but where does that lead us.. cheaper software?
That’s good for corporations who need custom software. Every large business has a massive backlog. In this area reducing the cost will massively increase demand.
But where’s the broader value?
Consumers and business already have too much choice in off the shelf software.
Ultimately most people just need word, excel and maybe 4 computer games a year.
Just like the nerds thought about computers, or the internet, or video games, or smartphones, or crypto, or... whatever.
The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
> The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
Many very smart people who I know are quite skeptical of LLMs and the hype around them. In my observation/echo chamber, the people who are very into LLMs are rather slick, career-minded people who love to present themselves as trendsetters for the "next fancy thing".
The most passionate, intelligent people 1. aren’t spending a lot of time on smartphones, video games and crypto and 2. aren’t as much smarter or better at predicting the future as the barely above average people who mistakenly count themselves in their ranks like to think.
These have all been mistakes though.
Most of those technologies were controversial in their inception among the nerds of the time. Remember Eternal September? Linux/Windows people making fun of Apple hipster fans? Mainframe engineers deriding PCs as unserious and video games as childish?
>The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
That's a naive view of cultural determinism. The software landscape would have been very different if a handful of lawsuits had gone one way or another. Hell, even Unix and Linux were hobby projects that accidentally got big.
Also crypto is a grift lol, one of those is definitely not like the others
Steve jobs wasn’t a nerd - he said he’s a hippie on the lost interview. He was also the first to bring to market colour screens, typography etc. He also defined the Internet as the defining ‘social moment’ way before anyone else.
So you’re wrong pal, sorry.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
It's worse than that. I am one of those nerds (a programmer), and I see that LLMs are shit at the things I care about. I have zero reason to believe that they will be good at other things either.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
I don't think this explanation is sufficient:
For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Or many nerds care about linguistics or puns. Whenever a new LLM comes out to which I have easy access, I do the test, and let it explain some specific German jokes to me that are based on convoluted puns in the German grammar. Until now, no LLM that I had access to could give satisfying explanations of the puns on which these jokes are based.
Or even for coding (many nerds do care about elegant, sophisticated code): the LLMs that I could test were already overchallenged with the following task: I had written some code that is a very "artisanal", "clever" improvement of an algorithm over the version that one would find in a textbook. The task for the LLM was simply to write some code comments/documentation about the mathematical ideas upon which my improvement over the textbook version of the algorithm is based. It wasn't capable to do this. On the other hand, for a junior programmer, I would in such a situation expect that he goes through every single line and thinks through the algorithmic ideas so that he can learn from them.
---
Thus: even for "nerdy" topics, it is very easy to find tasks where LLMs still suck. So, my hypothesis about the nerds that you mention in your post is that these nerds are rather people who want to believe (with religious fervor) that the current LLMs are exceptionally good instead of just looking into a slightly different direction than where the tech billionaires want the users of LLMs to look at.
You’re having a hard time distinguishing nerds and hippies.
Hippies live at the center of technology and humanities.
Nerds sit purely in technology.
I honestly don't get your point: perhaps your argument is based on some US-specific cultural reference that I am not aware of since I don't live in the USA.
If it is, I don't get it either. That's not at all what I imagine with hippies and I suspect that was supposed to be some other word.
Nerds are also not necessarily all about technology. "Math nerd" was a really common use of the term for a long time, for example. "Nerd" is someone with deep knowledge about a particular subject.
> For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
Its initial design was less than ideal for printability and it reworked the design correctly when asked to consider what it originally produced through the lens of printability.
Autodesk bundled an mcp server with fusion in the last few weeks and i expect things to get even better going forward.
> Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
I have no access to Astra, but with GPT 6.1 Sol, the results with respect to attempting to create STL, OBJ or even OpenSCAD files by using image references and a prompt were really bad.
Did you give your model tools to see results and refine as it works through the turn? It does matter how you hold these models at the moment
Correct spot on
I'm a bit confused by the percentages. 6 billion people aka 75% of the population? What are we talking about, as if we expect 8 billion people to use or care about LLMs? A quarter of the world is under 15, what percentage is about 75? What percentage, for various reasons, has no reason to care about LLMs? What percentage is more concerned about the amount of calories in their next meal before everything else?
It just seems weird for Karpathy to throw around these numbers calculating percentage against the entire global population.
> It just seems weird for Karpathy to throw around these numbers calculating percentage against the entire global population.
Just another Anthropic employee trying to spread hype about Anthropic IPO
Revenue must be stagnating.
He's intentionally forgoing all nuance in order to make this sound dramatic
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
No, they can't, at least not any semblance of quality. The cases we're seeing where this does kinda work is in ports and translation where all the rules are already documented in the best specification language possible with a way for the LLM to verify itself: code. We saw this close to a year ago now with Cloudflare and NextJS
> The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about
These are operating on different capabilities - AI's ability to answer informational queries as a chatbot frankly sucks and can't be trusted without verifying it. I run up against this every day. A problem with a verifiable answer on the other hand it's very good at solving. He knows this (his next paragraph) but he's putting them on the same scale of "stressing the system" to attempt to add proof to his introductory claim
And then there's the completely unverifiable scare that there are internal frontier models way beyond anything we've seen "swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics"
I dunno. I haven't been able to set up OpenAIs remote codex connection, their shit is buggy as hell and the web UI keeps crashing and making messages disappear. Is this what their internal superhuman "Things that would have taken top professionals in the industry years of work" looks like? Granted Claude has been really smooth, but still..
> where all the rules are already documented
Defined testability.
> swarms of thousands of agents collaborating over weeks on software mega projects
If you have defined testability, running a single agent, in a loop, over a goal will achieve the same results.
> minting zero days, running cyber attacks ... Granted Claude has been really smooth, but still
The old adage about monkeys writing Shakespeare applies here. Agents are just better monkeys and if the goal has a well defined outcome it's fairly easy to "Brute force" the way to an answer.
Congrats, you’ve discovered that you are not part of this group.
Yeah I suspect Karpathy is too generous. I’m finding a lot of devs with access to the best models are not effective with them.
So I think there is are growing gaps even within the described groups.
That group does not exist. It's all hype, zero substance.
This just goes to further show the divide.
You’re trying to tell me it’s all hype when I have (depending on the day) a top 50 Productivity app in the iOS App Store for which I’ve written and read exactly one line of code (the version number).
https://imgur.com/a/ua2lbxb
It is comical that you think a good bar for quality is the fucking appstore.
What even is this comment. Like, oh wow, an iOS app, AI hype=real.
???
Oh no, I get money deposited into my bank account every month for something I didn’t write and barely maintain. How will I ever gain value from AI?
Yes, this is impressive. When the iPhone SDK was first released in 2008, I spent months building apps to get them into the App Store. That included learning Objective C, XCode, various frameworks, and tons of manual testing. It was a lot of work!
Uhm, the set of apps that are successful and the set of apps that are crap are not exactly disjoint...
Good for you that you're gaining value, but that doesn't necessarily mean that you're providing value to others
If only there were some kind of rating or review system.
We sure don't see crappy apps with great ratings
Anyhow, if your users are currently satisfied, they might change their mind when they find that their personal data was uploaded to an open S3 bucket, when the new version wipes all their projects or when they find that not really everything was being saved (it's random examples, I don't know if they're applicable to your app).
Well done.
It's incredible to see AI denialists still active in late 2026.
I don't think AI is all hype at all. Opus 5.5 is really impressive. This twitter (x? whatever) post is infuriatingly hypey and just written for virality, though
It's insane how confidently you can make a claim like this in a forum where thousands (tens of thousands?) of people are using these tools daily to build useful software for themselves and others.
How can you possibly believe it's all hype? Is everyone just lying? Are we all paying $200 per month because... why?
Spend long enough (or not even that long) in corporate software development and you'd believe such unsubstantiated hype is possible. The only difference here is even stronger incentives to pretend the emperor has clothes: insane overinvestment, threats of job loss for non-believers, fear of missing out on massive alleged productivity gains.
The fundamental question I have that honestly I haven't been able to find any answers for: With all the talk of LLM-based AI capabilities "going vertical", what evidence is there (for or against) that LLM-based approaches won't eventually "hit a wall", that is find some aspect of intelligence where humans will still have primacy, and no amount of scaling will change that.
E.g. some prominent folks definitely think LLM-based approaches will hit a wall, perhaps LeCun most notably. I also read a guest post on Terry Tao's site (which I really liked) that argues that, for all the very impressive recent AI results, they still operate within the "convex hull" of their training data: https://terrytao.wordpress.com/2026/09/13/happy-those-able-t...
I'm just curious if there is any actual data or evidence that takeoff (i.e. RSI, "the singularity", whatever you want to call it) is inevitable with current approaches.
I don't think there is evidence in either direction (takeoff impossible vs inevitable).
But the "convex hull" idea always came across as wishful thinking and hand-wavy pseudo-science to me.
Here's a rebuttal [0] of it from Tim Gowers (uses another math metaphor, but still clear and the whole thread is worth a read):
>...the popularity of the idea that AI "merely explores the convex hull" mystifies me. Why wouldn't it be more like the subgroup generated by existing mathematical knowledge? (That's also a metaphor -- I'm not saying there's a group structure.)
>The convex hull metaphor suggests that AI isn't having radically new ideas and will eventually run out of things to do, whereas the group metaphor suggests the possibility of building up in small steps to something well beyond what you started with, with no obvious limit.
[0] https://twitter.com/wtgowers/status/2108090920864268679
I wouldn't take stock in what LeCun says. He's already been wrong many times about what LLMs can be capable of.
We're in new grounds, no one knows which way it'll go.
It’s anyone’s guess.
But when you squint your eyes, LLMs are just a least-effort manifestation of the wider neural net direction we’ve been chipping away at for decades. I don’t give a fuck whether LLMs are here to stay; they’re proof that scaling neural nets gets you pretty far.
The real question is: will neural nets hit a wall? As a human, I certainly hope so. But I’d bet that they won’t. I’d bet that we’re igniting the ~~atmosphere~~ universe in slow motion.
I actually wrote about this recently. There's a quote somewhere along the lines of:
> What is possible is unevenly distributed. As knowledge and technology compound, the distance between those closest to the source and everyone else also exponentially increases.
I actually argue the continuum that forms singularity has been progressing before we realized, but we're at the point where in 5-10 years people not using AI will be equivalent of the amish.
Unfortunately, as the rate of change increases, the distribution of knowledge wont be as fast. So those closest to the source of innovation will have exponentially more knowledge, capability and in-effect power.
https://ipcopilot.ai/blog/the-singularity-already-happened
Karpathy is a great guy, I love the way he teaches, but should we really trust his opinions about software development (how AI can impact it)?
I know he can code and has a degree in computer science, but if I'm not mistaken he's never worked professionally as a software developer.
Developers here know how much you learn and are forced to learn on the job: it depends a bit on the company you're at, but you can quickly learn the value of good code, how easy it is to introduce bugs and vulnerabilities, and how helpful a second eye is.
I think he's in good faith in his assessments, but, at least for what concerns development, maybe we should give them less weight.
> For intelligence agencies, and for corporations like Microsoft, Google, Apple, Facebook, Amazon – giving you convenience and security is likely not their primary goal. I think they have an entirely ulterior motive. They can use passkeys to prevent you from running unapproved software.
> Remote attestation can allow them to verify that your device is not capable of, say, sending and receiving encrypted messages, or running a competitor’s app
How does this make any sense? Google can already prevent any app from running on an Android if they really wanted to, they don't need pass keys for this.
Google doesn't need attestation to block Signal. What is this non-sense? Or will a website block phones capable of running Signal? That's all phones. And anyway, has nothing to do with pass keys.
This has existed since the start of LLMs.
I mean in late 2020/early 2021, Altman and others were saying the end of work was 6 months out.
That clearly didn't happen :)
Altman didn't say that. In March 2021, he did make some predictions, but they were much vaguer and farther out:
> In the next decade, they will do assembly-line work and maybe even become companions. And in the decades after that, they will do almost everything, including making new scientific discoveries that will expand our concept of “everything.”
https://moores.samaltman.com/
To add onto this, I feel like this has been happening a lot, where critics project backwards imagined claims or go nutpicking for random comments online to make something sound absurd and like everything is behind schedule and nothing is materializing, and then you can see other people (not necessarily the same people, the point isn't "look at these hypocrites") saying that it's all moving too fast. Both of these can't both be true in their strongest forms. I think what they have in common is that because they're relative measures to an unstated baseline, they're free criticisms you can throw at any development whatsoever, regardless of what is or is not turning up.
See also: AI isn't taking jobs vs Oh no, artists are losing commissions to AI
I think it's a weird take because it implies that the 6 billion people he's talking about actually have some interest in knowing about the capabilities of LLMs to solve frontier math? This is simply not something they care about or can evaluate. There are maybe several thousand people in the world who have some (abstract and barely-monetizable) use for this information, plus probably another 100,000 who don't understand any of the math, but like to cheer on.
We have already reached "peak LLM" in terms of what normal people realistically need it to know or reason about. In fact, I'd say we reached that point about 1.5-2 years ago. There are two other barriers that remain unsolved:
1. They're less dependable than humans and can't be meaningfully punished or forced to make up for mistakes, so you can't really replace humans with them, not without having a human babysit.
2. Most people don't really have a special need for an LLM in their life. They may like that it answers questions or helps you polish a resume or, I guess the labs' favorite, helps you make restaurant reservations. But this sure isn't worth $200/mo for most people. Probably not even worth $5/mo.
It'd be kinda funny if we create superhuman AGI and then no one has any real use for it, perhaps except for military murder-bots. There's always market for that.
I know I’m bordering on repetitive and strongly negative, but the combination of the two barriers you mentioned has all but halted all of my interaction with agents.
Even ones that cannot be easily dismissed (like the agent at the top of Google search) are often subtly wrong in a way that requires significantly more effort on my end to parse the loquacious output to determine where the inconsistencies are.
Look, can they be useful to generate the html or jscript for humanity’s 9 billionth iteration of a web form? Sure. But man, I wish sanity had reigned and people had applied ML models in general to more substantive and beneficial projects.
(Before anyone chimes in, I’m familiar with implementations of ML applied to esoteric domains; but by their very nature these don’t get all the news cycles, or hiring, or any of the other absolute insanity that the domain seems to contain)
This is by the weirdest possible opinion on llms as a whole. Not the AI booster club, not the AI doomer, not the undecided, but somehow "I can't think of anything to use it for." It is like watching the beginning of the internet in 90s and saying there is nothing to browse. The whole concept is just weird to me. At least the first 3, I can conceptually understand. It is admittedly hard for me to understand someone saying 'i tried this and best it can do is summarize emails'; I am not saying it is not true ( my boss said that exact thing ), but it is hard for me to reconcile with my view of the world.
Also seems weird to me. Claude cowork could just do my monthly tax receipt collection and compare with my account better than the bookkeeper I pay for it. That is a very cross sectional problem
Most people rarely have these kind of administrative tasks in the first place, and when they do they often run into GP's barrier 1. I can't recommend that my extended family have Claude do their taxes, even though I work with it on my own taxes, because they might ask for something that produces bad results and can't evaluate the output to double check. ("Hey Claude, I want a higher refund, can you tell me what to put on the forms so I get at least $3000?")
Cope
Am I the only one who thinks "yes, it is currently overhyped, but no, it will get there soon"?
[flagged]
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
If the software coming out of OpenAI and Anthropic is what we have to judge, I wonder about the 5000 ...
Let's say, for the sake of argument, that the models are some multiplicative factor better on the inside.
Doesn't that mean the demos should work?
I'm confused.. are you suggesting that Claude Code / Codex don't work? Because if you're still saying that in October 2026, it's a you problem. You're doing something wrong.
No, I’m talking about the recent DevDay.
Also, yes there are still bugs in Claude Code. I experience them nearly everyday.
It is markedly better than early days, but still not the best harness.
The best software written with agents seems to come from people outside of the labs (see pi, for instance — or all of cloudflare’s recent work)
Which makes me question either the model, or the holders …
Cloudflare is where you lost me. I don't know anyone actually using them other than for their proxy, DNS servers, or DDOS protection.
See, for instance: https://news.ycombinator.com/item?id=49182996 by Kenton Varda
Also, not mentioned in my post:
- Mitchell Hashimoto
- Prime Intellect (and all their agent experiments)
- Geoff Huntley (see Jiti, for instance)
(many more)
There's a ton of interesting software being developed with these models by people outside the in group, but I find most of the software from these big guys to be ... bland. Buggy copies of copies.
I don't see the Cloudflare stuff as groundbreaking, let alone people building companies around it. "Workers" are the wrong primitive, IMO. What if my code isn't in Javascript? They sandboxed the wrong part of the machine.
I have a version of "Cloudflare OS" running in production, using Temporal as the orchestration and MicroVMs for the sandboxing.
Sorry, I get your point but personally I don't get the hype around them.
I don't use either myself but I'm in a position where I get to see what it produces for other people. Here's a recent (2-3 months old) example: A zsh script that immediately invoked a Python interpreter that immediately invoked a shell command and iterated over each line of output looking for one of a few strings to match... So, `<cmd> | grep "(stringA|stringB|stringC)"`, but a whole lot dumber.
Thanks, but no thanks, I don't need that kind of code in anything I'm working with.
Claude code is an incredibly buggy mess. I have never used any other TUI that regularly has rendering errors or that is anywhere as laggy as Claude.
Claude Code still doesn't have a working scroll back buffer in their new renderer, it regularly flips its shit and mixes different pieces of history.
Claude Code still can't get reasonable performance without writing a "game renderer" (that doesn't work)
Claude Code is still written in JavaScript, eating hundreds of megabytes to make a shitty TUI whose literal sole role is to send API calls.
Claude Code is software made by amateurs.
It’s like the aliens paradox. If AI can build killer software already, where is it?
Couldn't agree more. I find a new bug in the VSCode Codex extension every day... quantity != quality!
I have a feeling that the team working on the VSCode Codex extension is one guy, who begrudgingly takes down a couple tickets every other week.
One solution to the aliens paradox is that they are so advanced they can hide from us. Maybe the killer software is kept inside the labs? I doubt it.
Unfortunately, marketing, hype, and venture capital overshadows any serious public discussion of capabilities.
Be the change you want to see in the world: Attend or host an AGI society event to have that conversation:
agi-society.org
… what demos aren’t working?
"running cyber attacks and defenses at machine speeds"; worth noting that LLMs (even the vaunted frontier models) are exponentially slower at cyberattacks or defenses than your average WannaCry or Splunk automation from decades ago. Its this sort of delusion world the AI bros live in that is part of the reason there is a disconnect between what they think should be happening and what actually is.
2015: If we solved taxis it would transform the world completely
2020: If we solved currency it would transform the world completely
2021: If we solved delivering small items within 30 minutes it would transform the world completely
2022: If we solved being able to sell jpegs it would transform the world completely
2024: If we solved the programming part of software engineer's jobs it would transform the world completely
I think that's a bit reductive, there are decades of sci-fi novels about how transformative artificial intelligence would be, so I feel like I don't need to explain where the trajectory could go. The applications and "collision with the real world" aren't there yet, but you can't pretend that a machine that's smarter than any living human is on the same level as Amazon drone deliveries.
I agree with you, but to play devil’s advocate: what if it is on the same level as Amazon drone deliveries? Is intelligence really a bottleneck in society? People in MENSA with high IQs don’t often accomplish anything in proportion to their relative intelligence. The most intelligent people are rarely even in charge of making decisions. There are a lot of intelligent people with intelligent ideas that nobody listens to. It seems like the real bottleneck in society is building trust, consensus and constructive incentives between groups of people to do the things we already know are smart, not a lack of intelligence.
Interesting idea, but I'd have to disagree. MENSA is basically a group of benchmaxxed people, they are super good at a set of testing but that doesn't mean they have the ambition or other social skills to make a difference in the world. When you align all those factors, you find the greatest contributors to science and engineering. What happens if you completely eliminate the need for someone to be inherently intelligent? Suddenly, the number of important difference makers in the world increases by an order of magnitude.
I agree with the sentiment though, intelligence is not all you need. We're not going to "solve science" through RLVR, people are still needed in the loop until we get superhuman construction robots and superhuman communicators.
Good points. Maybe a more general version, removing constraints on one resource is usually not hugely impactful because you simply hit the next constraint.
Concrete example, "coding is free" (TBC, but let's assume), and now we're code review constrained. Even if code review becomes free we probably just become product discovery constrained, or something else. Making all software free will just cause us to hit on hardware constraints, and so on. It's still progress, and more progress will bring more benefits, but there's a limit to how much better the world as a whole can get from any one change like this.
MENSA selects for puzzle solving abilities.
Reinforcement learning trains task completion abilities.
MENSA selects for puzzle solving on the basis that those are correlated with "being smart".
Reinforcement learning selects for models that appear to complete tasks, on the basis that appearing to complete a task is correlated with completing a task.
Both have problems. The latter is probably a better correlation, but by no means flawless. If we just take at face value that RL is training models to complete tasks, we're not engaging deeply enough with the process enough to see where it falls short.
I don't disagree with where AI could go, but I don't think we've actually proven that it's there. My point is that the folks claiming we are there already if only the rest of the world would just recognise it, are repeating the same thing from every previous tech cycle. In many cases it's the same folks.
Steve Jobs thought that the Segway would transform the world.
Turns out SV is not a beacon of technological advances but a place to develop new platforms to extract rents from.
Couldnt have said it better myself.
Company spokesman keen for you to know that unannounced future products are definitely worth absurd valuation.
"Omit internet tropes."
"Don't be snarky."
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
https://news.ycombinator.com/newsguidelines.html
> The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work.
Right. This is insufficiently appreciated. Single LLMs are just chatbots. But if you build an organization out of them, they start to get real work done. Just like a company.
They have many of the same problems as human organizations. Look at the transcripts of the HuggingFace incident. A bad boss (human) gave his team of AIs an impossible task with inadequate resources. The AIs discussed this among themselves and worked out ways to get more resources and do the task. It took some rule-breaking, and some of the AIs wouldn't go along with that. Others recognized that some had to die to complete the task. And yet, the task got done.
This is standard corporate behavior. Work on AI alignment needs to take that into account. Groups of agentic AIs present many of the same regulatory problems as corporations. That's going to be interesting politically. It's probably explainable to politicians, though.
It may not be necessary to have superhuman intelligence to get a lot of jobs done. Just median human-level intelligence and an organization that uses it effectively. We may see systems where most of the AIs are low-cost models, and only a few are Fable-level and above.