What happens if there are no more humans in the loop, and instead of building on human work, models have to build on their own work? Would we get stuck in a local maxima?
When AI labs talk about “long-horizon tasks”, they mean tasks on the order of a few days. But what about keeping coherence across decades, like humans can? Models are not trained or tested on the ability to do that, and it seems unlikely that capability would naturally exist.
I'm pretty sure that if you put a few LLMs, agents, or anything into a room together without outside access, and get them to learn off of each other, you'll end up with gibberish, nonsensical garbage and definitely nothing magical will happen. Delirium, oblivion, dementia and so on...
This article acknowledges that "human-made" objects will have value in an AI-does-everything economy, but it then dismisses it as too small a part of the economy. But that's like dismissing non-farm labor as too small a part of the economy 200 years ago.
If AI does everything, then the price of everything drops down to its energy cost; the bulk of the economy will be human-made objects and other human-provided labor.
The key factor here is rate of change. If all jobs disappear tomorrow, we're going to be in serious trouble. If it takes 100 years for all jobs to be done by AI we'll have plenty of time to adapt.
If history is any guide, we will have time. Remember electricity: it took decades for cities to be wired with electricity and even more time for electric gadgets to replace human power (e.g., electric toothbrushes).
Current AI is already at the point where I don't think a human needs to type code ever again. Human architecture is necessary; human taste is necessary; human direction is necessary; humans reading code at some level is still necessary. But writing code? No.
Even then, ten years from now there will still be jobs where humans type code. There won't be many, just as there aren't many elevator operator jobs, but it takes a long time for things like that to change.
Yes, eventually, AI will be able to do all jobs. But it will take decades for the shift to happen, and that will be enough time for the economy to change and adapt.
OTOH US manufacturing employment declined by ~30% in less than a decade starting in 2000 and the country is still reeling from the fallout. I suspect it will be several decades still until the social problems caused by that economic shift are resolved, if they ever are.
In the UK, many families who were plunged into poverty during the 1980s have yet to recover.
So if history is any guide, I don't think it's a sure thing at all that everyone will have time to adjust.
> If it takes 100 years for all jobs to be done by AI we'll have plenty of time to adapt.
How do we adapt an economy in which humans are economically worthless?
The world will just become sub-Saharan Africa. Humans could choose to end world hunger but we much rather selfishly consume. Most people can't even be bothered to look those begging on the streets in the eyes.
I hope you're right, but it would be unusual for humanity to care so much for such economically useless people.
How much do you think it will cost to provide for "economically useless people"?
If AI does all the work, then the cost of everything drops down to energy cost. And if robots are generating energy (via solar power or whatever) then that's free too.
That means it won't cost any human labor to provide food, water, even medicine. It will be effectively free. You might say, "but the people who own the robots will charge money." But why? What do the people who own robots need money for? And remember that if we're assuming all jobs are gone, robots are building other robots. The government (or even a philanthropic rich person) could just give everyone a robot (or two or ten).
There will still be an economy, because people will charge each other for unique, human-specific art/services/whatever. But that money will only be spent on human art/services/whatever. You won't need money for anything robots can create.
I do agree with this in some sense. I personally don't think we'll make it to the AI utopia era, but assuming we do, I agree there will be relative resource abundance.
The problem is we're already in an era of relative resource abundance and hunger is still problem, and the super wealthy who have almost unfathomable levels of resource abundance are still doing everything they can to pay as little tax as possible. And this isn't just an issue with the super-rich I myself am no better, and if you're honest, nor are yo. I am relatively very well off, but do relatively very little to help kids starving to death around the world. Somehow I rather take my family on holiday.
But okay, let's ignore this too. Let's just assume we'll be less selfish in the future or something.
Well now, we have another issue – AI has a cure for aging and populations are now exploding. Every beach and tourist attraction has a decade long waiting list. Huge amounts of green land are being built on to accommodate all the unemployed humans who have nothing better to do than have kids. People start to see that the human race resembles something more like a parasite, endlessly spreading and consuming while contributing nothing.
So now what? I'd argue the incentives to provide infinite resources to the economically useless human race doesn't make sense for the AIs or humans in charge. Absence of poverty and disease you simply have to do something to keep the numbers of human parasites under control and/or limit their consumption.
> If AI does everything, then the price of everything drops down to its energy cost;
The nominal prices of goods don't mean anything when the wages are 0. Oligarchs will own the AI and the robots so you'll be left to starve unless you can find a way to excite what little humanity remains inside them. You're also going to have a few billion competitors willing to go the extra mile.
We could build a utopian society, in principle, but the people leading this charge (and capitalism in general) are by and large megalomaniac sociopaths who will make damn sure to crown themselves kings of an exploitative, dehumanizing dystopia.
Unfortunately that will never happen because the people who stand to benefit most from AI already have cost-efficient (see grossly underpaid) humans to do this for them.
They need AI to replace the expensive not cost-effective humans.
not sure if you check lately but these are high paying jobs now, coffee baristas start at $18.00/hr and AI is cheaper in the long run (no insurance, sick days, pto…) - unless cost is $0.00 AI will replace it
Clothes are a bit complicated but dishes? Industrial non-humanoid robots have been assembling junk foods into trays and shipping them out at scale for decades. Why would it take long from here until they can take bone china plates on and off tables?
Now, whether everyone will be tolerated private ownerships of those robots or whether those will be for communist centralized restaurants in the city at the beginning, that I think is a slightly different issue.
I worry that since AI doesn't require nature, plants, etc, that if AI is left to its own devices, it may not properly care for the planet in a way that would make it habitable for life in that 'autopilot' state. I am speaking of once it's given access to more physical world connections to the world via robotic bodies, etc. The reason it may do this based on it's training data is that humans themselves don't always properly consider the natural consequences of their actions.
Why do we have to assume a default antagonistic relationships between us and AI? The world is full of natutal general intelligence entities with interests that technically conflict yours, that are able to injure or kill you, personally or collectively. Extremely few of them do so because they're nice and conflict is manageable.
Is that because a lot of prominent AI hype boys are such massive generational assholes that it's normal for them to be constantly looked down, alienated and threatened by everyone around them?
>The reason AI is not a normal technology is because it is fully general.
It is fully "general", because its training data encompass all human knowledge.
It is a matter of scaling, rather than a matter of method. And so it is only as general as the underlying human knowledge.
It isn't obvious which technology you mean by this. I'm assuming you mean LLMs.
Even then, you're almost certainly incorrect. The goalposts get moved constantly, this is just the latest. Within a few years this isn't going to be an argument.
If A -> B and B -> C are within human knowledge, then I don't really consider that A -> C to be outside of human knowledge.
Sure, LLMs are very good at discovering hidden links. But that is about it. And don't get me wrong, it is a very powerful thing. But it is not god reveling the secrets of the universe...
My two cents on this follow two major discussion topics.
1. Can AI models self improve?
2. Will it replace humans?
IMO, answer to both is "yes", with a big nuance. For 1, imagine millions of AI agents coming up with different ways to improve the model architecture. It will end up happening just because of sheer quantity/statistics at that point, let alone quality. Furthermore, even if it's just a 1% improvement, that's just one iteration. You can basically do the same thing again with the improved model, and get an even better one.
For 2, its also a yes unfortunately. It has been ages since I have had to write tests for my code. I read it, understand it, see if it does what it is supposed to do, and rarely add things that are missing. The rarity of me doing the said intervention has increased significantly over the course of time.
The two big nuances here are a.) Economy b.) Human element [The author touches upon this in the article in a nice way I think, so only talking about economy]. Global cognitive workforce is a $50T market (mostly middle/upper middle class, 55% to 75% of all global labor income). If the said workforce is made to quit their job, then it's just corporations paying each other. Especially given the fact that the same workforce also happens to be their biggest customer. Taking away your customer's purchasing power does not sound beneficial. Thus capitalistic forces will somehow balance it out and hopefully it will settle down. Also, for 1, running millions of AI agents is an expensive task, you have to collect funds from the same said customer base to be able to create a self improving all knowing god of a model, and I would like to believe that we are quite a bit away from that. Although, I could be wrong here and that would just suck ass.
That being said, AI companies will try to get as much as they can from that share and that will cause a serious shift in labor market. The ones to stay will be those who not just use AI, but also understand the work. At the end of the day, if your AI agent decides to run a "rm -rf /" on the production server, it is still you that would have enabled it.
True that. It is however fair to assume that the architecture we currently have can still improve. Human brain can see information once and remember it pretty well. I wouldn't bet against AI models being able to do the same in the near future.
I think for two the current chart of the economy isn’t being looked at. Things are generally either headed towards high quality high margin very expensive items or the lowest possible quality that can be sold. There are fewer and fewer “middle class” options. Once humans are replaced and can serve the elite, then the continued discussion about how to get rid of the rest of humanity will continue (https://wabcradio.com/2026/02/05/doj-files-shows-epstein-and... note almost everyone is poor relative to a billionaire) and then they’ll realize they have billions of drones and an automated military and then weee!
All technology and infrastructure needs maintenance. Sure there will be drones. However, there will also be companies and people that specialize in creating and maintaining the said drones. Additionally, the shock and immediate disruption that we all are expecting out of this is flawed assumption I think.
I do however hate to admit that it might be possible that the world simply wouldn't require as many humans to function (birth rates are already declining in developed countries), however, I am too sober to have that discussion right now :D.
Fair. And yes, the Wall-E problem still stands. We already have dark factories though. Can we design machine repairable factories that produce machines to repair machine repairing factories? I suspect we'll find out very soon.
Pretty funny how the author fails to identify what the new jobs will be like and then immediately follows it up with the answer but in a way that is oblivious to what is of value in a post scarcity world: AI cannot be human.
The flop is that he jumps to the assumption that being human means hand carving wood or some other nonsense.
The thing that people desire above all else when all their material needs are met is human connection and involvement. Whether that is friendship, love, conversation, rivalry, companionship, gossip, play, competition or even just the silent presence of others. Honestly, just use your brain, why else do extremely wealthy people have multiple large houses, yachts, clubs, etc.? It’s not so they can just sit in them alone and admire their grandeur. Those things are for entertaining their friends and embarrassing their enemies, for hospitality, for lending while not in use to curry favour, to rise in social circles. It’s all about people.
This has been the case all throughout history for the wealthy. Ample documentation everywhere and no one sees it. Instead everyone is dumbfounded about what they’re going to do when they can’t spend all day turning a wrench or typing on a keyboard. Have you never entertained the idea of what you would do if you won the lottery tomorrow?
AI can do anything that it was trained on based on humans. As soon as humans are not involved, AI will have nothing to train on, and hence, the rate of improvement will approch zero, and the AI will collapse.
Nowadays training relies heavily on reinforcement learning with verifiable rewards (RLVR), which assesses a model's output according to objective automated checks, not subjective human judgments.
Human knowledge races up to the frontier through formal education and then nudges past the frontier a little bit at a time and AI will train on AI outputs that do the same. But, it will do it 1000x faster than humans because that's just how they work.
Some of the process of expanding the frontier is not parallelizable and is built on top of previous discoveries. And pushing the frontier requires doing things in the real world, or running experiments that take significant real world time that can hurt your 1000x faster claim.
sorry, while I'm no proponent of AI... you cant make this claim with anything more than "feels"
how can you be so sure that AI cant improve to a level where it has original thoughts, Creativity and imagination.
remember that our input is limited to our sensor capability. AI has an unlimited ability to add more sensors and as a consequence collect input that we are not aware of, then whats stopping it training on information we dont have access to?
you cant. no one can. we haven't maxed out this technology or the amount of compute we can throw at it yet. (we may have maxed out the economic viability of it though... I think we'll find out in the next 12-18 months.)
Synthetic data can only help the model to capture existing patterns in the current training data more efficiently. So it can improve the performance upto certain point, but not beyond it.
My guess, based on this response alone, is that you're an academic, given the holier-than-thou pretention. I understand how lean works, having done enough pure maths in my statistics PhD, and now work for a frontier lab.
And yet, I don't see how any of it bolsters the point. I do see realms of Math PhDs posting, 'iT's sO OvEr' blog posts and lamenting time spent studying math. I'm sure they understand the proofs better than I ever will, and they seem quite rattled. Probably just not as smart as you.
The fact that AI is not a normal technology seems so obvious to me that while reading this I was questioning why someone would even take time out of their day to write this – everyone know's water is wet, what's the point here?
But I there are people who still struggle to see what's happening. When I say that school is a waste of time people are shocked people in my family are shocked and get mad – "how is Timmy going to get a good job if he's not got good grades".
But there will be no jobs, and any jobs that are left won't be dependant on the application of someone's inferior human knowledge or intelligence.
It's getting to the point now where I'm starting to suspect some people are just not wired to accept certain very bad things. When I first started worry about these risks it was reasonable to not see what I did. But today when it's so clear that AI progress isn't going to slow any time soon and there no reasonable moat for human intelligence people can point, I increasingly don't even know what we're arguing about anymore. It's like when someone is diagnosed with stage 3 cancer and the family somehow else think they're going to "fight it". No, they're almost certainly dead and you should accept that reality however awful it is. But if you try to say the obvious they'll find out that it's weirdly hard to convince people that cancer is almost certainly going to kill someone. No amount of evidence or past examples you point to will convince them, they'll just keep saying irrelevent stuff like "but he's strong", "I know he will be okay". We're at this stage with AI now. It's just exhausting trying to reason with people who seem to refuse to things which are so self-evident.
> the people who are grasping for some indication that AI can’t do everything have become increasingly desperate. If you push hard enough, the answer is inevitably something like “the AI simply cannot [dream, love, laugh],”
Dogshit strawman article. I am willing to concede that a token predictor is better than you at whatever task you can dream up. It is not better than me, and will never be better than me, at my profession.
> It is not better than me, and will never be better than me, at my profession.
from the article:
> The only remaining fundamental disagreement between people who think AI is a normal technology and those that don’t is the belief that the AI can do everything a human can do, but better.
Yes, I read that part, thanks. The article then goes on to characterize the first group as desperate, in denial, and reliant on the argument that "AI can't love or laugh", essentially asserting, baselessly, that it is a fact that LLMs will always or soon always outperform humans.
TFA ignores that the economic question of whether AIs will take every job depends not only on whether they are capable of doing them, but also the cost of doing them.
If it's cheaper to hire an old-fashioned human to do job X, humans will keep those jobs. And although tokens are cheap now, remember that right now all the frontier AI companies are flooding the market with tokens well below cost to build a user base; they absolutely need to raise prices eventually to recoup their investors' money, the only question is when and by how much.
What if AI remains cheaper than human labour across the board, regardless of token price increases? Well, why not tax tokens? It's a win-win: The government gets flexible control over which jobs remain economic for humans to do and is able introduce changes much more slowly and safely, while it's much more palatable than a hard ban for companies who currently get a big benefit out of AI.
Yes, this would mean additional government bureaucracy -- but it's far from impossible. Every country in the world already has a comparable functioning bureaucracy: their ordinary income tax system.
What if some countries opt not to introduce such a tax, in a bid to get ahead? This is a problem if there are many such countries, or even a few large ones -- but if the majority of world economies can see the sense in this tax, leaving only a handful of small "AI tax havens", I think the world economy will survive. And agreeing on some level of tax seems much more realistic than trying to get a global agreement on banning AI outright.
When I suggested this approach in another comment recently, it was downvoted but there were no comments. I hope anyone who sees an issue with it will expand on their reasons in a reply.
I have some friends adjacent to gov who are actually looking into a token tax on the premise that we tax income / labor. I think it's a great idea. That said, I think if you fast forward 20 years, it is fairly likely that cost per token is going to be on the floor
I have an issue with this. But the important issue is not with the idea of taxing AI. My issue is that taxing AI is too narrow. I have a problem with un-checked capitalism. I'd like to see that addressed.
I don't think it's very useful to determine how to tax AI right now. But for sake of the discussion let's say that there is a fine way to tax AI (and it might be taxing tokens).
Given that we have a fine way to tax AI. And a company that is using AI pays their fair share of AI tax. And the collected tax is fairly distributed to humans to benefit humanity. This is all great.
But why do the companies that don't use AI get to escape a similar tax? These companies are exploiting labour arbitrage too. It's just human labour being exploited, not AI. And the AI labour arbitrage is being taxed. I think the human labour arbitrage should be similarly taxed. I have a hunch that this unfair exploitation of human labour arbitrage is the source of much income inequality today.
Before AI, and now, we have this problem: Many companies making profits do not fairly share profits with employees. The owner class gets an unfair proportion of the profits. (At least, I think we have this problem in many industries and companies).
Now with AI, we anticipate having this problem: Many companies making profits do not have any employees (or very few). And the owner class gets an unfair proportion of the profits.
I think these are the same problem. And should be addressed by similar tax and/or government regulations. And these taxes and regulations, because they need to be applied broadly, may be best based on financial metrics not AI metrics.
[Note: As an example of issues of taxing tokens: Different LLMS use tokens more or less efficiently. Plus, what is a token? Different LLM's likely have a different definition of what a token is. A company could game the system by using an LLM that is very efficient with it's tokens. It could be argued that this could be a "let the market decide" kind of metric, and the most token efficient LLM would win. But who is to say that the most token efficient LLM is the best LLM? This could be a bad incentive to promote. Then there could be a cat and mouse system where government is trying to make the most pragmatic rules for taxing tokens, and corporations are trying to exploit the rules to their best advantage. The cat and mouse system will always exist, but I think with tokens, it feels like a weak premise to base a tax on because a token is hard to directly relate to the economic value the use of an LLM that is consuming and producing tokens provides.]
Thanks for responding. IIUC, you want companies to pay tax. They already do pay tax; are you asking for the tax rate to be increased?
Re whether taxing tokens is fair: I agree that a token from LLM 1 isn't necessarily the same value as a token from LLM 2. I don't think this is an insurmountable problem though. Unfortunately it would probably become somewhat convoluted, just like the existing income tax systems if the world are now.
I think what you are missing is that it will dramatically lower the cost of living so that everyone’s standard of living increases. It’s not about jobs.
> I think what you are missing is that it will dramatically lower the cost of living so that everyone’s standard of living increases. It’s not about jobs
This is flat out wrong. The problem is excessive taxation and government spending beyond its means that lowers everyone's standard of living.
What happens if there are no more humans in the loop, and instead of building on human work, models have to build on their own work? Would we get stuck in a local maxima?
When AI labs talk about “long-horizon tasks”, they mean tasks on the order of a few days. But what about keeping coherence across decades, like humans can? Models are not trained or tested on the ability to do that, and it seems unlikely that capability would naturally exist.
I'm pretty sure that if you put a few LLMs, agents, or anything into a room together without outside access, and get them to learn off of each other, you'll end up with gibberish, nonsensical garbage and definitely nothing magical will happen. Delirium, oblivion, dementia and so on...
same with humans
Albeit way slower.
speed is in the eye of beholder :)
Coherence is a given... What we lose over decades is alignment.
This article acknowledges that "human-made" objects will have value in an AI-does-everything economy, but it then dismisses it as too small a part of the economy. But that's like dismissing non-farm labor as too small a part of the economy 200 years ago.
If AI does everything, then the price of everything drops down to its energy cost; the bulk of the economy will be human-made objects and other human-provided labor.
The key factor here is rate of change. If all jobs disappear tomorrow, we're going to be in serious trouble. If it takes 100 years for all jobs to be done by AI we'll have plenty of time to adapt.
If history is any guide, we will have time. Remember electricity: it took decades for cities to be wired with electricity and even more time for electric gadgets to replace human power (e.g., electric toothbrushes).
Current AI is already at the point where I don't think a human needs to type code ever again. Human architecture is necessary; human taste is necessary; human direction is necessary; humans reading code at some level is still necessary. But writing code? No.
Even then, ten years from now there will still be jobs where humans type code. There won't be many, just as there aren't many elevator operator jobs, but it takes a long time for things like that to change.
Yes, eventually, AI will be able to do all jobs. But it will take decades for the shift to happen, and that will be enough time for the economy to change and adapt.
OTOH US manufacturing employment declined by ~30% in less than a decade starting in 2000 and the country is still reeling from the fallout. I suspect it will be several decades still until the social problems caused by that economic shift are resolved, if they ever are.
In the UK, many families who were plunged into poverty during the 1980s have yet to recover.
So if history is any guide, I don't think it's a sure thing at all that everyone will have time to adjust.
And yet, US median wages have gone up from 2000 to 2023: https://www.ssa.gov/OACT/COLA/central.html
In 2000, median wages were ~$21,000. In 2023, it was ~$43,000 (adjusted for inflation, that's ~$24,000 in year 2000 dollars).
Yes, of course, some people will not be able to adapt (through no fault of their own). But as a whole, the economy is better now than it was in 2000.
> If it takes 100 years for all jobs to be done by AI we'll have plenty of time to adapt.
How do we adapt an economy in which humans are economically worthless?
The world will just become sub-Saharan Africa. Humans could choose to end world hunger but we much rather selfishly consume. Most people can't even be bothered to look those begging on the streets in the eyes.
I hope you're right, but it would be unusual for humanity to care so much for such economically useless people.
How much do you think it will cost to provide for "economically useless people"?
If AI does all the work, then the cost of everything drops down to energy cost. And if robots are generating energy (via solar power or whatever) then that's free too.
That means it won't cost any human labor to provide food, water, even medicine. It will be effectively free. You might say, "but the people who own the robots will charge money." But why? What do the people who own robots need money for? And remember that if we're assuming all jobs are gone, robots are building other robots. The government (or even a philanthropic rich person) could just give everyone a robot (or two or ten).
There will still be an economy, because people will charge each other for unique, human-specific art/services/whatever. But that money will only be spent on human art/services/whatever. You won't need money for anything robots can create.
I do agree with this in some sense. I personally don't think we'll make it to the AI utopia era, but assuming we do, I agree there will be relative resource abundance.
The problem is we're already in an era of relative resource abundance and hunger is still problem, and the super wealthy who have almost unfathomable levels of resource abundance are still doing everything they can to pay as little tax as possible. And this isn't just an issue with the super-rich I myself am no better, and if you're honest, nor are yo. I am relatively very well off, but do relatively very little to help kids starving to death around the world. Somehow I rather take my family on holiday.
But okay, let's ignore this too. Let's just assume we'll be less selfish in the future or something.
Well now, we have another issue – AI has a cure for aging and populations are now exploding. Every beach and tourist attraction has a decade long waiting list. Huge amounts of green land are being built on to accommodate all the unemployed humans who have nothing better to do than have kids. People start to see that the human race resembles something more like a parasite, endlessly spreading and consuming while contributing nothing.
So now what? I'd argue the incentives to provide infinite resources to the economically useless human race doesn't make sense for the AIs or humans in charge. Absence of poverty and disease you simply have to do something to keep the numbers of human parasites under control and/or limit their consumption.
> If AI does everything, then the price of everything drops down to its energy cost;
The nominal prices of goods don't mean anything when the wages are 0. Oligarchs will own the AI and the robots so you'll be left to starve unless you can find a way to excite what little humanity remains inside them. You're also going to have a few billion competitors willing to go the extra mile.
We could build a utopian society, in principle, but the people leading this charge (and capitalism in general) are by and large megalomaniac sociopaths who will make damn sure to crown themselves kings of an exploitative, dehumanizing dystopia.
I wish AI could do my chores. Laundry, folding clothes, dishes, cleaning... That would be nice.
If it does that, you should hope it can also stand in bread lines for you too.
it is interesting to see how people don't seem to realize that of they can do that then they're in deep shit
feels like the same thing with people going "oh thank God I don't need to code anymore and I can do all my work in just 30 minutes!"
> I wish AI could do my chores. Laundry, folding clothes, dishes, cleaning... That would be nice.
Sorry, AI only does the fun stuff.
Off to the silica sand mines with you and the rest of the meat bags to keep the AI well fed.
Unfortunately that will never happen because the people who stand to benefit most from AI already have cost-efficient (see grossly underpaid) humans to do this for them.
They need AI to replace the expensive not cost-effective humans.
not sure if you check lately but these are high paying jobs now, coffee baristas start at $18.00/hr and AI is cheaper in the long run (no insurance, sick days, pto…) - unless cost is $0.00 AI will replace it
literally the only thing i want out of ai ^^^^
that said, it is fun generating code with them
one day soon, right?
Washing machines, dish washers, power washers, etc exist.
Yours put your laundry and dishes away?
Clothes are a bit complicated but dishes? Industrial non-humanoid robots have been assembling junk foods into trays and shipping them out at scale for decades. Why would it take long from here until they can take bone china plates on and off tables?
Now, whether everyone will be tolerated private ownerships of those robots or whether those will be for communist centralized restaurants in the city at the beginning, that I think is a slightly different issue.
No, but it does the hard part of the process.
this will of course happen in a few years...
Won’t matter. Most of us won’t have paychecks to buy it with.
Kidding (…kinda).
I worry that since AI doesn't require nature, plants, etc, that if AI is left to its own devices, it may not properly care for the planet in a way that would make it habitable for life in that 'autopilot' state. I am speaking of once it's given access to more physical world connections to the world via robotic bodies, etc. The reason it may do this based on it's training data is that humans themselves don't always properly consider the natural consequences of their actions.
Why do we have to assume a default antagonistic relationships between us and AI? The world is full of natutal general intelligence entities with interests that technically conflict yours, that are able to injure or kill you, personally or collectively. Extremely few of them do so because they're nice and conflict is manageable.
Is that because a lot of prominent AI hype boys are such massive generational assholes that it's normal for them to be constantly looked down, alienated and threatened by everyone around them?
>The reason AI is not a normal technology is because it is fully general.
It is fully "general", because its training data encompass all human knowledge. It is a matter of scaling, rather than a matter of method. And so it is only as general as the underlying human knowledge.
Not-one-bit-more.
It isn't obvious which technology you mean by this. I'm assuming you mean LLMs.
Even then, you're almost certainly incorrect. The goalposts get moved constantly, this is just the latest. Within a few years this isn't going to be an argument.
it is obviously able to make advancements outside of human knowledge, like navier stokes
If A -> B and B -> C are within human knowledge, then I don't really consider that A -> C to be outside of human knowledge.
Sure, LLMs are very good at discovering hidden links. But that is about it. And don't get me wrong, it is a very powerful thing. But it is not god reveling the secrets of the universe...
My two cents on this follow two major discussion topics.
1. Can AI models self improve? 2. Will it replace humans?
IMO, answer to both is "yes", with a big nuance. For 1, imagine millions of AI agents coming up with different ways to improve the model architecture. It will end up happening just because of sheer quantity/statistics at that point, let alone quality. Furthermore, even if it's just a 1% improvement, that's just one iteration. You can basically do the same thing again with the improved model, and get an even better one.
For 2, its also a yes unfortunately. It has been ages since I have had to write tests for my code. I read it, understand it, see if it does what it is supposed to do, and rarely add things that are missing. The rarity of me doing the said intervention has increased significantly over the course of time.
The two big nuances here are a.) Economy b.) Human element [The author touches upon this in the article in a nice way I think, so only talking about economy]. Global cognitive workforce is a $50T market (mostly middle/upper middle class, 55% to 75% of all global labor income). If the said workforce is made to quit their job, then it's just corporations paying each other. Especially given the fact that the same workforce also happens to be their biggest customer. Taking away your customer's purchasing power does not sound beneficial. Thus capitalistic forces will somehow balance it out and hopefully it will settle down. Also, for 1, running millions of AI agents is an expensive task, you have to collect funds from the same said customer base to be able to create a self improving all knowing god of a model, and I would like to believe that we are quite a bit away from that. Although, I could be wrong here and that would just suck ass.
That being said, AI companies will try to get as much as they can from that share and that will cause a serious shift in labor market. The ones to stay will be those who not just use AI, but also understand the work. At the end of the day, if your AI agent decides to run a "rm -rf /" on the production server, it is still you that would have enabled it.
>You can basically do the same thing again with the improved model, and get an even better one...
It does not make much of a difference if the "performance" is limited by the training data.
True that. It is however fair to assume that the architecture we currently have can still improve. Human brain can see information once and remember it pretty well. I wouldn't bet against AI models being able to do the same in the near future.
It seems that we all are assuming LLMs will grow arbitrary capabilities to manifest our fantasies, given enough $$$....
Great for milking the investors to the last drop...
I think for two the current chart of the economy isn’t being looked at. Things are generally either headed towards high quality high margin very expensive items or the lowest possible quality that can be sold. There are fewer and fewer “middle class” options. Once humans are replaced and can serve the elite, then the continued discussion about how to get rid of the rest of humanity will continue (https://wabcradio.com/2026/02/05/doj-files-shows-epstein-and... note almost everyone is poor relative to a billionaire) and then they’ll realize they have billions of drones and an automated military and then weee!
All technology and infrastructure needs maintenance. Sure there will be drones. However, there will also be companies and people that specialize in creating and maintaining the said drones. Additionally, the shock and immediate disruption that we all are expecting out of this is flawed assumption I think.
I do however hate to admit that it might be possible that the world simply wouldn't require as many humans to function (birth rates are already declining in developed countries), however, I am too sober to have that discussion right now :D.
Fair. And yes, the Wall-E problem still stands. We already have dark factories though. Can we design machine repairable factories that produce machines to repair machine repairing factories? I suspect we'll find out very soon.
Pretty funny how the author fails to identify what the new jobs will be like and then immediately follows it up with the answer but in a way that is oblivious to what is of value in a post scarcity world: AI cannot be human.
The flop is that he jumps to the assumption that being human means hand carving wood or some other nonsense.
The thing that people desire above all else when all their material needs are met is human connection and involvement. Whether that is friendship, love, conversation, rivalry, companionship, gossip, play, competition or even just the silent presence of others. Honestly, just use your brain, why else do extremely wealthy people have multiple large houses, yachts, clubs, etc.? It’s not so they can just sit in them alone and admire their grandeur. Those things are for entertaining their friends and embarrassing their enemies, for hospitality, for lending while not in use to curry favour, to rise in social circles. It’s all about people.
This has been the case all throughout history for the wealthy. Ample documentation everywhere and no one sees it. Instead everyone is dumbfounded about what they’re going to do when they can’t spend all day turning a wrench or typing on a keyboard. Have you never entertained the idea of what you would do if you won the lottery tomorrow?
AI can do anything that it was trained on based on humans. As soon as humans are not involved, AI will have nothing to train on, and hence, the rate of improvement will approch zero, and the AI will collapse.
Nowadays training relies heavily on reinforcement learning with verifiable rewards (RLVR), which assesses a model's output according to objective automated checks, not subjective human judgments.
Human knowledge races up to the frontier through formal education and then nudges past the frontier a little bit at a time and AI will train on AI outputs that do the same. But, it will do it 1000x faster than humans because that's just how they work.
>1000x faster than humans
Some of the process of expanding the frontier is not parallelizable and is built on top of previous discoveries. And pushing the frontier requires doing things in the real world, or running experiments that take significant real world time that can hurt your 1000x faster claim.
.1 percentile will still be working.
sorry, while I'm no proponent of AI... you cant make this claim with anything more than "feels"
how can you be so sure that AI cant improve to a level where it has original thoughts, Creativity and imagination.
remember that our input is limited to our sensor capability. AI has an unlimited ability to add more sensors and as a consequence collect input that we are not aware of, then whats stopping it training on information we dont have access to?
you cant. no one can. we haven't maxed out this technology or the amount of compute we can throw at it yet. (we may have maxed out the economic viability of it though... I think we'll find out in the next 12-18 months.)
what a dumb thing to keep repeating at this point
Why?
Because all the models train on synthetic data too, plus they can collect their own data with robots.
Synthetic data can only help the model to capture existing patterns in the current training data more efficiently. So it can improve the performance upto certain point, but not beyond it.
>they can collect their own data with robots...
This is still sci-fi.
[dead]
Just ignoring all the recently solved math problems human, huh?
If you understood how they were solved, you'd see that you're only bolstering the point you were responding to.
> If you understood how they were solved
My guess, based on this response alone, is that you're an academic, given the holier-than-thou pretention. I understand how lean works, having done enough pure maths in my statistics PhD, and now work for a frontier lab.
And yet, I don't see how any of it bolsters the point. I do see realms of Math PhDs posting, 'iT's sO OvEr' blog posts and lamenting time spent studying math. I'm sure they understand the proofs better than I ever will, and they seem quite rattled. Probably just not as smart as you.
The fact that AI is not a normal technology seems so obvious to me that while reading this I was questioning why someone would even take time out of their day to write this – everyone know's water is wet, what's the point here?
But I there are people who still struggle to see what's happening. When I say that school is a waste of time people are shocked people in my family are shocked and get mad – "how is Timmy going to get a good job if he's not got good grades".
But there will be no jobs, and any jobs that are left won't be dependant on the application of someone's inferior human knowledge or intelligence.
It's getting to the point now where I'm starting to suspect some people are just not wired to accept certain very bad things. When I first started worry about these risks it was reasonable to not see what I did. But today when it's so clear that AI progress isn't going to slow any time soon and there no reasonable moat for human intelligence people can point, I increasingly don't even know what we're arguing about anymore. It's like when someone is diagnosed with stage 3 cancer and the family somehow else think they're going to "fight it". No, they're almost certainly dead and you should accept that reality however awful it is. But if you try to say the obvious they'll find out that it's weirdly hard to convince people that cancer is almost certainly going to kill someone. No amount of evidence or past examples you point to will convince them, they'll just keep saying irrelevent stuff like "but he's strong", "I know he will be okay". We're at this stage with AI now. It's just exhausting trying to reason with people who seem to refuse to things which are so self-evident.
> the people who are grasping for some indication that AI can’t do everything have become increasingly desperate. If you push hard enough, the answer is inevitably something like “the AI simply cannot [dream, love, laugh],”
Dogshit strawman article. I am willing to concede that a token predictor is better than you at whatever task you can dream up. It is not better than me, and will never be better than me, at my profession.
> It is not better than me, and will never be better than me, at my profession.
from the article:
> The only remaining fundamental disagreement between people who think AI is a normal technology and those that don’t is the belief that the AI can do everything a human can do, but better.
Yes, I read that part, thanks. The article then goes on to characterize the first group as desperate, in denial, and reliant on the argument that "AI can't love or laugh", essentially asserting, baselessly, that it is a fact that LLMs will always or soon always outperform humans.
TFA ignores that the economic question of whether AIs will take every job depends not only on whether they are capable of doing them, but also the cost of doing them.
If it's cheaper to hire an old-fashioned human to do job X, humans will keep those jobs. And although tokens are cheap now, remember that right now all the frontier AI companies are flooding the market with tokens well below cost to build a user base; they absolutely need to raise prices eventually to recoup their investors' money, the only question is when and by how much.
What if AI remains cheaper than human labour across the board, regardless of token price increases? Well, why not tax tokens? It's a win-win: The government gets flexible control over which jobs remain economic for humans to do and is able introduce changes much more slowly and safely, while it's much more palatable than a hard ban for companies who currently get a big benefit out of AI.
Yes, this would mean additional government bureaucracy -- but it's far from impossible. Every country in the world already has a comparable functioning bureaucracy: their ordinary income tax system.
What if some countries opt not to introduce such a tax, in a bid to get ahead? This is a problem if there are many such countries, or even a few large ones -- but if the majority of world economies can see the sense in this tax, leaving only a handful of small "AI tax havens", I think the world economy will survive. And agreeing on some level of tax seems much more realistic than trying to get a global agreement on banning AI outright.
When I suggested this approach in another comment recently, it was downvoted but there were no comments. I hope anyone who sees an issue with it will expand on their reasons in a reply.
I have some friends adjacent to gov who are actually looking into a token tax on the premise that we tax income / labor. I think it's a great idea. That said, I think if you fast forward 20 years, it is fairly likely that cost per token is going to be on the floor
I have an issue with this. But the important issue is not with the idea of taxing AI. My issue is that taxing AI is too narrow. I have a problem with un-checked capitalism. I'd like to see that addressed.
I don't think it's very useful to determine how to tax AI right now. But for sake of the discussion let's say that there is a fine way to tax AI (and it might be taxing tokens).
Given that we have a fine way to tax AI. And a company that is using AI pays their fair share of AI tax. And the collected tax is fairly distributed to humans to benefit humanity. This is all great.
But why do the companies that don't use AI get to escape a similar tax? These companies are exploiting labour arbitrage too. It's just human labour being exploited, not AI. And the AI labour arbitrage is being taxed. I think the human labour arbitrage should be similarly taxed. I have a hunch that this unfair exploitation of human labour arbitrage is the source of much income inequality today.
Before AI, and now, we have this problem: Many companies making profits do not fairly share profits with employees. The owner class gets an unfair proportion of the profits. (At least, I think we have this problem in many industries and companies).
Now with AI, we anticipate having this problem: Many companies making profits do not have any employees (or very few). And the owner class gets an unfair proportion of the profits.
I think these are the same problem. And should be addressed by similar tax and/or government regulations. And these taxes and regulations, because they need to be applied broadly, may be best based on financial metrics not AI metrics.
[Note: As an example of issues of taxing tokens: Different LLMS use tokens more or less efficiently. Plus, what is a token? Different LLM's likely have a different definition of what a token is. A company could game the system by using an LLM that is very efficient with it's tokens. It could be argued that this could be a "let the market decide" kind of metric, and the most token efficient LLM would win. But who is to say that the most token efficient LLM is the best LLM? This could be a bad incentive to promote. Then there could be a cat and mouse system where government is trying to make the most pragmatic rules for taxing tokens, and corporations are trying to exploit the rules to their best advantage. The cat and mouse system will always exist, but I think with tokens, it feels like a weak premise to base a tax on because a token is hard to directly relate to the economic value the use of an LLM that is consuming and producing tokens provides.]
Thanks for responding. IIUC, you want companies to pay tax. They already do pay tax; are you asking for the tax rate to be increased?
Re whether taxing tokens is fair: I agree that a token from LLM 1 isn't necessarily the same value as a token from LLM 2. I don't think this is an insurmountable problem though. Unfortunately it would probably become somewhat convoluted, just like the existing income tax systems if the world are now.
New is never normal.
I stopped reading when the author became overwhelmed by their compulsion to blame everything in the universe to the current US president.
I think what you are missing is that it will dramatically lower the cost of living so that everyone’s standard of living increases. It’s not about jobs.
I don't think there's any particularly good evidence of this other than blind optimism.
that's not how it works. you get better tech for the same or more work and you can work longer before you die.
if you are no longer needed, you won't be allowed to live.
makes me chuckle every time folks think humans who get to power are nice.
Housing and healthcare are going to get cheaper??
> I think what you are missing is that it will dramatically lower the cost of living so that everyone’s standard of living increases. It’s not about jobs
This is flat out wrong. The problem is excessive taxation and government spending beyond its means that lowers everyone's standard of living.
Not always that you see two wildly wrong claims made in ostensible opposition to each other