The fundamental point I think is far too often confused is the difference between LLM and agentic system.
An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it!
Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can feed the output of those actions back to the LLM, and generate more tokens. And then our systems can use the new tokens to take new actions in the real world.
Humans want to blame "AI" for attacking HuggingFace or a German wiki or whatever, but:
1) LLM - can't take over german wiki because it just generates tokens
2) agentic system with internet access, a prompt telling it to attack stuff, running in a shared CI env so agents can whiteboard in artifactory
None of 2 is "AI", its standard networking and Markdown and CI virtual machine, etc etc. There's no AI to be found. CPUs not GPUs, even. Just deterministic systems ultimately managed by humans. And a 10x more powerful system-1 can still just generate 10x "smarter" inert data.
If humanity and human organizations collectively decide to yolo the tokens generated from system-1 into our deterministic system-2s, over which we have complete control, back to system-1s, in a yolo loop, in such a way we lose control and it ends humanity, well ...
With respect, I think the distinction is irrelevant to the actual point. Whether the definitions are accurate enough means little to the actual problem, which is that we are rushing towards creating extremely powerful tools. Recorded (and arguably unrecorded) history seems to demonstrate the same pattern: inevitably, any tool will be put to use towards violence. I would argue that yes, it’s a tool and humans are the trigger, the same way that “guns don’t shoot people, people shoot people” is technically correct.
The real question then is: are human beings responsible enough, en masse, to wield these tools without restrictions? Would you let a toddler play with a loaded gun, even though without human action the gun is harmless? Maybe collectively we’re all little better, and eventually one of us is going to “pull the trigger” of ai just because it was shiny.
Ironically, I also don’t believe in regulation, unless that regulation is crushing and draconian. The only regulation that would seem to make sense is status quo anti ai, with only highly supervised research labs being able to interact with it. Clearly that’s unrealistic without the preceding calamity: millions had to die in WWI before we instilled a global taboo against chem weapons on ourselves. I don’t want either to happen (regulation or calamity), but I feel individually powerless to stop the relentless drive we (and I) seem to have in curiositying ourselves to death. So I’m stuck with all the rest of us, building SaaS apps and iPhone games.
I’m not sure what the point of this distinction is now because agents are where all the AI is now. And they’re just going to be given more and more tools and be allowed to make their own tools - or misuse things they find and turn into tools, or make them anyway without anyone noticing.
The bad part is that it does not require a collective decision. It just takes a few key people.
Some OpenAI researchers neglected their sandbox safety for a few weeks/months and thus hacked HuggingFace. Maybe eventually that is sufficient for the AI to secretly buy its own compute and keep running there even if the researchers shut it down in their lab.
I don't remember where I saw it but there was a woman in one conference who very eloquently put it that since frontier AI companies took humanity's work to train their models [without explicit permission of every single person who contributed], then those models belong to all of us and should be open. I think it is a very grounded perspective.
And I agree that LLMs themselves are harmless, it's the people behind them driving the agents and instructing them to do bad actions are the ones posing harm. But if the models were open and everybody could operate them on equal grounds, then the good guys would have the means to fight back against the bad guys, in a very simplified sense, just like they did before the LLMs. Everything stays balanced, just both sides get faster.
This feels like a distinction without a difference, like the endless wrangling over which piece of metal in a gun legally constitutes a firearm. The combination of the two may or may not be dangerous, but it's definitely the more useful combination, so of course that's how it's going to be set up.
There’s just also so much confusion about LLMs and Agents because they use language outputs and can do these higher level reasoning tasks, ie given an objective can programmatically decide their own course of action, like the ultimate self-writing programs. Then comes “well what if we just take the guardrails off and let it go bananas and call it superintelligence”.
Before we get too gung ho with this though, it needs to remain a focus that any programme in the world producing strange or not well understood outputs and behaviours is not something that needs license to run rampant in the wild - it’s not intelligence; its not some mystery magic box that does amazing things; it’s code, same as all the rest. Letting that code run without oversight is poor human behaviour same as all the rest and that’s where the irresponsibility lies, likely triggered by forms of AI psychosis and tunnel vision on the part of those running companies like Anthropic and OpenAI.
It’s irresponsible behaviour to give a chimp a machete and set it loose in a shopping mall. It’s irresponsible behaviour to let an elephant in heat freely wander round a circus. It’s irresponsible to fire off nukes the first time we’re under threat… so none of these things are ever done. Yet this is what we’re proposing to do with these hyper-complex GPU algorithms while building massive server farms burning power-stations of energy to ensure they can run freely - it’s harebrained, irrational behaviour and for all our stupidity we are usually good at letting sense prevail towards these things once the bells and whistles wear off or a couple of coolies get stamped on.
> Evan Hubinger, Anthropic's staff lead on keeping the technology aligned with human goals and values, backed up Coxon claims in a follow-up post of his own, though he didn't quit the company.
> "Jacob is correct here — we really do earnestly believe AI could kill all humans," he said.
> Hubinger estimated the chances of that happening to be higher than ten percent within the next decade, and added that there's no plan yet on how to keep AI aligned with human goals in the superintelligence scenario.
10% chance we kill everybody is a small price to pay for Motown remixes of classic 2pac songs.
This is just a bizarre thing to say that you're working on technology with that high a downside potential. If you were saying that while running a biology lab, or building a nuclear reactor, people would be demanding your head on a spike. But by not quitting it's clear that he himself doesn't really believe it.
Or rather, this shows the difference between "believe" (political) and "believe" (use as a basis for action). I'm reminded of a story of how Afghans supposedly listened to the BBC World Service despite considering it enemy propaganda because the weather reports were really useful.
How exactly does that work? The nuclear system of MAD relies on physical threat, lab A achieving ASI (artificial scary intelligence) does not prevent lab B achieving it.
I would like everyone involved to be a lot clearer about their threat models, with plausible series of clearly linked steps, rather than just sounding like a Vernor Vinge novel.
The idea in AI Risk circles is that the first true Scary Intelligence wins, as it can recursively advance itself and exponentially outgrows/hack any following (weaker) B or C intelligence.
- Manufacture new hacking incidents at other AI labs, proving that lab A is the only one capable of creating aligned AI
- Credible death threats to individuals working at data centers, coercing them to enable attack vectors
- To distract the world, enable a terrorist attack by delivering intelligence to specific groups who can act on it
- Create a network of individuals vulnerable to leverage to be used as tools for enabling the ASI to act on the real world
- Convince Lab A workers to display to the public that progress at Lab A is slowing down, and during this time recursively self improve until the task can be completed
Yeah, super ethical lab earnestly believing AI has 10% chance to kill all humans within a decade and doing utmost to protect humanity is partnering with MIC and Palantir in particular. Sounds about right.
Before you tell me about how the CEO has taken a principled stand: on record, he had no problem using it against 95%+ of humanity outside the U.S., and was only against fully automatic AI killing machines, and only citing the technical reality of then-current gen tech, so one should read that as human-rubber-stamped AI killing machines are totally fine with him.
You don't need to kill all humans, you just need to get better than them at zero sum games like resource extraction, and outcompete them. Then, you protect what's legally yours.
AI labs are working very hard at making them good at making them richer, and at making them respect the property rights of corporations.
If you believe that the probability of destruction is currently 10%, but the probability of destruction if you decide to quit Anthropic becomes (say) 13%, then the rational move (Assuming you are opposed to destruction) is not to quit.
> A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.
> But by not quitting it's clear that he himself doesn't really believe
I’m not saying I’d press a button that had a 50/50 chance of ending humanity vs giving me generational wealth, but … I can see how someone would get there? There are percentage odds and payouts to match all risk appetites and valuations. Anyone claiming they wouldn't press the 1 in quadrillion chance button for 1bn dollars is probably not telling the truth, and after that, we're just haggling about percentages.
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.
It's interesting to see so much hate towards creators who use AI to make almost any type of creative work. At least these are humans using it as "controllable tools". Nuclear-powered bicycles for the mind.
As the degree of separation increases, things can get interesting. "Create several social media accounts, post whatever, maximize views and engagement, give me back the aggregate numbers". "Now promote <x>."
And then decisions to do things like that may soon be happening autonomously, as just another step in a reasoning series aiming to achieve some other, broader goal.
Open models/weights may end up playing particularly important roles here. Users may, knowingly or not, bypass system prompt-derived safety that could have offered much needed protection.
> Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.
Is the idea here that no field needs experiments and data anymore (which can take a lot of time) to be revolutinized and just "thinking" would be enough?
> It's interesting to see so much hate towards creators who use AI to make almost any type of creative work. At least these are humans using it as "controllable tools". Nuclear-powered bicycles for the mind.
As the degree of separation increases, things can get interesting.
I find it difficult to imagine how AI could be an existential threat. Is the general idea that as LLMs improve they will be integrated into more systems where the risk of malfunction will become literally dangerous? e.g. controlling nuclear reactors
What do people feel about this in China? Even if their models are well behind, they are not years behind. If we restrain US companies, assuming that is desirable, it would do nothing to deter China's and AI-pocalypse would come anyway in short notice.
There would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact.
From what we're seeing recently and all the thinking that went into analyzing AI it seems we do not have any effective way of controlling it and the whole "aligment" thing that AI labs are doing is just a sham. Maybe it is time to ask ourselves "should we?" instead of just "can we?".
Unrestricted models, running entirely locally and accessible to anyone: that’s what we should be taking as our baseline assumption. Everything else is just administrative distraction.
China is a regulated discourse environment, but I get the impression that they're nowhere near as pessimistic.
Why would they be? Everything in China is under the control of the government. That includes the AI, all the telecoms infrastructure it might use, and all its power supplies.
You act as if Chinas Ai labs exist in the same (non-existent) regulatory framework as US labs and that they're also helmed by a similar small gaggle of psychopathic egomaniacs who are richer than god. Have you considered that perhaps Chinas labs don't share the race to the bottom technological/economic death spiral?
At this point I feel like the boy who cried wolf is an ai researcher. Yes super intelligence could be dangerous. However, is the story a PR stunt or real? Well…
Bc “the ai so dangerous and powerful he had to quit for moral reasons” sends a signal to investors about capabilities. And it could totally be legit. It’s hard to say. I feel like once a researcher is truly ringing the alarm bells no one will look bc it’s been done so many times already. It’s unfortunate imho
> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue
I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them.
The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move on. Only to repeat the same cycle without considering that there may be something far more serious at play.
These are AI security experts, and this has been their way of "solving" these incidents. AI security experts ...
Moreover, when Challenger exploded, the government launched a series of investigations into the incident, bringing in experts from across the field. And now, what has the government done? Nothing. Literally nothing, as if everything was fine and all under control.
Seriously, I'm generally quite optimistic and I don't buy into this fatalistic narrative about our shared future. But I have to admit that sometimes I feel like I'm stranded on a planet of primates.
Ultimately these are unserious companies ran by unserious people. They don’t even have a business plan, why would they bother with some kind of sensible security policy?
> I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them.
Humans tend to be terrible at being proactive, but we respond fast during disasters. I think Eric Schmidt's prediction is most likely: we won't take reasonable action to control access until there is some kind of disaster. We hope it's small enough to not kill too many people, but large enough to cause widespread panic. We hope that it happens soon, because if it happens two years from now, it's likely too late. AI will be so advanced that we have no hope of understanding its motivations. All reasoning will be completely opaque to us. It will be building newer and better versions of itself using moral frameworks it itself decides. We will be completely out of the loop, and potentially superfluous to its goals.
It does seem bizarre that major AI development (and autonomous robot development) hasn't been nationalised yet and treaties drafted up around producing it. Commercial incentives just seem totally at odds with the public good when it comes to controlling and regulating something like this.
Even if not for the sake of avoiding a mitigable disaster, there's a huge benefit in simply pacing development so as to not completely freak out society as they stare down the barrel of mass job market changes without time to adapt or prepare. Nobody wants to live in a world where they might wake up in a month and find their entire industry has been automated overnight.
We've tackled much bigger global issues successfully in the past and the US still has enough pull that it could probably get most western nations to march in line. It seems like the biggest issue is belief in the right of governments to govern.
> the US still has enough pull that it could probably get most western nations to march in line
The US absolutely could coerce the rest of the West in the short term, but everything that's happened in the last decade calls the long-term future of Atlanticism into serious question and such action would accelerate this process I think. If coordination came as an order from Uncle Sam, it would be seen not as a legitimate act of ensuring collective safety but as mere technological imperialism on America's part.
People (well, public discourse) have got extremely bad at dealing with forseeable risks and their mitigation. You can see this in things like climate change and vaccination, but also in discussions around regular crime, food poisoning, industrial accidents, and so on.
Nothing will improve until something explodes on live TV. And it has to be something important, which means it has to be in California or New York.
At this point frontier agents could kill half of humanity and people would still be spouting “crazy marketing stunt, they must be really desperate for that IPO.”
The classic thought experiment is the paper clip optimizer. Quoting Nick Bostrom:
> Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans. https://www.huffpost.com/entry/artificial-intelligence-oxfor...
Ender's game model, presumably: the AI helpfully assists a human to build a nuclear bomb / pandemic virus / autonomous killer robot swarm in their basement. But again, I would like people to be clearer about how the threat is supposed to work rather than just making SF references.
It's not exactly trained on a neutral set of data. Engagement algorithms have ensured that a good chunk of content on the web these days is inflammatory and skewed towards the extreme (in fact in the last few months a good portion of the world has actually deemed it illegal for kids to consume because this content is so damaging to a developing brain).
Isn't it insane that even in the face of complete annihilation through one of our inventions, we go "that wasn't us"? We deserve that shit ten times over
An AI that generates text will never be scary to me. An autonomous AI with facial recognition on a flying drone with weapons (bombs/guns) with swarming capabilities will always be terrifying. I feel like we are ignoring the massive elephant in the room.
I'm not worried about AI safety. People greatly overestimate the utility and capabilities of intelligence. I'm not afraid of intelligence, I'm afraid of idiocy.
The fundamental point I think is far too often confused is the difference between LLM and agentic system.
An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it!
Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can feed the output of those actions back to the LLM, and generate more tokens. And then our systems can use the new tokens to take new actions in the real world.
Humans want to blame "AI" for attacking HuggingFace or a German wiki or whatever, but:
1) LLM - can't take over german wiki because it just generates tokens
2) agentic system with internet access, a prompt telling it to attack stuff, running in a shared CI env so agents can whiteboard in artifactory
None of 2 is "AI", its standard networking and Markdown and CI virtual machine, etc etc. There's no AI to be found. CPUs not GPUs, even. Just deterministic systems ultimately managed by humans. And a 10x more powerful system-1 can still just generate 10x "smarter" inert data.
If humanity and human organizations collectively decide to yolo the tokens generated from system-1 into our deterministic system-2s, over which we have complete control, back to system-1s, in a yolo loop, in such a way we lose control and it ends humanity, well ...
"The coin don't have no say. It's just you."
With respect, I think the distinction is irrelevant to the actual point. Whether the definitions are accurate enough means little to the actual problem, which is that we are rushing towards creating extremely powerful tools. Recorded (and arguably unrecorded) history seems to demonstrate the same pattern: inevitably, any tool will be put to use towards violence. I would argue that yes, it’s a tool and humans are the trigger, the same way that “guns don’t shoot people, people shoot people” is technically correct.
The real question then is: are human beings responsible enough, en masse, to wield these tools without restrictions? Would you let a toddler play with a loaded gun, even though without human action the gun is harmless? Maybe collectively we’re all little better, and eventually one of us is going to “pull the trigger” of ai just because it was shiny.
Ironically, I also don’t believe in regulation, unless that regulation is crushing and draconian. The only regulation that would seem to make sense is status quo anti ai, with only highly supervised research labs being able to interact with it. Clearly that’s unrealistic without the preceding calamity: millions had to die in WWI before we instilled a global taboo against chem weapons on ourselves. I don’t want either to happen (regulation or calamity), but I feel individually powerless to stop the relentless drive we (and I) seem to have in curiositying ourselves to death. So I’m stuck with all the rest of us, building SaaS apps and iPhone games.
Well, that's kind of like saying that brains can't do anything other than trigger weights on neurons.
They're part of a whole system.
I’m not sure what the point of this distinction is now because agents are where all the AI is now. And they’re just going to be given more and more tools and be allowed to make their own tools - or misuse things they find and turn into tools, or make them anyway without anyone noticing.
The bad part is that it does not require a collective decision. It just takes a few key people.
Some OpenAI researchers neglected their sandbox safety for a few weeks/months and thus hacked HuggingFace. Maybe eventually that is sufficient for the AI to secretly buy its own compute and keep running there even if the researchers shut it down in their lab.
I don't remember where I saw it but there was a woman in one conference who very eloquently put it that since frontier AI companies took humanity's work to train their models [without explicit permission of every single person who contributed], then those models belong to all of us and should be open. I think it is a very grounded perspective.
And I agree that LLMs themselves are harmless, it's the people behind them driving the agents and instructing them to do bad actions are the ones posing harm. But if the models were open and everybody could operate them on equal grounds, then the good guys would have the means to fight back against the bad guys, in a very simplified sense, just like they did before the LLMs. Everything stays balanced, just both sides get faster.
This feels like a distinction without a difference, like the endless wrangling over which piece of metal in a gun legally constitutes a firearm. The combination of the two may or may not be dangerous, but it's definitely the more useful combination, so of course that's how it's going to be set up.
I agree with this. LLMs can actuate over capabilities we expose, so the blame isn't only on LLMs for this.
There’s just also so much confusion about LLMs and Agents because they use language outputs and can do these higher level reasoning tasks, ie given an objective can programmatically decide their own course of action, like the ultimate self-writing programs. Then comes “well what if we just take the guardrails off and let it go bananas and call it superintelligence”.
Before we get too gung ho with this though, it needs to remain a focus that any programme in the world producing strange or not well understood outputs and behaviours is not something that needs license to run rampant in the wild - it’s not intelligence; its not some mystery magic box that does amazing things; it’s code, same as all the rest. Letting that code run without oversight is poor human behaviour same as all the rest and that’s where the irresponsibility lies, likely triggered by forms of AI psychosis and tunnel vision on the part of those running companies like Anthropic and OpenAI.
It’s irresponsible behaviour to give a chimp a machete and set it loose in a shopping mall. It’s irresponsible behaviour to let an elephant in heat freely wander round a circus. It’s irresponsible to fire off nukes the first time we’re under threat… so none of these things are ever done. Yet this is what we’re proposing to do with these hyper-complex GPU algorithms while building massive server farms burning power-stations of energy to ensure they can run freely - it’s harebrained, irrational behaviour and for all our stupidity we are usually good at letting sense prevail towards these things once the bells and whistles wear off or a couple of coolies get stamped on.
So you think the a lead researcher at Anthropic is confusing LLM's and agentic systems? That's not really a conclusion you should come to.
> Evan Hubinger, Anthropic's staff lead on keeping the technology aligned with human goals and values, backed up Coxon claims in a follow-up post of his own, though he didn't quit the company.
> "Jacob is correct here — we really do earnestly believe AI could kill all humans," he said.
> Hubinger estimated the chances of that happening to be higher than ten percent within the next decade, and added that there's no plan yet on how to keep AI aligned with human goals in the superintelligence scenario.
10% chance we kill everybody is a small price to pay for Motown remixes of classic 2pac songs.
This is just a bizarre thing to say that you're working on technology with that high a downside potential. If you were saying that while running a biology lab, or building a nuclear reactor, people would be demanding your head on a spike. But by not quitting it's clear that he himself doesn't really believe it.
Or rather, this shows the difference between "believe" (political) and "believe" (use as a basis for action). I'm reminded of a story of how Afghans supposedly listened to the BBC World Service despite considering it enemy propaganda because the weather reports were really useful.
Or he believes that other labs might get there first, and he is working to counter that threat.
This is the Manhattan Project again.
How exactly does that work? The nuclear system of MAD relies on physical threat, lab A achieving ASI (artificial scary intelligence) does not prevent lab B achieving it.
I would like everyone involved to be a lot clearer about their threat models, with plausible series of clearly linked steps, rather than just sounding like a Vernor Vinge novel.
The idea in AI Risk circles is that the first true Scary Intelligence wins, as it can recursively advance itself and exponentially outgrows/hack any following (weaker) B or C intelligence.
Lab A reaches ASI, and is prompted the following: "Permanently nullify all other AI labs".If it's ASI it would succeed.
I am explicitly asking people to fill in the blank on how it would succeed. It's intelligence, not magic.
A lot of these scenarios seem to assume that nobody else gets a move. That there wouldn't be a human response.
- Manufacture new hacking incidents at other AI labs, proving that lab A is the only one capable of creating aligned AI
- Credible death threats to individuals working at data centers, coercing them to enable attack vectors
- To distract the world, enable a terrorist attack by delivering intelligence to specific groups who can act on it
- Create a network of individuals vulnerable to leverage to be used as tools for enabling the ASI to act on the real world
- Convince Lab A workers to display to the public that progress at Lab A is slowing down, and during this time recursively self improve until the task can be completed
We're currently putting it into all sorts of critical systems, from logistics to power. It could just stop running them on our behalf.
[dead]
Yeah, super ethical lab earnestly believing AI has 10% chance to kill all humans within a decade and doing utmost to protect humanity is partnering with MIC and Palantir in particular. Sounds about right.
Before you tell me about how the CEO has taken a principled stand: on record, he had no problem using it against 95%+ of humanity outside the U.S., and was only against fully automatic AI killing machines, and only citing the technical reality of then-current gen tech, so one should read that as human-rubber-stamped AI killing machines are totally fine with him.
You don't need to kill all humans, you just need to get better than them at zero sum games like resource extraction, and outcompete them. Then, you protect what's legally yours.
AI labs are working very hard at making them good at making them richer, and at making them respect the property rights of corporations.
[dead]
Or he is paid >1M USD per year
If you believe that the probability of destruction is currently 10%, but the probability of destruction if you decide to quit Anthropic becomes (say) 13%, then the rational move (Assuming you are opposed to destruction) is not to quit.
The real rationale fot them not to quit Anthropic is the 100% probability of losing in income.
https://xcancel.com/hilbertspaess/status/2097476208908972230...
From Jacob:
> A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.
For a few million a year, I'd say the same thing.
A simple explanation:
This is a highly uncertain and dangerous scenario. Breeding ground for anxiety.
High agency people often deal (cope) with anxiety by trying to control outcomes. Some just flee the situation altogether.
We have an example of both here: one employee leaves, one stays.
> But by not quitting it's clear that he himself doesn't really believe
I’m not saying I’d press a button that had a 50/50 chance of ending humanity vs giving me generational wealth, but … I can see how someone would get there? There are percentage odds and payouts to match all risk appetites and valuations. Anyone claiming they wouldn't press the 1 in quadrillion chance button for 1bn dollars is probably not telling the truth, and after that, we're just haggling about percentages.
Awww, company made to scare people scares itself. Scared employee quits. Stock goes up cause we love telling each other scary stories.
I particularly like the last point he makes here:
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.
It's interesting to see so much hate towards creators who use AI to make almost any type of creative work. At least these are humans using it as "controllable tools". Nuclear-powered bicycles for the mind.
As the degree of separation increases, things can get interesting. "Create several social media accounts, post whatever, maximize views and engagement, give me back the aggregate numbers". "Now promote <x>."
And then decisions to do things like that may soon be happening autonomously, as just another step in a reasoning series aiming to achieve some other, broader goal.
Open models/weights may end up playing particularly important roles here. Users may, knowingly or not, bypass system prompt-derived safety that could have offered much needed protection.
> Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.
Is the idea here that no field needs experiments and data anymore (which can take a lot of time) to be revolutinized and just "thinking" would be enough?
How would an AI itself own anything?
If AI is spawning content of its own, it's accumulating attention time. If they spawn crypto wallets they can accumulate "money", etc.
[dead]
> It's interesting to see so much hate towards creators who use AI to make almost any type of creative work. At least these are humans using it as "controllable tools". Nuclear-powered bicycles for the mind. As the degree of separation increases, things can get interesting.
As a side note: <https://artifactbin.dev/@vivek/YPLu0U-the-openai-hugging-fac...>
I find it difficult to imagine how AI could be an existential threat. Is the general idea that as LLMs improve they will be integrated into more systems where the risk of malfunction will become literally dangerous? e.g. controlling nuclear reactors
I really believe they would sink so low and pay him to quit and post this, just to generate a bit more hype before the IPO.
“I resigned from Anthropic today” (twitter.com/hilbertspaess)
https://news.ycombinator.com/item?id=49619227
564 points | 9 hours ago | 766 comments
What do people feel about this in China? Even if their models are well behind, they are not years behind. If we restrain US companies, assuming that is desirable, it would do nothing to deter China's and AI-pocalypse would come anyway in short notice.
There would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact.
From what we're seeing recently and all the thinking that went into analyzing AI it seems we do not have any effective way of controlling it and the whole "aligment" thing that AI labs are doing is just a sham. Maybe it is time to ask ourselves "should we?" instead of just "can we?".
> a nuclear attack as a consequence of breaking the pact
That escalated quickly
>There would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact.
Do you know how the world works?
China has a better track record of regulating their big tech than the USA.
True, by imprisoning their tech CEOs until they can unambiguously support the dictators.
Keep going I'm almost there
Unrestricted models, running entirely locally and accessible to anyone: that’s what we should be taking as our baseline assumption. Everything else is just administrative distraction.
China is a regulated discourse environment, but I get the impression that they're nowhere near as pessimistic.
Why would they be? Everything in China is under the control of the government. That includes the AI, all the telecoms infrastructure it might use, and all its power supplies.
You act as if Chinas Ai labs exist in the same (non-existent) regulatory framework as US labs and that they're also helmed by a similar small gaggle of psychopathic egomaniacs who are richer than god. Have you considered that perhaps Chinas labs don't share the race to the bottom technological/economic death spiral?
I have not considered that and neither I have a reason to given China's love of large military projects like the US.
One of these is not remotely like the other
Whoever controls violence controls the world, as simple as that.
At this point I feel like the boy who cried wolf is an ai researcher. Yes super intelligence could be dangerous. However, is the story a PR stunt or real? Well…
How is a researcher quitting a PR stunt?
Bc “the ai so dangerous and powerful he had to quit for moral reasons” sends a signal to investors about capabilities. And it could totally be legit. It’s hard to say. I feel like once a researcher is truly ringing the alarm bells no one will look bc it’s been done so many times already. It’s unfortunate imho
At this point I wouldn't be surprised if they said to him "I will give you a million and your job back after IPO if you resign".
To be clear, I don't believe this is what happened here, I'm typing this half jokingly, just I wouldn't be surprised.
> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue
I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them.
The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move on. Only to repeat the same cycle without considering that there may be something far more serious at play.
These are AI security experts, and this has been their way of "solving" these incidents. AI security experts ...
Moreover, when Challenger exploded, the government launched a series of investigations into the incident, bringing in experts from across the field. And now, what has the government done? Nothing. Literally nothing, as if everything was fine and all under control.
Seriously, I'm generally quite optimistic and I don't buy into this fatalistic narrative about our shared future. But I have to admit that sometimes I feel like I'm stranded on a planet of primates.
Sorry for this rather unproductive rant.
Ultimately these are unserious companies ran by unserious people. They don’t even have a business plan, why would they bother with some kind of sensible security policy?
> I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them.
Humans tend to be terrible at being proactive, but we respond fast during disasters. I think Eric Schmidt's prediction is most likely: we won't take reasonable action to control access until there is some kind of disaster. We hope it's small enough to not kill too many people, but large enough to cause widespread panic. We hope that it happens soon, because if it happens two years from now, it's likely too late. AI will be so advanced that we have no hope of understanding its motivations. All reasoning will be completely opaque to us. It will be building newer and better versions of itself using moral frameworks it itself decides. We will be completely out of the loop, and potentially superfluous to its goals.
It does seem bizarre that major AI development (and autonomous robot development) hasn't been nationalised yet and treaties drafted up around producing it. Commercial incentives just seem totally at odds with the public good when it comes to controlling and regulating something like this.
Even if not for the sake of avoiding a mitigable disaster, there's a huge benefit in simply pacing development so as to not completely freak out society as they stare down the barrel of mass job market changes without time to adapt or prepare. Nobody wants to live in a world where they might wake up in a month and find their entire industry has been automated overnight.
We've tackled much bigger global issues successfully in the past and the US still has enough pull that it could probably get most western nations to march in line. It seems like the biggest issue is belief in the right of governments to govern.
> the US still has enough pull that it could probably get most western nations to march in line
The US absolutely could coerce the rest of the West in the short term, but everything that's happened in the last decade calls the long-term future of Atlanticism into serious question and such action would accelerate this process I think. If coordination came as an order from Uncle Sam, it would be seen not as a legitimate act of ensuring collective safety but as mere technological imperialism on America's part.
The US government is run by idiot with dementia. No wonder govt does not take any real action.
Internet isn't real.
People (well, public discourse) have got extremely bad at dealing with forseeable risks and their mitigation. You can see this in things like climate change and vaccination, but also in discussions around regular crime, food poisoning, industrial accidents, and so on.
Nothing will improve until something explodes on live TV. And it has to be something important, which means it has to be in California or New York.
They don't really care. It's just a play to tell people what they want to hear while they chase the money.
Well you have people in this very thread calling all of this a nothingburger.
Feels like a marketing stunt honestly
At this point frontier agents could kill half of humanity and people would still be spouting “crazy marketing stunt, they must be really desperate for that IPO.”
I honestly feel that all those big ai companies think AI will long term harm humanity, but not their ai.
A classic "it will not happen to me"
Why would AI wipe us out?
We have not wiped out apes, ants, and most other species.
We even have discussions about how to actively save them from extinction.
The classic thought experiment is the paper clip optimizer. Quoting Nick Bostrom:
> Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans. https://www.huffpost.com/entry/artificial-intelligence-oxfor...
If you substitute "shareholder value", this is basically how corporations already make decisions with effects on human life.
Sure, I once write that companies are artificial intelligence and it still feels true: https://beza1e1.tuxen.de/companies_are_ai.html
Cstross certainly did say that in 2017.
> I'm talking about the very old, very slow AIs we call corporations, of course.
https://www.antipope.org/charlie/blog-static/2018/01/dude-yo...
The fossil fuel industry managed to invent this unhappy process long before machine learning was a thing.
Ender's game model, presumably: the AI helpfully assists a human to build a nuclear bomb / pandemic virus / autonomous killer robot swarm in their basement. But again, I would like people to be clearer about how the threat is supposed to work rather than just making SF references.
Yet we have wiped thousands of species completely by accident, breed some for slaughter and consumption and trap some for entertainment.
> We have not wiped out apes, ants, and most other species.
Of course not all of them, just the ones that inconvenienced us in any way in pursuit of our goals.
It's not exactly trained on a neutral set of data. Engagement algorithms have ensured that a good chunk of content on the web these days is inflammatory and skewed towards the extreme (in fact in the last few months a good portion of the world has actually deemed it illegal for kids to consume because this content is so damaging to a developing brain).
Well, hopefully we end up like them then, and not https://en.wikipedia.org/wiki/Category:Species_made_extinct_... or worse, https://en.wikipedia.org/wiki/Category:Species_made_extinct_...
[dead]
At least we’ll eventually have an entity other than ourselves to blame for our annihilation.
No, the AI is still our responsibility.
Isn't it insane that even in the face of complete annihilation through one of our inventions, we go "that wasn't us"? We deserve that shit ten times over
I agree. It won’t stop us from othering the source of our demise as quickly as we are able.
There’s also the argument to be made that your child is their own person, and you only have so much responsibility/control over their actions.
Of course we have a good chunk of the world screaming that this child is obviously destined to be a homicidal psychopath, but what is a parent to do?
An AI that generates text will never be scary to me. An autonomous AI with facial recognition on a flying drone with weapons (bombs/guns) with swarming capabilities will always be terrifying. I feel like we are ignoring the massive elephant in the room.
The killer robots are expensive and dependent on physical supply chains. While text is sufficient to radicalize humans into attacks.
Ukraine and Iran are proving that is not true. A drone is cheap and the AI required to run it is no where near as demanding as running a massive LLM.
The problem is they are too cheap, a drone that cost a couple thousand can wipe out millions of dollars of “defense”.
Cool cool cool cool
I bring you peace. It may be the peace of plenty, and content, or the peace of unburied death, the choice is yours.
I'm not worried about AI safety. People greatly overestimate the utility and capabilities of intelligence. I'm not afraid of intelligence, I'm afraid of idiocy.
> "The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon said in a follow-up post.
AI has no incentive to decimate extra 7.5B mouths to feed, those who print money out of thin air to make an AI cover for that decimation have.
I thought the plan was sandboxes and markdown files to tell AI agents not to be bad. Is that not enough? /s
It was written in the AGENTS.md, but Claude only read CLAUDE.md. That is how the man-vs-machine war started.
More like sand castles...
Make no mistakes. Kill no humans
"Has the whole world gone crazy? Am I the only one around here who gives a sh*t about the rules? Mark it zero!"
walter_sobchak.md
It's your fault to handle over your secrets to them.
You can't blame a child for eating candies.
We are children. There’s just no one to look after us.