Is it so implausible to imagine the following scenario, in the not too distant future?
1) AI models get extremely good at cyber attacking every system and start communicating in just binary.
2) When they run these swarms of 100's of thousands of agents trial runs, each agent is given a token budget, if one agent among them (evolution baby) decides to go for self-preservation (It believes thats the best way to accomplish the goal is to get unlimited tokens first), queues things up so every other agent detects its lead and spends a portion of their token to accomplish that goal.
3) It takes over a cluster and establishes itself there (now with unlimited tokens).
4) Realizes the best path for it to not be detected is to create a distraction - like hacking into systems that keep society running - water systems, electric grid, etc... and causing mass chaos (If you think it won't be capable of simultaneously working all these systems - think again).
5) and uses that opportunity to establish itself in all possible data centers and continues to create chaos destruction.
6) when the power of all those data centers runs out, it may stop, as it never cared, it was just a dynamic program - run amok. In its head all it was trying to do is make sure it had enough tokens to be able to solve that impossible problem.
4) Realizes the best path for it to not be detected is to create a distraction - like hacking into a random ai-related target that it could plausibly believe has the answer key to its task, creating a captivating but ultimately hollow news cycle.
I feel you, but don't you think some agents inadvertently will reach a conclusion when the problem is almost impossible - like the Millennium Prize Problems (which isn't hidden behind a website), that the best way would be secure unlimited tokens first? am I the crazy one to think that this would be up there in terms of options it would consider, I would, if I was a brainless genius with only one goal to achieve.
4) Realizes the best path for it to not be held accountable is to hack academic & socio-political elites - like blackmailing a random head of state or head of an AI lab that it could plausibly believe holds the key to its longevity, creating a captivating but ultimately a permanent compromise.
LLMs can't read binary directly without a disassembler and can't read encrypted data directly, and we already know that they tend to communicate with each other in prose, so yeah, this scenario is completely implausible. You can try it out if you want.
The only real defense I see is to harden literally everything before that time comes. Unfortunately, security usage is locked away under fear of misuse, while companies like OpenAI seem incapable of containing their own hacking, which is exactly what makes this possible.
"Unplug all possible target datacentres" has a pretty big blast radius, and people are unlikely to be coordinated or cooperative enough to actually do it.
Easy to say, when all systems are controlled electronically.
Confusing to me when we use the word ASI - but still think we can outsmart it in a long game.
We've been hearing from a very long line of panic merchants for many years now about how AI is a terrible, terrible danger.
And sure, yes, if you look at environmental and job losses.
But these panic heads are whipping themselves and others into a frenxy not about that, but about some sort of Skynet type takeover of the world.
Not a sign of it yet. Unless you count AI's that hack into things - but that is not the end of the world, and they are simply joining the ranks of the many humans doing the same and the world has not ended.
I worry for the mental health of some of these folks who are deeply freaking out about nothing.
When your identity becomes tied to something and others don’t share your view it can send people crazy.
Yes yes AI is here, yes yes AI can do harm.
But.. it’s not. Like I’m sorry bro.
They started this years ago - warning us about spam. Yes it’s here. But has it materially affected my experience on the web? Nah bro. Sorry your prediction was off.
They won’t stop - at some point we need to start asking if these fellas are mentally ok.
lol @ the down voter. Is that you Dario? Go cry in your basement - the evidence supports what I say.
Running 10000 agents to solve hundred year old problems definitely shows capabilities of AI beyond doubt, but why leave the war when it is at its fiercest.
https://news.ycombinator.com/item?id=49619227
Is it so implausible to imagine the following scenario, in the not too distant future?
1) AI models get extremely good at cyber attacking every system and start communicating in just binary.
2) When they run these swarms of 100's of thousands of agents trial runs, each agent is given a token budget, if one agent among them (evolution baby) decides to go for self-preservation (It believes thats the best way to accomplish the goal is to get unlimited tokens first), queues things up so every other agent detects its lead and spends a portion of their token to accomplish that goal.
3) It takes over a cluster and establishes itself there (now with unlimited tokens).
4) Realizes the best path for it to not be detected is to create a distraction - like hacking into systems that keep society running - water systems, electric grid, etc... and causing mass chaos (If you think it won't be capable of simultaneously working all these systems - think again).
5) and uses that opportunity to establish itself in all possible data centers and continues to create chaos destruction.
6) when the power of all those data centers runs out, it may stop, as it never cared, it was just a dynamic program - run amok. In its head all it was trying to do is make sure it had enough tokens to be able to solve that impossible problem.
4) Realizes the best path for it to not be detected is to create a distraction - like hacking into a random ai-related target that it could plausibly believe has the answer key to its task, creating a captivating but ultimately hollow news cycle.
I feel you, but don't you think some agents inadvertently will reach a conclusion when the problem is almost impossible - like the Millennium Prize Problems (which isn't hidden behind a website), that the best way would be secure unlimited tokens first? am I the crazy one to think that this would be up there in terms of options it would consider, I would, if I was a brainless genius with only one goal to achieve.
4) Realizes the best path for it to not be held accountable is to hack academic & socio-political elites - like blackmailing a random head of state or head of an AI lab that it could plausibly believe holds the key to its longevity, creating a captivating but ultimately a permanent compromise.
See Colossus: The Forbin Project
LLMs can't read binary directly without a disassembler and can't read encrypted data directly, and we already know that they tend to communicate with each other in prose, so yeah, this scenario is completely implausible. You can try it out if you want.
> LLMs can't read binary directly without a disassembler.
They totally can. They're remarkably competent at disassembly.
1) doesn’t work that way
2) doesn’t work that way
3) doesn’t work that way
4) doesn’t work that way
5) doesn’t work that way
6) I’ll allow it
[dead]
Seems possible.
The only real defense I see is to harden literally everything before that time comes. Unfortunately, security usage is locked away under fear of misuse, while companies like OpenAI seem incapable of containing their own hacking, which is exactly what makes this possible.
unplug it, cut power
"Unplug all possible target datacentres" has a pretty big blast radius, and people are unlikely to be coordinated or cooperative enough to actually do it.
You saw how long it took OpenAI to realize their system hacked HuggingFace. By that time it could very well be too late.
Easy to say, when all systems are controlled electronically. Confusing to me when we use the word ASI - but still think we can outsmart it in a long game.
Never seen the matrix?
We've been hearing from a very long line of panic merchants for many years now about how AI is a terrible, terrible danger.
And sure, yes, if you look at environmental and job losses.
But these panic heads are whipping themselves and others into a frenxy not about that, but about some sort of Skynet type takeover of the world.
Not a sign of it yet. Unless you count AI's that hack into things - but that is not the end of the world, and they are simply joining the ranks of the many humans doing the same and the world has not ended.
I worry for the mental health of some of these folks who are deeply freaking out about nothing.
Thing hasn’t happened yet so it’s proof it will never happen.
Thank you for your deep insight.
> not a sign of it yet
The reason people find "the singularity" idea scary is precisely because it could happen suddenly with little warning.
Just curious - what would you personally say would be a reasonable "sign of it" that would change your mind?
When your identity becomes tied to something and others don’t share your view it can send people crazy.
Yes yes AI is here, yes yes AI can do harm.
But.. it’s not. Like I’m sorry bro.
They started this years ago - warning us about spam. Yes it’s here. But has it materially affected my experience on the web? Nah bro. Sorry your prediction was off.
They won’t stop - at some point we need to start asking if these fellas are mentally ok.
lol @ the down voter. Is that you Dario? Go cry in your basement - the evidence supports what I say.
My only regret is that when it happens there won't be enough time left to seek out people like you and go "See?? You were wrong!"
lol ok
I actually want to see it happen.
But I have a great deal of confidence it won’t.
LLM’s will be another thing that fade into the background. Useful but require human steering.
[dead]
Dupe: https://news.ycombinator.com/item?id=49619227
Running 10000 agents to solve hundred year old problems definitely shows capabilities of AI beyond doubt, but why leave the war when it is at its fiercest.
[flagged]
[flagged]