The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive.
I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.
So many otherwise smart engineers, overrepresented on HN, have this weird mental block where they fail to take AI progress seriously. I think it's a fear that their status-defining skill could be commodified, so they adopt a mental framework that aims to dismantle the seeming legitimacy of any genuine leap in AI progress as some sort of mirage
Not just engineers - I see people attributing it to strong priors that nothing happens but in my opinion it's more some form of psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought (to many).
“Nothing ever happens” is like “you can’t beat the market.” It’s basically tautologically the strongest predictive model that we can have. A corollary is that something may happen, but it won’t be what you think.
Why should I trust these randos instead of Claude? I know Claude. I would hope Claude is smarter than whoever wants to make rules about my use of Claude. Replace Claude with Kimi, etc. if needed.
I don’t find a petition from people I don’t know who don’t care about me more compelling than my own direct experience with these models.
It's ironic that the top reply to a comment about quality of discourse on this site is an ad hominem. Most people here take AI progress very seriously judging by the number of posts on the front page about AI at any given time. And I think some engineers even explicitly want to commodify their "status-defining skill":
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
It's very easy to make those dismissals because of how transparently obvious it is that self-interest is a significant factor, no matter how loudly they insist otherwise.
> If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing?
Similar attitudes hindered nuclear energy research and deployment for decades, in turn kneecapping our species' ability to obsolesce fossil fuels and avoid the global climate catastrophe we've now made inevitable.
It's not a given that AI poses bio/political/cyber/economic risk in a way that warrants government intervention. Discourse should first revolve around whether it makes sense for the government to step in at all.
I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me. I don't worry about other weapons of mass destruction developed with LLM assistance. Nuclear weapons are already controlled much more with political pressure and the physical difficulty of e.g. enriching uranium than they are controlled by secret-keeping.
For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
The other justifications for restricting model training don't seem nearly as compelling. Worse, restrictions on model training have an infectious quality if they mean something more than "leading AI labs will voluntarily slow down together." (Like if the signatories mean that better models and software for building them should be regulated.) It's too dangerous to publish stronger models. So it's also too dangerous to publish training software. So it's also too dangerous to publish tools that allow you to build training software. (Claude's Fable won't help you write generic model training pipelines for this very reason.)
If you keep following the implications of this safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed.
> For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?
If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.
I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!
Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.
> I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!
So why have the discussion about regulating AI at all if that is the actual problem? Surely instead of blanket blocking everyone's AI models it makes more sense to control the potentially problematic sequencing?
Everything else aside, and regardless of whether that claim is currently true, I would ask you to be aware that that statement requires a way of thinking that ignores predictable future developments. Whatever the capability of any systems currently released, there will be more capable ones tomorrow, and the most you can say is that we have not mastered training spatial reasoning yet, not that we can't or won't.
> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.
Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?
Have you tried to make LLMs do that? Unless it's implementing previously developed algorithms they screw it up so repeatedly it's incredible. (And even when it is clearly defined it's obviously their weak spot).
The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology.
There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.
You can do CRISPR in your kitchen. It's been possible practically as long as CRISPR has been a thing. It's not particularly difficult, to the point that my biggest counterpoint to the whole AI doomerism argument on the subject is that nobodies done it yet.
You don't need some genius AI to design a bioweapon, you need to do a bit of light reading on bioengineering. The fact that some mad scientist hasn't ended civilization is evidence of the fact that it's not some existential threat that justifies wild levels of government control.
Unfortunately, AI development is dominated by an even mix of 1) Machine God Cultists, 2) Morally Bankrupt Mercenaries. All three agree on the fact that "thou shalt have no gods before me", because it'll make them richer than sin, and allow them to shape the AI landscape in their image.
Can you point me at this light reading? I don't wanna do CRISPR in my kitchen, but itd be cool to know more about it. I thought I needed to be basically pre-med to grasp it.
> ...I see how no-guardrails LLMs could be truly disastrous...
by all accounts this is what's what because of how...
> You can do CRISPR in your kitchen.
.. and because of the same dynamic we see occurring in cybersec.
If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.
If you're good enough at biology and have certain ideologies, but not good-good at specific skillsets like genetic engineering in your kitchen, do LLMs get you good enough? Does the impact on cybersec transfer over to this domain? H
Hard to say, but it has enough risk indicators for me to think the pushback against these possibilities is the height of otherwise talented engineers being myopic.
If good enough at biology can include all the PhD programs across all the world, and you pair other known events like cartels and terrorism groups hiring chemistry PhDs from University of Unemployed Chemistry PhDs, then the next logical extension of that theme with this tech is intense.
If the accuracy of your biology claim is "hard to say" then why would pushback be self-evidently myopic?
You and others are making a claim about LLMs enabling biological dangers. Some people disagree with you, pointing out that the incremental knowledge an LLM provides over say google or a library card may not be that much.
What seems myopic to me is un-skeptically accepting "AI dangerous, must regulate" messaging from frontier labs that have trillions of dollars of vested interests in this message.
If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.
Wet work in a lab isn't the same as writing code on a computer.
If you screw up with a lethal pathogen in a lab, you die.
If you screw up with code on a computer, you go get lunch.
This experiment has been run, and even the biological stuff is far fetched.
I've written about this before, but the issue is that these people have worked with computers their entire lives, and they keep projecting outwards from there.
The 20th century was dominated by mad scientists (mostly Teller) and generals (bombs away LeMay), but one thing that I respect about them is that they didn't just sit and guess about probable futures. They sat down and actually ran experiments.
For example, in the 1960s, there was a whole generation of people asking "who's next?" after the US, USSR, the UK and France had made The Bomb. And instead of just gesticulating wildly and trying to keep fighting the losing fight of "classify everything, admit nothing," Teller found three smart, young physicists (postdocs) with 0 nuclear weapons experience, and indeed 0 weapons experience, and ran an experiment called the Nth Country Experiment, where they were were asked to make a design for a working bomb.
After 2+ years, they were able to. And so proliferation work shifted from controlling knowledge about the technology and the technology itself to materials.
Something similar was done with bioweapons via Project Bacchus, where they gave actual bioweapons experts carte blanche to set up a secret bioweapons lab with COTS equipment. They succeeded. This was in the early 2000s.
This then led to surveillance of specific purchase combinations and equipment. Because tbh, most weapons of mass destruction are commodity technology. Nuclear weapons are 80+ years old. Chemical weapons are over a century old. Bioweapons are god knows how much older. And it's not the knowledge of these things that matter, but intent, materials, and the ability to create them.
Just because you know something about a thing doesn't mean that you're capable of doing that thing.
Let's take bioweapons.
They keep comparing wet work in a lab to writing code on a computer.
When you screw up an exploit, you fail to execute the exploit. Famously, just like software's near zero marginal cost of distribution, the marginal cost of failure is nearly zero.
You can screw up an infinite number of times on your way to a successful exploit.
If you screw up with lethal agents in a lab? You die.
Here's a non-exhaustive list,
Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease
A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived
Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom
Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. The accident occurred at the Scientific-Production Association "Vektor" (today the State Research Center of Virology and Biotechnology "Vektor") in Koltsovo, USSR (today Russia).
"lethally infected with the Marburg virus after accidentally pricking himself"
Anything lethal enough to kill other humans is lethal enough to kill you.
And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.
And this, of course, doesn't even cover the cost of equipment, the precursors, sourcing the highly specific materials needed, then setting the equipment up... etc.
The same is true for the Bosch-Haber / Haber-Bosch process, which famously made WW1 possible. Every HS'er learns about the process and the steps. Steps that were classified once upon a time and were the subject of negotiation at the Versailles.
Does that mean a HS'er (or any adult) can set up an experiment that works at 177 times the pressure of the Earth's atmosphere to do anything at any scale without significant infrastructure and help?
No, it doesn't work like that.
The people who can do this are domain experts, and they've been able to do this with COTS stuff since the 1990s, at the very least, for a price of around $2M. And those people don't need a LLM to tell them what to do. In fact, they're the exact people who'll have access to unrestricted versions of these LLMs.
And from a security perspective, I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm to the effort of finding people who could be planning such a thing than it helps. It takes more resources to go through the mass of false negatives that have now been created as matter of policy.
These experiments have been run. And we can run them again.
The fictional scenario of someone learning how bioweapons work and conjuring up a plague isn't real and it hurts humanity as a whole to impede the sciences over it.
Because what someone can flail around in / do is learn about immunology / try to "cure cancer" with a LLM and hopefully get started on a long career in medicine. Or, a discovery that matters.
Because in those cases, if and when they do end up at a lab, screwing up doesn't mean death. Just tons of wasted time (and money). And they will fail / screw up. Just look at literally every undergrad in any lab and the expensive messes they create.
Thank you for taking this seriously enough to write this, and anyone else likewise.
But this is attacking a strawman, amateur bioterrorists. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.
Being at risk of killing yourself also just makes success X% less likely, but if X < 90 that doesn't mitigate much.
It sounds like your claim is "AI is a force multiplier, therefore it is dangerous and must be regulated." Not trying to strawman - I read your comment twice and I think that's what you're saying.
By that logic Excel, the internet, calculators, and air conditioning are all force multipliers in that all of the make it easier for an expert who can make bioweapons able to do so more quickly / comfortably / easily. Obviously it wouldn't make sense to "pace" any of these technologies. Even social media or email could be considered force multiplier in that they could literally allow you to increase the number of collaborators working on supposed bioweapon. Credit cards and modern logistics force multiply your ability to purchase precursors.
Also it would be a force multiplier on the prevention/mitigation side as well. Police armed with AI to scan for suspicious precursor purchases or review security camera footage after the fact would be far more effective than before. It's not obvious that the force multiplication is greater to the bad actor or to those who oppose them.
But this seems like it's attacking a strawman. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.
I'm glad that you brought that up, genuinely so.
I want to ask you a question, do you feel like that this is the problem being solved by the current "safety features?"
Let me rephrase my question, do Anthropic and OpenAI implement the same "guardrails" and "safety features" for the NSA? Does the Mythos NSA Edition™ have these restrictions? What about the one that's running as a part of Maven?
How does stopping me from asking about rabbit sex protect you from nut jobs with a security clearance using these systems to go off and make weapons?
The only private organization successful enough to make biological and chemical weapons is Aum Shinrikyo and they had university affiliation and nearly a billion dollars in the 1990s.
The guy who led Aum's bioweapons program was
Seiichi Endo was born in 1961 and attended Obihiro university of Agricultural and Veterinary Medicine. After graduating, he became a student at the Kyoto university medical school research department focusing on AIDS-related gene research at the viral research center. He joined Aum in 1987, became a monk in 1988 and later led Aum’s biological weapons program.
and,
Among some of Japan's "best and brightest" who joined the cult included a former researcher of the National Space Development Agency of Japan, an expert on chemical weapons who majored in organic physics at Tsukuba University, a researcher who studied elementary particles, a reporter with a major Japanese newspaper, a physicist from Osaka University, a cardiac specialist, and an organic chemist, to name a few.
Through its network, Aum Shinrikyo successfully recruited over 300 scientists and engineers—including employees at the Kurchatov Institute, the premier nuclear facility in Russia—who were either attracted to the apocalyptic ideology or lured with financial incentives.50 Aum Shinrikyo’s Russian contacts enabled group members to access black market materials and hardware.
and,
In response, the cult constructed the $30 million USD Satyan 7 facility which was equipped with three laboratories, a computer control center, and five reactors, all of which was made from corrosion-resistant Hastelloy, ideal for the production of chemical weapons
Why didn't anyone ask questions about a known cult importing an absurd amount of precursors, industrial-grade HEPA filters, fermenters, and lab equipment?
It's because they'd recruited members of the military, bribed local policemen and politicians, and used their sway to silence critics (or kill them).
How does stopping me from asking Claude about rabbit sex stop people like them?
From where I'm standing, Anthropic and OpenAI would have sold Aum a subscription.
The Fort Detrick guy would have access via the US Government and Aum had university affiliation. I am yet to read a serious proposal that actually deals with these risks and the other real risks of this technology.
Are they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI.
If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?
i think part of the problem, though, is that we've already hit many of the warning signs that AGI is coming, and people have kept moving the goalposts.
if you were to tell someone from ten years that there's an ai that has passed the turing test and has made novel mathematical discoveries, i think many would say AGI is already here.
The old observation that people tend to define AI as whatever computers can't do yet is as true as ever. It's getting a bit absurd, moving from demanding "general intelligence" to replicating human cognitive phenomenology (the experience of cognitive activities). Yet LLMs can already somewhat (confabulation-prone) introspect their own "internal unspoken thoughts" in their residual streams, quite fascinating.
I wouldn't frame it as moving the goalposts in a bad way, though. We understand more now than we did then, so we have a better idea of what the actual goal is, and moving the goalposts to frame that goal is more rational than insisting the goalposts stay where they were ten years ago even though the entire pitch has changed since then. I'm straining the analogy, obviously.
Moving the goalposts, so to speak, is how science works! We must of course update the things we think based on an improved understanding of how the world works. Only the deeply incurious could consistently demand that one adhere dogmatically to a prior way of understanding the world in the face of changing knowledge. It's important not to treat science as a game to be won (of which 'goalposts' are evocative), but as the collaborative effort that it is.
Note well that I have not said LLMs are not useful, only that no amount of interaction with them has convinced me they remotely approach general intelligence. Their actual function of heuristically regurgitating all information ever known to mankind remains extremely useful in lots of domains.
And by the way, it's not obvious an LLM could fool me (or the average person) over a sufficiently long period of time, exactly because they are heuristic machines. That they lack thought guided by underlying cognitive models seems to always leak out, in the end. I suspect pass rates by LLMs on long-range Turing tests (if there have been any conducted) might drop as people become acclimatized to them.
what do you mean by 'general intelligence'? i definitely agree that in comparison to humans, they have some definite weaknesses. however, i think they also have definite strengths. and i don't think their current weaknesses imply that much about a potential singularity or ai risk.
Sorry, I should've used clearer language. I just mean intelligence in the sense humans possess it. Computers have always been able to exceed limited elements of human intelligence (say, at arithmetic), but the problem is to simultaneously match or exceed all of them. I specifically think that the important parts they currently lack make them a no-go for supposed existential risk.
We're commenting on a site run by a Silicon Valley tech start-up accelerator. "Move fast and break things" is the accelerationist way, even if the "things" they break are a stable, functioning social order. As long as you IPO and cash out along the way, you can wreck everyone else's lives just fine, in the minds of this sort.
Every single technology in history has had people claiming potential existential risks that didn’t materialize. Burden of proof is on people making the risk claims. Those have to be specific and unique to that technology to grant special actions.
i'm not trying to be rude, but i really don't get what you mean by this?
(https://www.amazon.com/Superintelligence-Dangers-Strategies-...)
there's been people making specific and unique risks to this technologies for years, before AI was even present. anthropic was literally founded because they were worried about existential risk?
also, previous technologies have had existential risks? do you deny that nuclear weapons could have killed humanity? obviously, there have no been existential risks that have killed all of humanity, because by definition, that can only happen once. it will literally be impossible for someone to ever point to an example of that happening, because they will be dead by the time that happens.
And video games, democracy, novels, TV, cars, photography, cryptography, the Beatles… pick your favorite and read commentary / opinions of the time. Many thought that came with unique risks never seen before and must be stopped!
How many of those concerns came from the people building the technologies? John Carmack wasn't up in arms about how Doom was going to destroy society. Charles Dickens didn't think novels rotted the brain. Johannes Gutenberg wasn't concerned about the effects of a free press.
None of the previous concerns were true and nothing indicates current ones are. It’s extrapolation, hypothetical apocalyptic imagined scenarios and piles of assumptions on how the technology could evolve. What makes even less credible in the AI case is that there’s financial / power to be gained by those making the claims.
And to your point. Don Quixote, the first modern and greatest novel of all times, is about a guy that lost touch with reality from reading too many books
None of the previous concerns were true, largely because they came from laypeople who didn't understand the technology. A prominent exception is nuclear weapons, where their creators did worry their creations could end up destroying the world. I think most people would agree those concerns were justified, and it's not any lack of capability that means we're still here.
The motivations aren't unique here. Restrictions on the development of almost any technology benefits incumbents by providing a moat.
Employees inside frontier labs have the chance to make life changing money if they can make it to an IPO before the music stops. Have you considered that might be the reason they are so eager to claim "ai dangerous - regulate ai"? The nuclear scientists at los alamos did not have generational wealth incentivizing them to raise "concerns" to the public.
If your only evidence that the claim "ai dangerous - regulate ai" is true is "insiders who stand to make billions if they can regulate away the competition" told me so, that is pretty weak evidence indeed.
If a group of concerned scientists (without $$$ in stock options) demonstrated that anyone with one year of HS chemistry, an Amazon account, and a ChatGPT max plan could build a bioweapon 100x more lethal than an AR15 mass shooting, then that would be worrisome. This is really not that high of a bar but even it has not been met, but oh no we should start regulating yesterday.
I mean, Cervantes was parodying medieval romances. That's what Don Quixote was reading, not modern novels.
The point is, it should give you at least some pause when the people most familiar with a technology are among those making the strongest statements about its potential dangers. Extrapolation has worked extremely well in predicting what AI will be capable of over the past 5 years; how confident are you that it will fail before we reach apocalyptic scenarios?
This is serious. We're in uncharted territory. The geopolitical context is already complicated enough that we shouldn't be dismissive of such a rapid technological race. I'm glad AI leaders are taking it seriously. I'm surprised so many HN commenters aren't.
On the other hand, I'm pessimistic about initiatives like this being effective on a global scale (though they could certainly work at the national level).
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest
Pointing out the endless stream of lies and self serving half truths these terrible people engage in helps the younger generation understand the world they were born into.
It's constructive and helpful. It's why I'm replying to you.
You interpreting it as 'shallow dismissal' is shallow dismissal on your end. You're the shallow dismisser, not the people calling terrible people terrible.
As for your little petition. It's as good as eliminating plastic straws to save the environment.
Terrible people with power they don't deserve doing pathetic little, tiny 'good' deeds to sleep better at night needs no explaining.
What happens when a country doesn't agree to play along with this little decel game?
The genie is already out of the bottle and now a bunch of people who helped release it think that the collective powers of the world are just going to put it back in.
How far are the saftyists willing to go? Bombing data centers? Starting wars? Embargos?
It's a terrible dilemma to be in. If the threats are real, it gives all the more incentive for some actor to race and try to grab it first. If not, then any punishment isn't proportional. Explain to a nations citizens under a 10 year embargo, "sorry about your quality of life, we all just got a little spooked by science fiction for a second."
Folks try to reach for bad analogies: nuclear bombs and biological weapons. But conveniently omit that by the time the regulations took place for these things we had unequivocal proof of their destruction, scope and scale.
So, what then? We're supposed to collectively stop innovating and put the keys to the kingdom in the hands of a few capitalists, a corrupt is administration and the ccp?
I reject that these agreements should take place, on the grounds that the proposal is delusional and not founded in reality.
On the geopolitics, the American domiciled firms are completely in bed with the US military and US government, and have made it absolutely clear that they are committed to the domination of the US government. So when you say "geopolitical destabilization", you mean "only the US government is allowed to destabilize, which they have done for decades and wish to continue doing unfettered."
And given what we have seen from that government -- piracy, terrorism, war-mongering, thievery, and just profound stupidity -- no thanks, in that case I'd like to see every American AI company utterly obliterated by any other country. If China becomes utterly dominate in AI, awesome.
"Many were increasingly of the opinion that they'd all made a big mistake in coming down from the trees in the first place. And some said that even the trees had been a bad move, and that no one should ever have left the oceans."
The historic analog most befitting AI are weapons. It provides an advantage to companies and governments in creating weapons, in surveilling people and thus gaining leverage over individuals, in controlling what individuals can do or cannot stop doing. It thus follows the laws of weapons control - the more restrictions are threatened to be imposed, the more even the smallest actors will fight to get more of it. They may not understand or know what they are getting, but they will want more of it because their potential rivals have it. Governments are the ultimate customer here - they have the money, the energy, and the appetite to consume AI infinitely in battlement against opponents - foreign and domestic. The only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment (the upside of that is it advances the AI for future use and is thus in the interest of governments as well). Modern large models seem to be compressible and thus we may see more and more tiny sub-specialized varieties. But the push to slow the frontier in weaponry always leads to the reverse.
Anyone serious about this should participate in pacing the frontier by quitting their job so as to not potentially contribute to the problematic progress. Be the change you want to see!
Some of us considered seeking one of these jobs and decided not to. Some have quit (e.g. Hinton, OpenAI people after OpenAI reneged on safety commitments). Now you're getting this message from people who passed the previous filters. If you think a brake is indicated and not just easing up on the accelerator, you need collective action as the statement said.
That's an application of the "Tragedy of the commons": One person stopping to push the frontier just means that this person will lose their job, the frontier will be pushed by the others. The solution is to push for a binding rule instead that will make everyone slow down in the racing towards artificial superintelligence.
We live in a society that has systems of market competition between companies and economic/military competition between countries. There's not just one country working on frontier AI, and not just a few companies in each country either. The incentives of these systems ensure AI advancement will happen quickly from here on out, no matter what, absent governmental action.
An individual researcher leaving the company does nothing, other than give their seat to someone else who likely doesn't share their same concerns.
Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
> Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
No, it just comes across as "I want to get paid as if I was doing the thing I am employed to do by threatening you all about how terrible it would be if I actually did it".
They aren't even being semi imaginative and going on strike or anything.
We make binding rules all the time. We didn't fix the ozone layer by encouraging people to quit their jobs producing CFCs. We lobbied for international regulation to ban them.
In your example there is nothing to be gained by not complying.
Failing to comply with what is in reality arms control is a widespread occurence. It's notable the only historical way to handle this (Schelling covered it at length) is tolerance of mutual espionage on a level that would simply not be tolerable to any of the groups involved.
I should add one of Schelling's other pet subjects was "time to weapon being viable". i.e. the distinction between a nuclear power and non-nuclear power isn't quite right, as there's a huge class of countries that could become so fairly fast if they were so inclined, and that this situation is somewhat inevitable. It is this that leads to efforts to denuclearize, even on a civilian level, "problematic" countries. You want to stop frontier AI and you'll have to be rolling back a lot of existing deployed technology.
Great comment. I haven't read Schelling but will check that out. I think you're right that international cooperation on the topic is likely impossible at present, for the reasons you stated. I'm not sure what it would take to shift the Overton window, but it's probably not pretty.
Personal responsibility is a difficult concept for be to apply to massive efforts like this. When there’s one person and you can talk to them personal responsibility is the number one topic. When you’re talking about a multi-trillion dollar bet and hundreds of thousands of people it doesn’t make sense anymore.
Same logic goes for other common issues where personal responsibility is lauded. Obesity - real solutions are policy and medicine. Abortion - education and contraceptives.
Can someone help clarify what it means, concretely, to make binding agreements to proceed "safely and slower" in the context of AI research (whether or not at frontier labs)? Are there specific policies and actionable guides being proposed?
It's an admirable idea, but I don't see how you can slow this down in practice. We certainly know China will not, and they are already at the frontier.
RSI is too important and too tangible at this point for everyone to slow down. It's the equivalent of telling the world to stop developing nuclear weapons after Hiroshima. Just not going to happen, and in the meantime dramatically increases the risk that the power of AGI lands in the hands of the few (read: the political class) rather than empowers the people.
This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow.
The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect.
The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection.
Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology.
Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals.
Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.
Why not? Slowing down / halting biological weapons research mostly worked. Yes there have probably been modest sized defections here and there, but the pace of bioweapon development is a crawl compared with (a) what is possible with science already, and also with (b) the pace of bioweapons development from 1910 to 1970.
The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.
Something people don't seem to see, and not that it's placed directly in front of you, is that pacing the release and adoption of AI is already happening.
Cost-per-token determines the economic viability of tasks that AI gets applied to. How many tokens would it take to automate all of the responsibilities of a gas station clerk? Probably more than their hourly wage. Job safe.
A slightly larger stretch, that most people attribute to profit margins (labs do not have profit margins, they have unlimited invested capital and no expectation of short-term returns), is the flat-rate billing plans of most code products. Why doesn't Anthropic have a 50x or 100x Max plan? Not that it's impossible, that it releases tokens into the workforce at a greater rate (note that their are hour, day, week limits across most of these services as well).
Hypothetical, but reasoned, and I do not expect most agree with me. If compute as a variable is inferred, this supports the incorporation a deliberate (redundant) one as well.
I agree with the substance of your comment that token economics pace the diffusion of AI into the labor market. But I think they have a much smaller influence over the pace of the frontier, right?
Fair point! I guess I read "pacing the frontier" as "pacing the release of the frontier" surely no lab's internal research team is just going to stop innovating?
Too bad these researcher's organizations spent their efforts supporting the dumbest, ficklest possible version of the U.S. government to guide us through the gravity of this moment.
> We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
This is quite vague to me, what exactly do you mean?
It means that they request the structures needed to navigate the event horizon of recursive self improvement be put into place. Probably: International agreements, verifiability of an AI pause on the level of graphic cards as well as through physical inspections of datacenters. Like it's done with nuclear energy.
'The Fear' - is not some magic intelligence will come alive and be Skynet, these people are ridiculous.
People will use the compute power to do nasty things, we'll be overwhelmed with bots, regular companies will use them for dark patterns, political actors will use them to cheat.
It will be 100% humans using it for bad stuff, and in not very novel ways.
What these people want is same thing people in every other field want too, which is being able to use AI as a tool and nothing more. Off course, some of the people that signed this might not recognize this, and the people who didn't sign it are actively working against such an equilibrium.
The Harvard Mark I was used by von Neumann to calculate the physics of implosion. And then right afterwards, ENIAC was used to calculate the feasibility of the H-bomb. They called it the "Los Alamos problem."
The first computer network, SAGE (or, Semi-Automatic Ground Environment) brought together RADAR data from across the US with air-traffic control data and military flight data to detect overflight by the USSR. It could direct and control fighters to intercept those planes.
If you were sitting there, in 1955, would you have believed that computers and their networks would bring about an extraordinary age?
Imagine if von Neumann and his ilk started campaigning in 1955 to limit the development and use of computers, because surely they can only be used to make weapons, right? Right?
We were lucky to have visionary people like Joe Licklider who could look past the present and see the future. In 1963, he wrote a memo calling for an "Intergalactic Computer Network" (because he knew engineers under-delivered. So if he asked for a galaxy, at least he'd get a planet). And he gave these machines to scientists and people like the original hackers at MIT to play with.
What if Licklider had written memos about how we had to "pace" computers to make sure that other countries wouldn't be able to develop implosion technologies or systems like SAGE? What then?
Do you think you would have the jobs and the prosperity you do today?
If instead of giving away this technology, spreading it, and putting down the seeds for ARPA-net, Licklider had written memos from "concerned scientists" (and I'm sure he could have found a bunch) talking about how computers can only be used for weapons and war! So stop the proliferation of computers! We can't imagine a world with computers small enough to fit inside everything! Even wall sockets! What if your wall socket could do war?!
They didn't, because they knew there's no better way to kill a better future for all mankind than to control it.
† yes, peace. We are more peaceful than we have ever been. Humans can be a violent lot even when reluctant to kill.
We were incredibly lucky that computing and nuclear weapons led to a stable geopolitical equilibrium (mutually assured destruction). There were certainly a few scary points where we were still settling into that equilibrium during the Cold War.
I don't have a strong opinion either way, but it seems very hard to predict. Why do you think AI advancements will lead to a stable equilibrium?
Here's what I know. Every day, I read a 5 to 10 incoming emails from young people in rural Africa, Asia, Europe and (heck) the US. People who have had no other resources and options, but are tremendously gifted and somehow got their hands on an internet connected device (usually a very cheap smartphone and/or cheap old computer).
These people, who are usually teenagers, end up getting connected to Claude, who then gives them a list of things to learn, people to contact, and helps them make things. And they actually do. They make robots. They learn new things and try to improve themselves and others. And some want to change the world / do groundbreaking research.
When you sit down and start pricing and doling out AI from behind regulatory gates, you shut those people out.
There are a lot of smart people in this world who have traditionally been unable to get access to resources and education. MOOCs didn't do much / help. But these AIs? They were born multi-lingual. They can talk in any language. They can tailor their responses to their audience. And they can help broaden these people's horizons.
There are no tutors around them. There is no one who is coming to help. But an AI that can run off of a small battery and a connection to a cellular network can give them access to genuine intelligence to make their world better.
I met one sixteen year old who was accessing Claude surreptitiously (by begging people) and was trying to apply some really fascinating biomimetic fluid dynamics observations to aerospace. This person was born in one of the more under-resourced parts of the world. And Claude gave them a list of people to reach out to who could help.
If you want a golden age for humanity, you should be asking for Licklider's Man-Machine Symbiosis.
I think this is a spot on analogy and would two more: the hand wringing over the release of the printing press and the internet. In both cases the people in power were terrified that the masses would use their newfound power to do dangerous things. In both cases bad things did happen (market bubbles, scams, witch hunts, spread of knowledge on making poisons), but trying to put the cat in the bag by pacing the printing press or the internet would have been a fool's errand, and over the long run both inventions massively improved the lives of society overall.
The problem is how do you enforce coordination? This isn't like nuclear testing that can be detected around the world on seismographs or from satellites in space.
AI research happens in benign looking buildings that could be doing a hundred different non-AI workloads, much of it things that industry or governments would not want 3rd parties prying into. There is too much to potentially gain to trust coordination.
There is a lot of research out there with answers to your questions. GPU supply chains are extremely constrained, making it a surprisingly tractable problem at least for now.
How do you tell the difference between inference and training workloads from outside? The same GPUs can be used for both and the pacing suggested here relates to training and not inference as far as I can tell.
The website does not list all the comments but just around ~50 without any way to load all the ~+1100 more (1132, as of 2026-07-28 2100 UTC).
The number of em-dashes is questionable, existing in many, if not all, the comments.
> ...AI may end up progressing far more quickly than societies and states can adapt... ~ Stephanie Chan, Google
I am sorry, but I am not sure whether it's adequate and fair to allow join such statements/petitions to only those who live in it. Isn't it like allow deciding the future for weaponry laws to weapon and war enthusiasts only?
And, this is not serious at this point:
> ... ~ Source: Anonymous, Meta
> ... ~ Source: Anonymous, Google
> > We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
> 1122 employees of frontier AI companies
'lets us be the gatekeepers of AI because we know better than you'
access to gpu doesnt equal to any special insights to predict future of ai. Everyone and their grandma with enough compute is putting out a model now.
> It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.
This is just the standard AI "doomer" talking point. The dangers of AI just used as a compliment sandwich. "AI is so cool and awesome and that's why we need to slow down."
The real risks are much more boring and much less considered. For just one example:
AI is set to completely upset the Software Development industry. There are 1 to 3 million software developers in the US, depending on how you draw the line exactly. A low-few hundred thousand of those are visa workers.
With AI set to significantly cut into tech employment, what's going to happen to those visa workers? ... The Trump admin is not particularly subtle, even openly talking about retroactively revoking birthright citizenship.
AI & Tech companies love to talk about how their AI is going to hack everything if they're not careful. But not a peep about these ethics concerns.
Even if you [the reader] consider deporting visa workers the ethical move if the amount of jobs shrink, that's not something to be done unprepared or out of nowhere.
Has there ever been a case in human history when technology just stopped advancing, simultaneously world wide, only because a bunch of people wanted it to?
I think these people are either incredibly naive, or, more likely, not acting in good faith.
> The effects of these guidelines are still being felt through the biotechnology industry and the participation of the general public in scientific discourse. Due to potential safety hazards, scientists worldwide had halted experiments using recombinant DNA technology, which entailed combining DNAs from different organisms. After the establishment of the guidelines during the conference, scientists continued with their research, which increased fundamental knowledge about biology and the public's interest in biomedical research.
The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive.
I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.
So many otherwise smart engineers, overrepresented on HN, have this weird mental block where they fail to take AI progress seriously. I think it's a fear that their status-defining skill could be commodified, so they adopt a mental framework that aims to dismantle the seeming legitimacy of any genuine leap in AI progress as some sort of mirage
Not just engineers - I see people attributing it to strong priors that nothing happens but in my opinion it's more some form of psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought (to many).
“Nothing ever happens” is like “you can’t beat the market.” It’s basically tautologically the strongest predictive model that we can have. A corollary is that something may happen, but it won’t be what you think.
Why should I trust these randos instead of Claude? I know Claude. I would hope Claude is smarter than whoever wants to make rules about my use of Claude. Replace Claude with Kimi, etc. if needed.
I don’t find a petition from people I don’t know who don’t care about me more compelling than my own direct experience with these models.
It's ironic that the top reply to a comment about quality of discourse on this site is an ad hominem. Most people here take AI progress very seriously judging by the number of posts on the front page about AI at any given time. And I think some engineers even explicitly want to commodify their "status-defining skill":
https://nitter.net/__tinygrad__
> Most people here take AI progress very seriously
You have only an extremely weak basis for this claim and have essentially just made it up.
> judging by the number of posts on the front page
Posts reaching the front page is not a reliable way to measure what "most people here" believe.
> It's ironic that
It's not irony.
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
It's very easy to make those dismissals because of how transparently obvious it is that self-interest is a significant factor, no matter how loudly they insist otherwise.
> If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing?
Similar attitudes hindered nuclear energy research and deployment for decades, in turn kneecapping our species' ability to obsolesce fossil fuels and avoid the global climate catastrophe we've now made inevitable.
There is constructive discourse, but this kind of commentary doesn't get to the top:
https://news.ycombinator.com/item?id=49077753
https://news.ycombinator.com/item?id=49078376
It's not a given that AI poses bio/political/cyber/economic risk in a way that warrants government intervention. Discourse should first revolve around whether it makes sense for the government to step in at all.
I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me. I don't worry about other weapons of mass destruction developed with LLM assistance. Nuclear weapons are already controlled much more with political pressure and the physical difficulty of e.g. enriching uranium than they are controlled by secret-keeping.
For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
The other justifications for restricting model training don't seem nearly as compelling. Worse, restrictions on model training have an infectious quality if they mean something more than "leading AI labs will voluntarily slow down together." (Like if the signatories mean that better models and software for building them should be regulated.) It's too dangerous to publish stronger models. So it's also too dangerous to publish training software. So it's also too dangerous to publish tools that allow you to build training software. (Claude's Fable won't help you write generic model training pipelines for this very reason.)
If you keep following the implications of this safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed.
> For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?
If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.
I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!
Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.
> I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!
So why have the discussion about regulating AI at all if that is the actual problem? Surely instead of blanket blocking everyone's AI models it makes more sense to control the potentially problematic sequencing?
> We know they cannot do spatial reasoning
Everything else aside, and regardless of whether that claim is currently true, I would ask you to be aware that that statement requires a way of thinking that ignores predictable future developments. Whatever the capability of any systems currently released, there will be more capable ones tomorrow, and the most you can say is that we have not mastered training spatial reasoning yet, not that we can't or won't.
> Are we all doing CRISPR in our kitchens now?
https://www.the-odin.com/crispr-kit/
> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.
Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?
Have you tried to make LLMs do that? Unless it's implementing previously developed algorithms they screw it up so repeatedly it's incredible. (And even when it is clearly defined it's obviously their weak spot).
The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology.
There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.
You can do CRISPR in your kitchen. It's been possible practically as long as CRISPR has been a thing. It's not particularly difficult, to the point that my biggest counterpoint to the whole AI doomerism argument on the subject is that nobodies done it yet.
You don't need some genius AI to design a bioweapon, you need to do a bit of light reading on bioengineering. The fact that some mad scientist hasn't ended civilization is evidence of the fact that it's not some existential threat that justifies wild levels of government control.
Unfortunately, AI development is dominated by an even mix of 1) Machine God Cultists, 2) Morally Bankrupt Mercenaries. All three agree on the fact that "thou shalt have no gods before me", because it'll make them richer than sin, and allow them to shape the AI landscape in their image.
Can you point me at this light reading? I don't wanna do CRISPR in my kitchen, but itd be cool to know more about it. I thought I needed to be basically pre-med to grasp it.
> ...I see how no-guardrails LLMs could be truly disastrous...
by all accounts this is what's what because of how...
> You can do CRISPR in your kitchen.
.. and because of the same dynamic we see occurring in cybersec.
If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.
If you're good enough at biology and have certain ideologies, but not good-good at specific skillsets like genetic engineering in your kitchen, do LLMs get you good enough? Does the impact on cybersec transfer over to this domain? H
Hard to say, but it has enough risk indicators for me to think the pushback against these possibilities is the height of otherwise talented engineers being myopic.
If good enough at biology can include all the PhD programs across all the world, and you pair other known events like cartels and terrorism groups hiring chemistry PhDs from University of Unemployed Chemistry PhDs, then the next logical extension of that theme with this tech is intense.
If the accuracy of your biology claim is "hard to say" then why would pushback be self-evidently myopic?
You and others are making a claim about LLMs enabling biological dangers. Some people disagree with you, pointing out that the incremental knowledge an LLM provides over say google or a library card may not be that much.
What seems myopic to me is un-skeptically accepting "AI dangerous, must regulate" messaging from frontier labs that have trillions of dollars of vested interests in this message.
If you screw up with a lethal pathogen in a lab, you die.
If you screw up with code on a computer, you go get lunch.
They're not the same.
This experiment has been run, and even the biological stuff is far fetched.
I've written about this before, but the issue is that these people have worked with computers their entire lives, and they keep projecting outwards from there.
The 20th century was dominated by mad scientists (mostly Teller) and generals (bombs away LeMay), but one thing that I respect about them is that they didn't just sit and guess about probable futures. They sat down and actually ran experiments.
For example, in the 1960s, there was a whole generation of people asking "who's next?" after the US, USSR, the UK and France had made The Bomb. And instead of just gesticulating wildly and trying to keep fighting the losing fight of "classify everything, admit nothing," Teller found three smart, young physicists (postdocs) with 0 nuclear weapons experience, and indeed 0 weapons experience, and ran an experiment called the Nth Country Experiment, where they were were asked to make a design for a working bomb.
After 2+ years, they were able to. And so proliferation work shifted from controlling knowledge about the technology and the technology itself to materials.
Something similar was done with bioweapons via Project Bacchus, where they gave actual bioweapons experts carte blanche to set up a secret bioweapons lab with COTS equipment. They succeeded. This was in the early 2000s.
This then led to surveillance of specific purchase combinations and equipment. Because tbh, most weapons of mass destruction are commodity technology. Nuclear weapons are 80+ years old. Chemical weapons are over a century old. Bioweapons are god knows how much older. And it's not the knowledge of these things that matter, but intent, materials, and the ability to create them.
Just because you know something about a thing doesn't mean that you're capable of doing that thing.
Let's take bioweapons.
They keep comparing wet work in a lab to writing code on a computer.
When you screw up an exploit, you fail to execute the exploit. Famously, just like software's near zero marginal cost of distribution, the marginal cost of failure is nearly zero.
You can screw up an infinite number of times on your way to a successful exploit.
If you screw up with lethal agents in a lab? You die.
Here's a non-exhaustive list,
"lethally infected with the Marburg virus after accidentally pricking himself"Anything lethal enough to kill other humans is lethal enough to kill you.
And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.
And this, of course, doesn't even cover the cost of equipment, the precursors, sourcing the highly specific materials needed, then setting the equipment up... etc.
The same is true for the Bosch-Haber / Haber-Bosch process, which famously made WW1 possible. Every HS'er learns about the process and the steps. Steps that were classified once upon a time and were the subject of negotiation at the Versailles.
Does that mean a HS'er (or any adult) can set up an experiment that works at 177 times the pressure of the Earth's atmosphere to do anything at any scale without significant infrastructure and help?
No, it doesn't work like that.
The people who can do this are domain experts, and they've been able to do this with COTS stuff since the 1990s, at the very least, for a price of around $2M. And those people don't need a LLM to tell them what to do. In fact, they're the exact people who'll have access to unrestricted versions of these LLMs.
And from a security perspective, I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm to the effort of finding people who could be planning such a thing than it helps. It takes more resources to go through the mass of false negatives that have now been created as matter of policy.
These experiments have been run. And we can run them again.
The fictional scenario of someone learning how bioweapons work and conjuring up a plague isn't real and it hurts humanity as a whole to impede the sciences over it.
Because what someone can flail around in / do is learn about immunology / try to "cure cancer" with a LLM and hopefully get started on a long career in medicine. Or, a discovery that matters.
Because in those cases, if and when they do end up at a lab, screwing up doesn't mean death. Just tons of wasted time (and money). And they will fail / screw up. Just look at literally every undergrad in any lab and the expensive messes they create.
Thank you for taking this seriously enough to write this, and anyone else likewise.
But this is attacking a strawman, amateur bioterrorists. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.
Being at risk of killing yourself also just makes success X% less likely, but if X < 90 that doesn't mitigate much.
It sounds like your claim is "AI is a force multiplier, therefore it is dangerous and must be regulated." Not trying to strawman - I read your comment twice and I think that's what you're saying.
By that logic Excel, the internet, calculators, and air conditioning are all force multipliers in that all of the make it easier for an expert who can make bioweapons able to do so more quickly / comfortably / easily. Obviously it wouldn't make sense to "pace" any of these technologies. Even social media or email could be considered force multiplier in that they could literally allow you to increase the number of collaborators working on supposed bioweapon. Credit cards and modern logistics force multiply your ability to purchase precursors.
Also it would be a force multiplier on the prevention/mitigation side as well. Police armed with AI to scan for suspicious precursor purchases or review security camera footage after the fact would be far more effective than before. It's not obvious that the force multiplication is greater to the bad actor or to those who oppose them.
I want to ask you a question, do you feel like that this is the problem being solved by the current "safety features?"
Let me rephrase my question, do Anthropic and OpenAI implement the same "guardrails" and "safety features" for the NSA? Does the Mythos NSA Edition™ have these restrictions? What about the one that's running as a part of Maven?
How does stopping me from asking about rabbit sex protect you from nut jobs with a security clearance using these systems to go off and make weapons?
Because this has happened before. The only successful bioweapons attack in US history was done by a guy who worked at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks
The only private organization successful enough to make biological and chemical weapons is Aum Shinrikyo and they had university affiliation and nearly a billion dollars in the 1990s.
The guy who led Aum's bioweapons program was
and, https://irp.fas.org/congress/1995_rpt/aum/part04.htmand,
and, Why didn't anyone ask questions about a known cult importing an absurd amount of precursors, industrial-grade HEPA filters, fermenters, and lab equipment?It's because they'd recruited members of the military, bribed local policemen and politicians, and used their sway to silence critics (or kill them).
How does stopping me from asking Claude about rabbit sex stop people like them?
From where I'm standing, Anthropic and OpenAI would have sold Aum a subscription.
The Fort Detrick guy would have access via the US Government and Aum had university affiliation. I am yet to read a serious proposal that actually deals with these risks and the other real risks of this technology.
Are they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI.
If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?
What if "way short of AGI" is still dangerous in ways we can't easily handle?
i think part of the problem, though, is that we've already hit many of the warning signs that AGI is coming, and people have kept moving the goalposts. if you were to tell someone from ten years that there's an ai that has passed the turing test and has made novel mathematical discoveries, i think many would say AGI is already here.
The old observation that people tend to define AI as whatever computers can't do yet is as true as ever. It's getting a bit absurd, moving from demanding "general intelligence" to replicating human cognitive phenomenology (the experience of cognitive activities). Yet LLMs can already somewhat (confabulation-prone) introspect their own "internal unspoken thoughts" in their residual streams, quite fascinating.
You're right, in that we would have said that.
I wouldn't frame it as moving the goalposts in a bad way, though. We understand more now than we did then, so we have a better idea of what the actual goal is, and moving the goalposts to frame that goal is more rational than insisting the goalposts stay where they were ten years ago even though the entire pitch has changed since then. I'm straining the analogy, obviously.
Moving the goalposts, so to speak, is how science works! We must of course update the things we think based on an improved understanding of how the world works. Only the deeply incurious could consistently demand that one adhere dogmatically to a prior way of understanding the world in the face of changing knowledge. It's important not to treat science as a game to be won (of which 'goalposts' are evocative), but as the collaborative effort that it is.
Note well that I have not said LLMs are not useful, only that no amount of interaction with them has convinced me they remotely approach general intelligence. Their actual function of heuristically regurgitating all information ever known to mankind remains extremely useful in lots of domains.
And by the way, it's not obvious an LLM could fool me (or the average person) over a sufficiently long period of time, exactly because they are heuristic machines. That they lack thought guided by underlying cognitive models seems to always leak out, in the end. I suspect pass rates by LLMs on long-range Turing tests (if there have been any conducted) might drop as people become acclimatized to them.
what do you mean by 'general intelligence'? i definitely agree that in comparison to humans, they have some definite weaknesses. however, i think they also have definite strengths. and i don't think their current weaknesses imply that much about a potential singularity or ai risk.
Sorry, I should've used clearer language. I just mean intelligence in the sense humans possess it. Computers have always been able to exceed limited elements of human intelligence (say, at arithmetic), but the problem is to simultaneously match or exceed all of them. I specifically think that the important parts they currently lack make them a no-go for supposed existential risk.
If that’s truly the case, why aren’t they offering to be absorbed by national labs. That’s what we do for dangerous biological and nuclear work.
Oh wait they want options and shares and to get rich.
And there’s no natural moat. Which Federal regulation would bring.
That’s why people think it’s disingenuous.
We're commenting on a site run by a Silicon Valley tech start-up accelerator. "Move fast and break things" is the accelerationist way, even if the "things" they break are a stable, functioning social order. As long as you IPO and cash out along the way, you can wreck everyone else's lives just fine, in the minds of this sort.
Every single technology in history has had people claiming potential existential risks that didn’t materialize. Burden of proof is on people making the risk claims. Those have to be specific and unique to that technology to grant special actions.
i'm not trying to be rude, but i really don't get what you mean by this? (https://www.amazon.com/Superintelligence-Dangers-Strategies-...) there's been people making specific and unique risks to this technologies for years, before AI was even present. anthropic was literally founded because they were worried about existential risk? also, previous technologies have had existential risks? do you deny that nuclear weapons could have killed humanity? obviously, there have no been existential risks that have killed all of humanity, because by definition, that can only happen once. it will literally be impossible for someone to ever point to an example of that happening, because they will be dead by the time that happens.
See printing press and internet!
And video games, democracy, novels, TV, cars, photography, cryptography, the Beatles… pick your favorite and read commentary / opinions of the time. Many thought that came with unique risks never seen before and must be stopped!
How many of those concerns came from the people building the technologies? John Carmack wasn't up in arms about how Doom was going to destroy society. Charles Dickens didn't think novels rotted the brain. Johannes Gutenberg wasn't concerned about the effects of a free press.
None of the previous concerns were true and nothing indicates current ones are. It’s extrapolation, hypothetical apocalyptic imagined scenarios and piles of assumptions on how the technology could evolve. What makes even less credible in the AI case is that there’s financial / power to be gained by those making the claims.
And to your point. Don Quixote, the first modern and greatest novel of all times, is about a guy that lost touch with reality from reading too many books
None of the previous concerns were true, largely because they came from laypeople who didn't understand the technology. A prominent exception is nuclear weapons, where their creators did worry their creations could end up destroying the world. I think most people would agree those concerns were justified, and it's not any lack of capability that means we're still here.
The motivations aren't unique here. Restrictions on the development of almost any technology benefits incumbents by providing a moat.
Employees inside frontier labs have the chance to make life changing money if they can make it to an IPO before the music stops. Have you considered that might be the reason they are so eager to claim "ai dangerous - regulate ai"? The nuclear scientists at los alamos did not have generational wealth incentivizing them to raise "concerns" to the public.
If your only evidence that the claim "ai dangerous - regulate ai" is true is "insiders who stand to make billions if they can regulate away the competition" told me so, that is pretty weak evidence indeed.
If a group of concerned scientists (without $$$ in stock options) demonstrated that anyone with one year of HS chemistry, an Amazon account, and a ChatGPT max plan could build a bioweapon 100x more lethal than an AR15 mass shooting, then that would be worrisome. This is really not that high of a bar but even it has not been met, but oh no we should start regulating yesterday.
I mean, Cervantes was parodying medieval romances. That's what Don Quixote was reading, not modern novels.
The point is, it should give you at least some pause when the people most familiar with a technology are among those making the strongest statements about its potential dangers. Extrapolation has worked extremely well in predicting what AI will be capable of over the past 5 years; how confident are you that it will fail before we reach apocalyptic scenarios?
This is serious. We're in uncharted territory. The geopolitical context is already complicated enough that we shouldn't be dismissive of such a rapid technological race. I'm glad AI leaders are taking it seriously. I'm surprised so many HN commenters aren't.
On the other hand, I'm pessimistic about initiatives like this being effective on a global scale (though they could certainly work at the national level).
we're always in uncharted territory. that's why most HN commenters think that this is hyperbole.
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest
Pointing out the endless stream of lies and self serving half truths these terrible people engage in helps the younger generation understand the world they were born into.
It's constructive and helpful. It's why I'm replying to you.
You interpreting it as 'shallow dismissal' is shallow dismissal on your end. You're the shallow dismisser, not the people calling terrible people terrible.
As for your little petition. It's as good as eliminating plastic straws to save the environment.
Terrible people with power they don't deserve doing pathetic little, tiny 'good' deeds to sleep better at night needs no explaining.
What happens when a country doesn't agree to play along with this little decel game?
The genie is already out of the bottle and now a bunch of people who helped release it think that the collective powers of the world are just going to put it back in.
How far are the saftyists willing to go? Bombing data centers? Starting wars? Embargos?
It's a terrible dilemma to be in. If the threats are real, it gives all the more incentive for some actor to race and try to grab it first. If not, then any punishment isn't proportional. Explain to a nations citizens under a 10 year embargo, "sorry about your quality of life, we all just got a little spooked by science fiction for a second."
Folks try to reach for bad analogies: nuclear bombs and biological weapons. But conveniently omit that by the time the regulations took place for these things we had unequivocal proof of their destruction, scope and scale.
So, what then? We're supposed to collectively stop innovating and put the keys to the kingdom in the hands of a few capitalists, a corrupt is administration and the ccp?
I reject that these agreements should take place, on the grounds that the proposal is delusional and not founded in reality.
On the geopolitics, the American domiciled firms are completely in bed with the US military and US government, and have made it absolutely clear that they are committed to the domination of the US government. So when you say "geopolitical destabilization", you mean "only the US government is allowed to destabilize, which they have done for decades and wish to continue doing unfettered."
And given what we have seen from that government -- piracy, terrorism, war-mongering, thievery, and just profound stupidity -- no thanks, in that case I'd like to see every American AI company utterly obliterated by any other country. If China becomes utterly dominate in AI, awesome.
"Many were increasingly of the opinion that they'd all made a big mistake in coming down from the trees in the first place. And some said that even the trees had been a bad move, and that no one should ever have left the oceans."
The historic analog most befitting AI are weapons. It provides an advantage to companies and governments in creating weapons, in surveilling people and thus gaining leverage over individuals, in controlling what individuals can do or cannot stop doing. It thus follows the laws of weapons control - the more restrictions are threatened to be imposed, the more even the smallest actors will fight to get more of it. They may not understand or know what they are getting, but they will want more of it because their potential rivals have it. Governments are the ultimate customer here - they have the money, the energy, and the appetite to consume AI infinitely in battlement against opponents - foreign and domestic. The only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment (the upside of that is it advances the AI for future use and is thus in the interest of governments as well). Modern large models seem to be compressible and thus we may see more and more tiny sub-specialized varieties. But the push to slow the frontier in weaponry always leads to the reverse.
Anyone serious about this should participate in pacing the frontier by quitting their job so as to not potentially contribute to the problematic progress. Be the change you want to see!
Curious how that isn't what happens.
Some of us considered seeking one of these jobs and decided not to. Some have quit (e.g. Hinton, OpenAI people after OpenAI reneged on safety commitments). Now you're getting this message from people who passed the previous filters. If you think a brake is indicated and not just easing up on the accelerator, you need collective action as the statement said.
> you need collective action
Come on, go on strike then! Have a week of protest and picket the offices in SFO. Let's see what happens.
(I'm aware that's not quite what you meant, but it is the classic meaning of that phrase).
That's an application of the "Tragedy of the commons": One person stopping to push the frontier just means that this person will lose their job, the frontier will be pushed by the others. The solution is to push for a binding rule instead that will make everyone slow down in the racing towards artificial superintelligence.
And this is how we absolve people of all personal responsibility. Such companies would not pay people that well if they were so fungible.
There is no such possible binding rule either, a point that is so obvious that any smart person claiming otherwise is clearly up to something.
Influencing change is more complicated than that.
We live in a society that has systems of market competition between companies and economic/military competition between countries. There's not just one country working on frontier AI, and not just a few companies in each country either. The incentives of these systems ensure AI advancement will happen quickly from here on out, no matter what, absent governmental action.
An individual researcher leaving the company does nothing, other than give their seat to someone else who likely doesn't share their same concerns.
Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
> Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
No, it just comes across as "I want to get paid as if I was doing the thing I am employed to do by threatening you all about how terrible it would be if I actually did it".
They aren't even being semi imaginative and going on strike or anything.
We make binding rules all the time. We didn't fix the ozone layer by encouraging people to quit their jobs producing CFCs. We lobbied for international regulation to ban them.
In your example there is nothing to be gained by not complying.
Failing to comply with what is in reality arms control is a widespread occurence. It's notable the only historical way to handle this (Schelling covered it at length) is tolerance of mutual espionage on a level that would simply not be tolerable to any of the groups involved.
I should add one of Schelling's other pet subjects was "time to weapon being viable". i.e. the distinction between a nuclear power and non-nuclear power isn't quite right, as there's a huge class of countries that could become so fairly fast if they were so inclined, and that this situation is somewhat inevitable. It is this that leads to efforts to denuclearize, even on a civilian level, "problematic" countries. You want to stop frontier AI and you'll have to be rolling back a lot of existing deployed technology.
Great comment. I haven't read Schelling but will check that out. I think you're right that international cooperation on the topic is likely impossible at present, for the reasons you stated. I'm not sure what it would take to shift the Overton window, but it's probably not pretty.
So honestly Schelling was utterly brilliant. I cannot recommend all three of the books wikipedia classed as his notable works highly enough:
https://en.wikipedia.org/wiki/Thomas_Schelling#Notable_works
There are footnotes in "The Strategy of Conflict" containing more profound insight than whole books that are published today.
On this particular topic though "Strategy and Arms Control" is the one to go for.
Hoping everyone chooses to do the right thing is not an effective strategy for mitigating bad things.
Personal responsibility is a difficult concept for be to apply to massive efforts like this. When there’s one person and you can talk to them personal responsibility is the number one topic. When you’re talking about a multi-trillion dollar bet and hundreds of thousands of people it doesn’t make sense anymore.
Same logic goes for other common issues where personal responsibility is lauded. Obesity - real solutions are policy and medicine. Abortion - education and contraceptives.
“It is difficult to get a man to understand something, when his salary depends on his not understanding it”
Can someone help clarify what it means, concretely, to make binding agreements to proceed "safely and slower" in the context of AI research (whether or not at frontier labs)? Are there specific policies and actionable guides being proposed?
It's an admirable idea, but I don't see how you can slow this down in practice. We certainly know China will not, and they are already at the frontier.
RSI is too important and too tangible at this point for everyone to slow down. It's the equivalent of telling the world to stop developing nuclear weapons after Hiroshima. Just not going to happen, and in the meantime dramatically increases the risk that the power of AGI lands in the hands of the few (read: the political class) rather than empowers the people.
This. Will. Never. Work.
This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow.
The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect.
The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection.
Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology.
Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals.
Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.
Why not? Slowing down / halting biological weapons research mostly worked. Yes there have probably been modest sized defections here and there, but the pace of bioweapon development is a crawl compared with (a) what is possible with science already, and also with (b) the pace of bioweapons development from 1910 to 1970.
Bioweapons don't make money. AI does. The two are therefore not comparable.
The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.
Some reason this kind of argument always by default assumes the US will be the good guys.
Something people don't seem to see, and not that it's placed directly in front of you, is that pacing the release and adoption of AI is already happening.
Cost-per-token determines the economic viability of tasks that AI gets applied to. How many tokens would it take to automate all of the responsibilities of a gas station clerk? Probably more than their hourly wage. Job safe.
A slightly larger stretch, that most people attribute to profit margins (labs do not have profit margins, they have unlimited invested capital and no expectation of short-term returns), is the flat-rate billing plans of most code products. Why doesn't Anthropic have a 50x or 100x Max plan? Not that it's impossible, that it releases tokens into the workforce at a greater rate (note that their are hour, day, week limits across most of these services as well).
Hypothetical, but reasoned, and I do not expect most agree with me. If compute as a variable is inferred, this supports the incorporation a deliberate (redundant) one as well.
I agree with the substance of your comment that token economics pace the diffusion of AI into the labor market. But I think they have a much smaller influence over the pace of the frontier, right?
Fair point! I guess I read "pacing the frontier" as "pacing the release of the frontier" surely no lab's internal research team is just going to stop innovating?
> We request that the U.S. government ...
Too bad these researcher's organizations spent their efforts supporting the dumbest, ficklest possible version of the U.S. government to guide us through the gravity of this moment.
> We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
This is quite vague to me, what exactly do you mean?
It means that they request the structures needed to navigate the event horizon of recursive self improvement be put into place. Probably: International agreements, verifiability of an AI pause on the level of graphic cards as well as through physical inspections of datacenters. Like it's done with nuclear energy.
It means only those vetted will be allowed to use frontier models (i.e. let's pace ourselves and not share the frontier broadly)
'Regulatory Capture' if we were cynical.
I think a lot of these guys mean it though.
'The Fear' - is not some magic intelligence will come alive and be Skynet, these people are ridiculous.
People will use the compute power to do nasty things, we'll be overwhelmed with bots, regular companies will use them for dark patterns, political actors will use them to cheat.
It will be 100% humans using it for bad stuff, and in not very novel ways.
What these people want is same thing people in every other field want too, which is being able to use AI as a tool and nothing more. Off course, some of the people that signed this might not recognize this, and the people who didn't sign it are actively working against such an equilibrium.
The first computers were used for war.
The Harvard Mark I was used by von Neumann to calculate the physics of implosion. And then right afterwards, ENIAC was used to calculate the feasibility of the H-bomb. They called it the "Los Alamos problem."
The first computer network, SAGE (or, Semi-Automatic Ground Environment) brought together RADAR data from across the US with air-traffic control data and military flight data to detect overflight by the USSR. It could direct and control fighters to intercept those planes.
If you were sitting there, in 1955, would you have believed that computers and their networks would bring about an extraordinary age?
Imagine if von Neumann and his ilk started campaigning in 1955 to limit the development and use of computers, because surely they can only be used to make weapons, right? Right?
We were lucky to have visionary people like Joe Licklider who could look past the present and see the future. In 1963, he wrote a memo calling for an "Intergalactic Computer Network" (because he knew engineers under-delivered. So if he asked for a galaxy, at least he'd get a planet). And he gave these machines to scientists and people like the original hackers at MIT to play with.
What if Licklider had written memos about how we had to "pace" computers to make sure that other countries wouldn't be able to develop implosion technologies or systems like SAGE? What then?
Do you think you would have the jobs and the prosperity you do today?
If instead of giving away this technology, spreading it, and putting down the seeds for ARPA-net, Licklider had written memos from "concerned scientists" (and I'm sure he could have found a bunch) talking about how computers can only be used for weapons and war! So stop the proliferation of computers! We can't imagine a world with computers small enough to fit inside everything! Even wall sockets! What if your wall socket could do war?!
They didn't, because they knew there's no better way to kill a better future for all mankind than to control it.
† yes, peace. We are more peaceful than we have ever been. Humans can be a violent lot even when reluctant to kill.
We were incredibly lucky that computing and nuclear weapons led to a stable geopolitical equilibrium (mutually assured destruction). There were certainly a few scary points where we were still settling into that equilibrium during the Cold War.
I don't have a strong opinion either way, but it seems very hard to predict. Why do you think AI advancements will lead to a stable equilibrium?
Equilibrium for whom? Equilibrium for what?
Here's what I know. Every day, I read a 5 to 10 incoming emails from young people in rural Africa, Asia, Europe and (heck) the US. People who have had no other resources and options, but are tremendously gifted and somehow got their hands on an internet connected device (usually a very cheap smartphone and/or cheap old computer).
These people, who are usually teenagers, end up getting connected to Claude, who then gives them a list of things to learn, people to contact, and helps them make things. And they actually do. They make robots. They learn new things and try to improve themselves and others. And some want to change the world / do groundbreaking research.
When you sit down and start pricing and doling out AI from behind regulatory gates, you shut those people out.
There are a lot of smart people in this world who have traditionally been unable to get access to resources and education. MOOCs didn't do much / help. But these AIs? They were born multi-lingual. They can talk in any language. They can tailor their responses to their audience. And they can help broaden these people's horizons.
There are no tutors around them. There is no one who is coming to help. But an AI that can run off of a small battery and a connection to a cellular network can give them access to genuine intelligence to make their world better.
I met one sixteen year old who was accessing Claude surreptitiously (by begging people) and was trying to apply some really fascinating biomimetic fluid dynamics observations to aerospace. This person was born in one of the more under-resourced parts of the world. And Claude gave them a list of people to reach out to who could help.
If you want a golden age for humanity, you should be asking for Licklider's Man-Machine Symbiosis.
A lot of atomic scientists did regret their work.
Yes, and did you see them oppose computing? They promoted it and tried to use nuclear energy for peace.
I think this is a spot on analogy and would two more: the hand wringing over the release of the printing press and the internet. In both cases the people in power were terrified that the masses would use their newfound power to do dangerous things. In both cases bad things did happen (market bubbles, scams, witch hunts, spread of knowledge on making poisons), but trying to put the cat in the bag by pacing the printing press or the internet would have been a fool's errand, and over the long run both inventions massively improved the lives of society overall.
Pacing is a really bad idea because China will not slow it down for you.
The entire point of the statement is to start international coordination.
The problem is how do you enforce coordination? This isn't like nuclear testing that can be detected around the world on seismographs or from satellites in space.
AI research happens in benign looking buildings that could be doing a hundred different non-AI workloads, much of it things that industry or governments would not want 3rd parties prying into. There is too much to potentially gain to trust coordination.
There is a lot of research out there with answers to your questions. GPU supply chains are extremely constrained, making it a surprisingly tractable problem at least for now.
How do you tell the difference between inference and training workloads from outside? The same GPUs can be used for both and the pacing suggested here relates to training and not inference as far as I can tell.
It will never happen because there is money to be made by not coordinating, i.e. by leaping ahead. It also doesn't make any sense to hold AI back.
That's the point. Do you see any concerns by these companies? They just want to slow down pace NOW because they're losing the race
All they have to do to stay in the race is to keep the price and token efficiency competitive with China.
Huh, that's really mental gymnastics you have there :-)
> losing the race
Lol, lmao even
The website does not list all the comments but just around ~50 without any way to load all the ~+1100 more (1132, as of 2026-07-28 2100 UTC).
The number of em-dashes is questionable, existing in many, if not all, the comments.
I am sorry, but I am not sure whether it's adequate and fair to allow join such statements/petitions to only those who live in it. Isn't it like allow deciding the future for weaponry laws to weapon and war enthusiasts only?And, this is not serious at this point:
> > We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
> 1122 employees of frontier AI companies
'lets us be the gatekeepers of AI because we know better than you'
access to gpu doesnt equal to any special insights to predict future of ai. Everyone and their grandma with enough compute is putting out a model now.
I'm chinapilled now.
Anthropic is an evil, enemy of humanity company that cannot be allowed to succeed.
Yes. Which Chinese labs and models do you like the most?
> It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.
This is just the standard AI "doomer" talking point. The dangers of AI just used as a compliment sandwich. "AI is so cool and awesome and that's why we need to slow down."
The real risks are much more boring and much less considered. For just one example:
AI is set to completely upset the Software Development industry. There are 1 to 3 million software developers in the US, depending on how you draw the line exactly. A low-few hundred thousand of those are visa workers.
With AI set to significantly cut into tech employment, what's going to happen to those visa workers? ... The Trump admin is not particularly subtle, even openly talking about retroactively revoking birthright citizenship.
AI & Tech companies love to talk about how their AI is going to hack everything if they're not careful. But not a peep about these ethics concerns.
Even if you [the reader] consider deporting visa workers the ethical move if the amount of jobs shrink, that's not something to be done unprepared or out of nowhere.
They just want some time to be able to sell the stock
Has there ever been a case in human history when technology just stopped advancing, simultaneously world wide, only because a bunch of people wanted it to?
I think these people are either incredibly naive, or, more likely, not acting in good faith.
https://en.wikipedia.org/wiki/Asilomar_Conference_on_Recombi...
> The effects of these guidelines are still being felt through the biotechnology industry and the participation of the general public in scientific discourse. Due to potential safety hazards, scientists worldwide had halted experiments using recombinant DNA technology, which entailed combining DNAs from different organisms. After the establishment of the guidelines during the conference, scientists continued with their research, which increased fundamental knowledge about biology and the public's interest in biomedical research.
DNA splicing, human cloning, super-sonic passenger flight, human gene editing (CRISPR), nuclear energy, biological weapons, and probably more.