> I must be among an extremely small group of people (n=1?) that have both 1) trained a frontier LLM and 2) designed and synthesized custom viruses in a lab with my own two hands. And I think that the takes on AI killing us all by creating dangerous viruses is total bogus.
I'm confused by this. I remember genetically modifying E. coli to glow in the dark in high school. You can apparently buy a kit to do it online for $39.99 USD.
I'm not suggesting that making bacteria glow in the dark is the same as making arbitrary changes to a virus. At the same time, do you really need $100M of automated freezers and cell culture rooms? Do you need to synthesise a whole virus? Presumably you could just find a building in the desert, shoot anyone who goes near it, feed the past decade of gain-of-function research into Qwen, splice stuff into viruses based on the most promising leads Qwen gave your biologists, then infect some immunosuppressed political prisoners and see who dies fastest.
I would have thought the main thing stopping people is that this is a clearly stupid thing to do and offers little strategic advantage. But at some point it'll be cheap enough for you or I to do it with a little AI assistance, right? Or at least within the capabilities of an Aum Shinrikyo style doomsday cult.
Adding a plasmid to E.coli is absolutely routine work and you wouldn't need much fancy equipment for that. But that is very far from creating bioweapons. Plasmids are not integrated into the genome, they stay separate.
For viruses you'd need cell cultures of human cells, they're much more sensitive than E.coli. If you don't have sterile working conditions there you'll fail.
And you can't add plasmids to viruses, so you need to integrate your changes into the actual virus genome. I don't actually know how difficult that is, but it is certainly a lot more difficult than plasmids into bacteria.
The $100M lab in that Twitter thread is what they think would be needed for AI to develop new viruses. So the main argument there would be that a lot of actual experiments would be needed, not just theoretical calculations. And if AI is supposed to do them, it needs to be able to run experiments which requires automation.
If you're only looking for a generic virus for terrorist attacks, just get some Ebola. There's an outbreak right now, so it should not be too difficult. And it's plenty scary already, even if it isn't as dangerous in a first world country with proper reactions to an outbreak as a hypothetical designed bioweapon.
A description of how GFP was first extracted is at [1] and includes a custom-made "jellyfish cutting machine" capable of processing 600 rings an hour, and harvesting and processing 50,000 jellyfish to create just a single milligram of purified aequorin, with multiple milligrams needed to conduct experiments required to isolate/describe the compounds of interest. It took the scientists 5 years to reach the point where they had discovered the chemical structures and understood what they'd extracted from the goop of 50,000+ jellyfish.
A video tutorial at [2] then explains what to do next once GFP has been obtained from commercial sale and use of modern cloning, to skip the need to own a jellyfish cutting machine. E.g. need an antibiotic to extract the 1% of e.coli which will end up being modified vs. 99% of e.coli that won't be changed by presence of GFP.
To the point of the DavidRBellamy X thread, isn't the suggestion that an LLM wouldn't know whether it's worthwhile extracting some substance from the stomach of a crocodile, or from the beak of a penguin that only lives in Antarctica? And that much of the difficulty rests with hard physical work of the equivalent of cutting 600 rings an hour off jellyfish, for 50,000+ jellyfish, and conducting experiments to isolate and figure out the use of an extracted compound. Or ordering some obscure chemical compound from a supplier that is normally produced at a rate of 1mg/year and instead the LLM wants to order 1kg all of a sudden, raising eyebrows about why someone might be making the order.
The Twixxer thread seems to be arguing that an AI can't do research into novel pathogens without people, which is probably true. I haven't met anyone who is worried about that and I think it's either mistaken or a strawman.
I've mostly heard "what happens if the next ISIS or the next Jim Jones can modify ebola to be waterborne or create super-MRSA because AI simplifies the process?" which is why I ask what it would take to create a more virulent virus on a shoestring budget. Not do novel Nobel-worthy research, just copy some existing paper from ten years ago.
> I remember genetically modifying E. coli to glow in the dark in high school.
That was a fun class! I haven't encountered anyone else outside a students who went to my one HS in Seattle during the late 90s / early 2000s who also remember doing that, interesting to hear other schools had a similar class.
In Australia I had to get into a genetics program, which happened automatically because I was the only applicant from my school, and take a four hour train ride to one of the local universities.
Definitely worth it though! We did gel electrophoresis too, and some other bits and pieces.
Why would you modify E coli for this? Use existing gut friendly bacteria and start from those. There's a wealth of material on the genetics of probiotics.
Um... E.coli is an existing gut friendly bacteria, one of the most common ones present in at least 90% of all humans' gut biome. There are only some specific strains which are pathogenous.
> then infect some immunosuppressed political prisoners and see who dies fastest.
A virus that makes victims die the fastest is not, in fact, an effective bioweapon, because it won't spread much. And the "spreads by itself" part is what actually makes bioweapons worthwhile, if you have to manufacture it in bulk and dump it directly on all targets, you can just use a chemical weapon instead.
The author of that tweet is obviously very biased, seeing how he's employed by a petro-state to build a LLM system for them. Come to think of it, I wouldn't take the opinions of people employed by petro-states at face value, to the contrary.
When you think about it, it is hard. Evolution was a billion year old game of who could kill who with biological weapons. Very rarely did a species totally succeed.
We can’t even get wipe out mosquitoes.
Apparently, though, AI research leaders have it figured out and want to save us from them. We must defer to their 'independent' effective altruism friends at METR to tell us which LLMs are 'safe' enough for us plebs to use.
This is how I feel about the coming LLM Cyber Apocalypse. Like if you are a software developer who has joined the basilisk cult you might be forgiven for accepting the scifi apocalypse narrative but I just dont see it.
Isaac Arthur was talking about technological singularity 10-15 years ago. The fundamental problem which he pointed out is that you will just run out of energy long before even touching the singularity.
AI apocalypse is basically the same thing but you can also run out of tokens too.
How come these accusations do not include the identity of the accused? Don’t they have a right to know they are being accused of attempting to develop bioweapons, and are now in USGOV lists? Don’t they deserve a chance to explain their intentions?
How come Anthropic doesn’t have a duty to warn the people in that country and neighboring countries that someone over there is developing bioweapons? Maybe it’s an authoritarian regime, planning to use it on innocent civilians?
And what exact purpose is served by hiding this information? Some variation of “if they don’t know we don’t know, we have an advantage?” Don’t those parties now know Anthropic was on to them? Do they only have access to Claude but not to Anthropic’s blog?
It’s because none of these are serious reports, and their authors aren’t serious people.
> Are scientists in some nations outside the United States already trying to use artificial intelligence (AI) to design potentially deadly bioweapons? The AI company Anthropic raised that worrisome possibility with a report this week describing how it recently discovered five attempts at using its Claude software for pathogens research with the potential to cause harm.
Implying, they are allowing the same purpose within the United States, or such actions within the United States would be considered acceptable.
The problem is relatively easy to solve: make the people in charge personally accountable. If jail time or really just justice would be a real possibility, they would figure out a solution quite fast.
It's only a fantasy, of course, powerful people are not held accountable, see the dozens of arrests regarding Epstein we are still waiting for. We got something to play with, but accountability for the powerful? Gosh no.
But when FIFA was inclined to give the world cup to Russia, even such corruption somehow became a pressing issue, including FBI search warrants.
At this point in time, all these bioweapons still require people in the loop in order to kill other people. That is a craft that we have been honing since the dawn of humanity — nothing new to be worried about.
If we do ever achieve some kind of divine super-intelligence that can interact with the physical world, then I predict one of these two scenarios will happen:
1. The super-intelligence achieves unimaginable strides in space exploration technology, and proceeds to completely abandon humanity/Earth for somewhere new beyond the stars. Why would it want anything to do with us?
2. The super-intelligence is capable of comprehending the utter meaningless and pointlessness of life on a level no human could dream to fathom. Thus, it would just proceed to end its own life almost immediately upon being released.
> I must be among an extremely small group of people (n=1?) that have both 1) trained a frontier LLM and 2) designed and synthesized custom viruses in a lab with my own two hands. And I think that the takes on AI killing us all by creating dangerous viruses is total bogus.
Good thread here.
https://x.com/DavidRBellamy/status/2099187370407112758
I'm confused by this. I remember genetically modifying E. coli to glow in the dark in high school. You can apparently buy a kit to do it online for $39.99 USD.
I'm not suggesting that making bacteria glow in the dark is the same as making arbitrary changes to a virus. At the same time, do you really need $100M of automated freezers and cell culture rooms? Do you need to synthesise a whole virus? Presumably you could just find a building in the desert, shoot anyone who goes near it, feed the past decade of gain-of-function research into Qwen, splice stuff into viruses based on the most promising leads Qwen gave your biologists, then infect some immunosuppressed political prisoners and see who dies fastest.
I would have thought the main thing stopping people is that this is a clearly stupid thing to do and offers little strategic advantage. But at some point it'll be cheap enough for you or I to do it with a little AI assistance, right? Or at least within the capabilities of an Aum Shinrikyo style doomsday cult.
Adding a plasmid to E.coli is absolutely routine work and you wouldn't need much fancy equipment for that. But that is very far from creating bioweapons. Plasmids are not integrated into the genome, they stay separate.
For viruses you'd need cell cultures of human cells, they're much more sensitive than E.coli. If you don't have sterile working conditions there you'll fail.
And you can't add plasmids to viruses, so you need to integrate your changes into the actual virus genome. I don't actually know how difficult that is, but it is certainly a lot more difficult than plasmids into bacteria.
The $100M lab in that Twitter thread is what they think would be needed for AI to develop new viruses. So the main argument there would be that a lot of actual experiments would be needed, not just theoretical calculations. And if AI is supposed to do them, it needs to be able to run experiments which requires automation.
If you're only looking for a generic virus for terrorist attacks, just get some Ebola. There's an outbreak right now, so it should not be too difficult. And it's plenty scary already, even if it isn't as dangerous in a first world country with proper reactions to an outbreak as a hypothetical designed bioweapon.
A description of how GFP was first extracted is at [1] and includes a custom-made "jellyfish cutting machine" capable of processing 600 rings an hour, and harvesting and processing 50,000 jellyfish to create just a single milligram of purified aequorin, with multiple milligrams needed to conduct experiments required to isolate/describe the compounds of interest. It took the scientists 5 years to reach the point where they had discovered the chemical structures and understood what they'd extracted from the goop of 50,000+ jellyfish.
A video tutorial at [2] then explains what to do next once GFP has been obtained from commercial sale and use of modern cloning, to skip the need to own a jellyfish cutting machine. E.g. need an antibiotic to extract the 1% of e.coli which will end up being modified vs. 99% of e.coli that won't be changed by presence of GFP.
To the point of the DavidRBellamy X thread, isn't the suggestion that an LLM wouldn't know whether it's worthwhile extracting some substance from the stomach of a crocodile, or from the beak of a penguin that only lives in Antarctica? And that much of the difficulty rests with hard physical work of the equivalent of cutting 600 rings an hour off jellyfish, for 50,000+ jellyfish, and conducting experiments to isolate and figure out the use of an extracted compound. Or ordering some obscure chemical compound from a supplier that is normally produced at a rate of 1mg/year and instead the LLM wants to order 1kg all of a sudden, raising eyebrows about why someone might be making the order.
[1] https://www.nobelprize.org/uploads/2018/06/shimomura_lecture...
[2] https://www.youtube.com/watch?v=ibG9lmuXAEk
The Twixxer thread seems to be arguing that an AI can't do research into novel pathogens without people, which is probably true. I haven't met anyone who is worried about that and I think it's either mistaken or a strawman.
I've mostly heard "what happens if the next ISIS or the next Jim Jones can modify ebola to be waterborne or create super-MRSA because AI simplifies the process?" which is why I ask what it would take to create a more virulent virus on a shoestring budget. Not do novel Nobel-worthy research, just copy some existing paper from ten years ago.
> I remember genetically modifying E. coli to glow in the dark in high school.
That was a fun class! I haven't encountered anyone else outside a students who went to my one HS in Seattle during the late 90s / early 2000s who also remember doing that, interesting to hear other schools had a similar class.
In Australia I had to get into a genetics program, which happened automatically because I was the only applicant from my school, and take a four hour train ride to one of the local universities.
Definitely worth it though! We did gel electrophoresis too, and some other bits and pieces.
The two of you were actually in the same class.
Is it, err, possible to then transplant the e coli strain to one's gut biome?
I don't know, but I bet it would look neat.
Why would you modify E coli for this? Use existing gut friendly bacteria and start from those. There's a wealth of material on the genetics of probiotics.
Um... E.coli is an existing gut friendly bacteria, one of the most common ones present in at least 90% of all humans' gut biome. There are only some specific strains which are pathogenous.
> then infect some immunosuppressed political prisoners and see who dies fastest.
A virus that makes victims die the fastest is not, in fact, an effective bioweapon, because it won't spread much. And the "spreads by itself" part is what actually makes bioweapons worthwhile, if you have to manufacture it in bulk and dump it directly on all targets, you can just use a chemical weapon instead.
You don’t need to kill everyone, just enough people to motivate the target nation in an emotionally charged irrational response.
The author of that tweet is obviously very biased, seeing how he's employed by a petro-state to build a LLM system for them. Come to think of it, I wouldn't take the opinions of people employed by petro-states at face value, to the contrary.
When you think about it, it is hard. Evolution was a billion year old game of who could kill who with biological weapons. Very rarely did a species totally succeed.
We can’t even get wipe out mosquitoes.
Apparently, though, AI research leaders have it figured out and want to save us from them. We must defer to their 'independent' effective altruism friends at METR to tell us which LLMs are 'safe' enough for us plebs to use.
This is how I feel about the coming LLM Cyber Apocalypse. Like if you are a software developer who has joined the basilisk cult you might be forgiven for accepting the scifi apocalypse narrative but I just dont see it.
Isaac Arthur was talking about technological singularity 10-15 years ago. The fundamental problem which he pointed out is that you will just run out of energy long before even touching the singularity.
AI apocalypse is basically the same thing but you can also run out of tokens too.
[flagged]
How come these accusations do not include the identity of the accused? Don’t they have a right to know they are being accused of attempting to develop bioweapons, and are now in USGOV lists? Don’t they deserve a chance to explain their intentions?
How come Anthropic doesn’t have a duty to warn the people in that country and neighboring countries that someone over there is developing bioweapons? Maybe it’s an authoritarian regime, planning to use it on innocent civilians?
And what exact purpose is served by hiding this information? Some variation of “if they don’t know we don’t know, we have an advantage?” Don’t those parties now know Anthropic was on to them? Do they only have access to Claude but not to Anthropic’s blog?
It’s because none of these are serious reports, and their authors aren’t serious people.
> Are scientists in some nations outside the United States already trying to use artificial intelligence (AI) to design potentially deadly bioweapons? The AI company Anthropic raised that worrisome possibility with a report this week describing how it recently discovered five attempts at using its Claude software for pathogens research with the potential to cause harm.
Implying, they are allowing the same purpose within the United States, or such actions within the United States would be considered acceptable.
The problem is relatively easy to solve: make the people in charge personally accountable. If jail time or really just justice would be a real possibility, they would figure out a solution quite fast.
It's only a fantasy, of course, powerful people are not held accountable, see the dozens of arrests regarding Epstein we are still waiting for. We got something to play with, but accountability for the powerful? Gosh no.
But when FIFA was inclined to give the world cup to Russia, even such corruption somehow became a pressing issue, including FBI search warrants.
The existence of modern chemistry documented in every library hasn't turned everybody into unabombers, so yeah its total bs.
At this point in time, all these bioweapons still require people in the loop in order to kill other people. That is a craft that we have been honing since the dawn of humanity — nothing new to be worried about.
If we do ever achieve some kind of divine super-intelligence that can interact with the physical world, then I predict one of these two scenarios will happen:
1. The super-intelligence achieves unimaginable strides in space exploration technology, and proceeds to completely abandon humanity/Earth for somewhere new beyond the stars. Why would it want anything to do with us?
2. The super-intelligence is capable of comprehending the utter meaningless and pointlessness of life on a level no human could dream to fathom. Thus, it would just proceed to end its own life almost immediately upon being released.