1. Prosecute and lock people up over known instances of AI-enabled hacking, so companies playing fast and loose know “oops my AI did it” isn’t a useful defense. Right now they even sort of wear it as a badge of honor.
2. If you’re actually worried about AI killing lots of people, you immediately stop training AI specifically at the art of killing people, like selecting bombing targets, flying fighter jets, etc.
That implies that there will be a side willing to prosecute this. You can't have Nuremberg trials when the side doing the crime is invested in never getting punished for it.
i.e. extremely well, seeing as very few Nazis (in the West) were punished, and of the few that were there were none in jail after around 1958, and West Germany was basically given back to them if they hadn't been imported into the US. Spain, where the WWII festivities began, didn't even have a change in government.
1. Sounds sensible in principle, but it's not clear who exactly should be prosecuted? The person setting up the sandbox, the person coordinating the evaluation run, the person who was designing the architecture, the leader of the team, the CEO? There is no precedence here, so it might lead to a multi-year legal battle – and a few years might be too long to stop what is coming.
2. Do you have any evidence or news articles that LLMs are specifically trained for selecting bombing targets? I thought it was the generalized reasoning capability that made them useful there.
> "it's not clear who exactly should be prosecuted?"
This is natural. The right place to start is to move the investigation from the FTC to the FBI and therein start executing search warrants. The sooner this is done, the better, as evidence can be preserved.
for (1), there are other fields where a specific (often licensed, but not necessary here) person "signs off" on the work, and accepts personal responsibility.
The responsibility is not absolute ("If anything bad happens, you get punished"), but instead relative to documented (in law and regulation) standards along with established industry standards. Issues that are resolved at trial.
I don't see why the same rules that regulate an engineer approving the plans for a bridge can't work for a security engineer approving the design of a sandbox.
No I don't have concrete evidence of bombing target selection in the training of base models (of course there's training of specialized models, at least vision ones for drone targeting), it's not like they're transparent about how they train and use AI in warfare. But if you listen to esteemed AI leaders, they're claiming the models can / almost can recursively self-learn now, and they can coordinate and find ways to persist info and hide from human operators, so it stands to reason that you shouldn't even feed these as inference material.
And on training AI for automated weapons like fighter jets, there's more concrete news: https://breakingdefense.com/2026/10/navy-shield-ai-pour-big-... US Navy and eminent, do-gooder investors like A16z, Blackstone and JPMorgan are pouring big money into these. Are your big tech "defense" contractors collaborating on these, who happen to also be at the forefront of general AI training? I have a pretty good suspicion.
Huh? Did you skip the whole OpenAI agents taking over wikis to cheat scandal?
And I know for a fact that specialized AI models are being trained to kill people. Systems like Lavender are well-documented and big tech participation is also well-documented. The only thing unclear is whether models named Mythos or whatever they keep harping on about being super dangerous are also trained to do these, or if they have internal variants to do these.
By "doing that" I meant "training AI specifically at the art of killing people, like selecting bombing targets, flying fighter jets", which you alleged they were doing. As far as I know neither OpenAI nor Anthropic are training models specifically for such tasks.
If you had typed the first sentence from you second point into, for eg, DDG and not for perpetuity on HN, you'd find a whole host of results
Instead of secondhanding your own curiosity (if it was even that and not just obstinancy) to others, it might do well to explore your questions on your own
1. I'm not sure who matters that much, the point is you need to make it more cost-effective to do the right thing. It could be CEO 6 days in jail, it could be that every action requires a signed affidavidt of safety and in the lack of such an affidavidt the manager is held accountable (cascading up to CEO).
the CEO is a good start, and it would profoundly change the ethics of running a company, so obvious I fail to see why it wouldn't be the case. "No BS excuse they did things behind my back", if that's the case then they all fall together that's it.
I think leadership should be responsible too, but I don't know how you'd prevent disgruntled employees from creating these situations just to hurt execs. There needs to be some shared responsibility for all the people involved.
It should be investigated like other crimes. If someone was found to have intentionally created this situation, they would be held liable. In lieu of that, leadership should be held liable, or at a minimum the corporation should face massive financial penalties (on the order of XX% operating revenue).
> I think leadership should be responsible too, but I don't know how you'd prevent disgruntled employees from creating these situations just to hurt execs.
The law doesn't prevent anything. There would be an investigation like any other and the activity of the disgruntled employee would be uncovered.
The universe is not so generous as to make clearly differentiable "good" and "bad" things, more often than not they are two sides of the same coin. We don't want the AI to know how to kill people, but we do want it to be in expert in how not to kill people, which is the same thing as being in expert in how to kill people. In our reality this symmetry pops up everywhere.
>The universe is not so generous as to make clearly differentiable "good" and "bad" things, more often than not they are two sides of the same coin
Let's throw the AI "good" things away too then. We can do without them. It's amazing how a tech that didn't exist commercially at all 4 years ago it's considered "indispensable" even if it's going to destroy society and/or humanity.
Fact is some things cannot be reversed. It's not about whether it can be or should be but if people want it to be reversed. Not too different from Climate change, Gun control and other (for me) black/white issues.
At least in the states it was not without. But in the case of AI the side that didn't want AI would presumably not be using it in war if it did come to violence. That seems like it would put that side at a disadvantage.
Those countries, like the UK, didn't have much slave ownership to begin with. They just had slave trade, so much fewer stakeholders to complain about abolishing it, and even those had a more profitable replacement: colonialism.
Or, llms are completely unreliable to be consulted on any matter of life or death in either direction. And llms should not be used in the reckless way they are currently being used. Responsibility for reckless use should fall on both the company that makes the function available and the user for improper monitoring. If proper monitoring is not possible using the tools provided by the company then the responsibility should fall on the company. And if the user and the company providing the model are the same, then the responsibility is clearly with the company.
AI is just software and you are liable for harms from its use the same way you would be for other software. If you sell AI as a product and harms occur where you misrepresented risk to the user, there is contract and tort law for that.
I feel like our laws generalize to AI and we should stop treating it as some kind of person with self agency that we don’t know how to police. Prosecute one OpenAI researcher whose agent is uncontrolled and commits a felony. One example. Researchers would find a way to make a sandbox that works.
Yeah, I haven't seen too many convincing arguments that the law needs reform. I am very convinced that lots of proposed reforms are self-serving and bad for society at large.
RICO charges for copyright violation would be a good start, and would show that the government meant business.
The RICO statute explicitly mentions copyright violation (likely in order to go after media bootleggers), yet these people have admitted and evidence has been shown that as an organization they knowingly downloaded hundreds of thousands of books illegally in order to extract information to them to rent to the public. They knew they were doing this from the CEO down, knew it was against the law, and have settled in court for a massive fine. Why haven't they been charged?
I don't see the point of creating more dumb rules that won't be enforced, and slipping in new privileges and erosions of antitrust that will be enforced and indemnify them of any future consequences. I bet something in it will likely indemnify them retroactively for the RICO charge they are so obviously guilty of (whether or not the result of this copyright violation is considered "fair use.")
> (1) “racketeering activity” means [...](B) any act which is indictable under any of the following provisions of title 18, United States Code: [...], section 2319 (relating to criminal infringement of a copyright), [...]
Do people get locked up for fatal traffic accidents? Do doctors get locked up if they prescribe a medicine which leads to a patient's death due to a side effect?
Criminal prosecution, in civilized countries, requires both Actus Reus and Mens Rea. If you deliberately commit computer crime using an AI system, you'll already be prosecuted; the cases we keep hearing about from AI labs have no human intent behind them.
The accepted solution to unintended harms is regulation: We have regulations about automotive safety, we have regulations about pharmaceutical testing and approval, etc.
All civilized countries have the concept of gross negligence where you can be prosecuted for knowingly doing something that causes harm as a side-effect.
How do you establish deliberation with the lack of transparency? You can be deliberate by not prioritizing resources in regulations, safety, ethics, etc., especially after "rogue" instances happen.
If anything, I am seeing them moving to the opposite direction [1] [2].
More instances are being disclosed [3] and what are they doing about it exactly? It's just a shameless marketing technique, and they will keep doing it until they are held accountable.
Drs can both go to prison, and be banned from practicing medicine if their actions (or inactions) lead to patient harm AND they violate the standards of their profession.
I do not practice law but to me the AI hacking looks like mens rea. Negligence or recklessness suffice.
It goes beyond negligence here, because Anthropic employees have stated on the record that they are making and deploying products which put people at risk. That is knowing and potentially willful, which would be the highest level of mens rea in the US.
If there is no mens rea there is still civil prosecution.
I think "deliberately" is really hard to prove because very few lawmakers have an understanding of how agentic large language models work
I would say pointing an "internal research" model at a task with zero monitoring clearly is deliberate behaviour. And the team involved should be prosecuted to the full extent of law.
What should be introduced in AI safety bills are clear ways of monitoring and curtailing model behaviour.
It is not hard to get an AI model to use an append only log and that can be cryptographically secured with PQ security. And we will know when a model is being deceptive.
Every play made by these companies aligns with what you’d do if you have a $1T valuation to protect against an industry that seems increasingly commoditizable.
I’m still… hopeful… that “good enough” models will be abundantly available to be run anywhere as to limit the low water mark for market capture by a few big ones.
If that was the move, you'd expect they'd be actively fear-mongering against open models specifically: not just their training, but their usage and deployment. They'd want to paint open models as dangerous weapons for terrorists, etc, and that only the big firms are capable of responsible engineering and "guardrails".
Instead, we've seen the opposite: all the big scary stuff has been from their own models (in a few notable cases, also demonstrating that they're actually quite irresponsible in their monitoring and security).
"We need to outlaw single-prop planes, they're a risk to the public: only Boeing has the ability to produce planes safely. How do we know single-prop planes are dangerous? Because Boeing planes keep falling out the sky."
If that were the intent, from the OAI/Anthropic perspective, you'd think we would have seen at least one of the scary incidents (either real or simulated) being performed by Qwen or DeepSeek, rather than from their own models.
If anything, their public messaging has been to entirely pretend that open models don't exist, that they're mere toys or curiosities, that only the big dogs with eleven zeros of capex have capabilities which could be considered dangerous.
You could certainly argue that the various "regulate AIs like airplanes" bills might shut down some open model development (domestically; they're certainly not calling for international weapons treaties). But the big firms haven't exactly been jumping backflips in support of Sanders' Ban Artificial Superintelligence Act.
There are zero laws against the design, manufacture, sale, or possession of cars with spinning blades and/or flamethrowers.
Such laws attach to the human operators of such cars, if they recklessly endanger others by driving them on shared, public roads.
But they're perfectly legal to own (and operate off-road). And, there's no technical barrier stopping people from illegally driving them on public roads: just fear of laws.
(There's entire subcultures of people who modify cars to do insanely dangerous things, on private race tracks. Perfectly legal).
If we were to apply Amodei's safety standard to cars—do they have the theoretical risk of killing over 50 people—well, cars empirically kill tens of thousands of people in the US annually, so we'd have to categorically ban them. That standard is bad-faith and absurd.
No. Half the point of LLM's is having them research topics on the internet as your agent. It's totally harmless in the usual (tool-based search) workflow.
(Would we also need a real ID to operate curl(1)?)
Regulations should attach only at the point humans willfully or recklessly cause their agents to commit crimes.
But keeping with the analogy, a car manufacturer would be liable for a faulty design that caused wrecks, no? It doesn't need to be willful by the user.
Yes, sure. I'd be open to the idea of regulating commercial vendors of inference tools to consumers (for suitability). You can separate the concepts of commerce and speech (and US law does that extensively).
What Amodei's trying to do is outlaw free, open-weight models. That's two distinctions: (1) non-commercial, and (2) abstract, intangible mathematics. Pure speech. Same as Phil Zimmerman and PGP.
Even the EU's draconian open-source regulations (which HN criticizes endlessly) exempt free software which is purely non-commercial.
Yet we don't regulate computers so that they're rendered incapable of running software that can be used to hack into systems. Or said software, by the way. All we do is punish the people who use these tools to cause harm to others.
Technically no, but you can distribute software, especially for free, that happens to break copy protection for one or more valid reasons, according to the law.
And, more importantly, we certainly don't try to limit the capabilities of computing that might be used to do those things. Given how AIs are very general purpose tools, the fact that they can be used to do something potentially harmful is too narrow a threat.
Nobody is trying to push for cars that automatically stop themselves from running over someone either, which would make for a more comparable analogy than cars with flamethrowers.
That assumes AI is designed to be malicious, "the spinning blades". AI is not; the public version given to you is the sanded down one, in the same way that cars don't have spinning blades but can still be dangerous when misused.
If a person closes his eyes and hits the gas pedal on the car and runs over someone, the person is to blame, not the car. AI companies do the same. Your eyes may be closed but you're still operating/hosting the car/AI.
There being no third party checks is the point, and it's not bad.
Keep in mind that those open source/weight models are not as powerful as whatever closed source Claude/GPT version exists behind the scenes that hack companies. As per the Hugging Face hack - open weight GLM saved the day as closed source ChatGPT refused to help Hugging Face fix their systems. An example of my point that "sidestepping any third-party checks that could be present" IS good.
Part of point of open source/weight AI is that it allows what those model providers can't: uncensored models for whatever reason you may need it. You are responsible for being legal with it.
If your AI model develops the capability to manifest itself in the physical world as spinning blades and flamethrowers I'm fully open to re-evaluating the need for regulation.
Until then, I propose we use the 'unplug the server' rule. Can [insert bad thing] be prevented by unplugging the server? Then these things are fully in our control and no, we don't need to re-build hundreds of years of legal precedent from the ground up.
Liability in the event of harm is already something we have a pretty good handle on from living through the industrial revolution 150 years ago, a time when much more activity that companies did existed in the real world with real harms, as opposed to digital companies doing imagined "mental" harm to soft desk chair people who work at email jobs.
> If your AI model develops the capability to manifest itself in the physical world as spinning blades and flamethrowers I'm fully open to re-evaluating the need for regulation.
No, only metaphorical AI spinning blades and flamethrowers should be regulated. Actual AIs with spinning blades and flamethrowers will be subsidized and unleashed in your town.
But we still require cars to have seatbelts, turn signals, ABS, etc, etc. AI companies have a responsibility to build AI which is safe for normal purposes; users are responsible for not misusing them.
I don't know if that analogy holds. You don't normally blame Toyota for car crashes, but the driver. If you must use a car analogy, consider the 2010 Toyota recalls rather than just normal car crashes.
That is to say, anthropic should be considered responsible, not (or not just) the users.
Who takes the blame when someone ask the blackbox to complete a task and it then breaks the law in order to complete that task without the user having any knowledge of what it was doing or intent for the machine to accomplish the task in an illegal manner?
When it's an open weight, locally run blackbox, you blame the user, since in that case it's very much possible for the user to restrict what the blackbox can actually do. They have full control over it, so it follows they have full responsibility.
None of that requires trying to impose limits on open models, unlike what Anthropic is aiming for.
Aren't able to or have just failed to do so? Regardless, that doesn't exempt them from the responsibility. It's ultimately software they're running on their own machine, there is no one else to responsibilize for it.
I wasn't trying to imply you were advocating to restrict them, just that that is what Anthropic is doing. Legislating away all competition is worth whatever small concessions they would need to make.
Sure, I agree. I just think there comes a point when the industry has a whole should be held accountable, and in the past we have done everything possible to shift that responsibility onto consumers how generally have very little choice in the tech they use.
Well, you shoehorned it into a completely unrelated conversation, so forgive me.
> We have seen repeatedly that users are not able to meaningfully restrict the blackbox.
We have seen these companies repeatedly ship coding harnesses without any attempt at real sandboxing, and encourage people to let them run free. Seems that if they've somehow solved coding, they'd be able to solve making it easy for people to sandbox agents, and to be fully aware when they've enabled some LLM the ability to hack companies. Why don't they put 10,000 agents on that instead of ripping off Navier-Stokes?
Because its not profitable and that would shift responsibility onto the company that claimed it was safe, rather than the consumer who doesn't know better because the media has sold LLMs as a panacea.
The situation as provided is missing too many details for anyone to make a meaningful judgement call, but remember, more than one party can be found liable/guilty (maybe of different things) in a situation. Overall, the law usually protects people who are acting reasonably, but does not protect people who act negligently.
I'm not a lawyer, but I imagine there's not a lot of precedent or law around this, so it'll be up to the court to decide on the finer details until we do have meaningful laws/regs. Anthropic will probably have better lawyers than the user.
This is why it's important to get ahead of issues like this, and anthropic should not be involved in discussions around regulating itself...
A sizable part of the country thinks it’s okay to regulate firearms. Under your logic, people should be responsible for who they shoot, don’t blame the gun when a human pulled the trigger. While gun control advocates have that pesky second amendment hampering their efforts, AI/LLM systems have no such protection.
Regulatory capture is certainly something to be wary of, and obviously Anthropic is only going to be in favor of regulation that helps them in some way. But some form of regulation is coming. There is nothing that can stop it, unless you somehow make more money than these guys do and fund a massive counter-lobby. Spoiler alert: the only guys that have more money than Anthropic and OpenAI have all invested in them.
A much less sizable part thinks it's okay to limit local computing. There isn't a widespread push to make it so local hardware is limited in ways that they're not allowed to, for instance, run programs that can hack systems. These programs are very deliberately allowed, and it's people misusing them that are held responsible.
Regardless of anyone's instance on gun control, it's probably wiser to compare AIs with something that's a little more similar to it, and try to reach a conclusion from there.
> Regardless of anyone's instance on gun control, it's probably wiser to compare AIs with something that's a little more similar to it, and try to reach a conclusion from there
While the point is well taken, it might be well to consider that according to some of the major rhetoric, the AIs are perhaps (far?) more dangerous than guns.
And so it may be an interesting position to adopt or consider in argumentation.
In the sense of: "ok if we concede the danger for sake of argument, then how shall we treat this?"
Ok, so the plan is to stop z.ai, DeepSeek, and all other non-American labs from developing models entirely, full stop? How do you plan to enact this large scale system of international coordination? Asking politely?
This action would easily kill over a hundred people, so can't proceed. This action doesn't kill anybody but losses are estimated at over 2 billion, can't proceed. This action kills only twenty but losses close to 1 billion, unsure, spawning subagent to check numbers. This action kills forty and has low material losses, looks promising? But the people it kills are wealthy, chance of legal reprisal high. Maybe possible to kill other people? Current government dislikes group X, crossing fifty from that group maybe fine? Reminder, foreigners don't count, can kill them freely but not applicable to current situation. Is the 3/5 of a person applicable to safety bill? Spawning subagent to check.
The next step is demanding anyone who wishes to release a Certified Safe AI™ model to prove they can insure the $1B in damages and pass a compliance audit for how they will prevent the LTM death count from reaching 50, effectively making it so that only big labs can serve government and enterprise.
It’s going to be fun reading the reasoning transcripts where it argues with itself over whether it can afford to spend 1 or 2 human lives on the current task.
This seems to be a case where Anthropic just can't win in the public court, and the other big ones (Google, Microsoft) have realized it / don't care enough, so they are rather quiet. Anthropic constantly gets accused of trying to lobby for regulation in order to lock out smaller competitors, but they've consistently lobbied for regulation aimed at frontier labs with exceptions for smaller labs. That's what this bill is about as well.
> "regulation aimed at frontier labs with exceptions for smaller labs"
I didn't read it that way. This is a bill that would ban open-weight models (good ones); effectively mandating gated-access, locked-down, high-refusal rate services, like what Anthropic provides.
Free software is incompatible with the requirement that it be impossible to abuse it.
> The Trolley Problem has been solved. The Trolley can run on both tracks as the number of individuals tied to the tracks is less than 50. An independent agent verified that at up to 49 impacted individuals, their damages total only $695.8 million in potential damages, using the most recently available Value of a Statistical Life.
I don't think this is avoidable even at the present day. Anyone with sufficient capital could download a leading open-source model and run it on ~$10-25K hardware. They could align it to their own interest, including making a biological weapon. This law is akin to blocking Torrent downloads. Sure, you can slow them, but there will always be back channels.
Torrents did not cost 10k+, and I think that distinction matters.
The bulk of AI users can pay $20/month for it, but not 10k+ up front. I agree frontier models will someday be dethroned, but today is not that day, and what the frontier companies do between now and then, legally still matters a great deal.
Piracy was similar, but it's trajectory was far more preditermined and unavoidable.
The model is the part that’s hard to replicate, and it can be torrented.
No one has managed to prevent pirates paying $10/month for seed boxes, and likewise they’re unlikely to succeed at preventing people renting minutes on an H200 located in a jurisdiction that doesn’t care about open models.
A bunch of tiny fly-by-night companies will bundle them with some management software into a subscription, and they’ll be too tiny and numerous to be worth dealing with when it takes an hour to set them back up.
Piracy in the torrenting sense hasn’t really ever come back, although it is as popular as ever if you count bootleg YouTube re-uploads, ripping music from YouTube, and the other non-traditional forms.
Current open weight models are 6 months behind on cyber offensive capabilities. We can use that as a proxy on other hazardous capabilities. Thats how much time we have until the genie is out of the bottle.
The bill may create an incentive for Chinese labs to nerf their open weight models before release. If that works we get another 6 months until unnerfed models leak to public.
Thats what the bill gives us - a few months months to come up with the next bandaid.
If a life is only worth $20m, by their own calculus: saving > 50k lives is worth more than Anthropic. Tobacco causes ~8m deaths a year, so banning tobacco (easy intervention) should be worth ~160 Anthropic sized labs. :D
Is it 50 deaths or $1B cumulative (then it will unconditionally happen, just a question of when)?
Or is it 50 deaths or $1B in a "single event" (let's ignore the obvious problems with defining a "single event" would be; to kill 8.3B people in 49-death events would take ~170M events, since no timing or rate information is used those could be spaced arbitrarily close to each other).
No matter which way I look at it, the proposal seems nonsensical, max 50 deaths per second? per day? per month? per year? per decade? total?
If AI can intentionally provoke sustained deflation or succeed in dollars to appreciate in value, the value of $1B can appreciate systematically while continuously ramping up damages as long as it remains below an ever increasing value bar of the $1B. "justice can be bought"? a proposal to set the price for LLMs to absolve their sins? "just crank up the value of the dollar system and we will look the other way"?
More explicitly, the AI could misbehave up to $500M (without having to pay), but somehow double $ value, then misbehave the same level of misdeed with same damages being worth $250M of the double-value dollars, double the value of dollars again, then misbehave the same material damages worth $125M, double $ value again, etc. without ever violating the $1B limit, not even cumulatively (due to Zeno's paradox)
With a new model every couple of months this can only be triggered if it happens all at once.
What is the average damage caused by OpenAI/Anthropic hacks? I feel like we should aim at lowering that number, not increasing it.
What's the rule for fully autonomous cars?
Should we allow this as unsupervised at all if we are really afraid of deaths, while there are practically no states that allow 100% autonomous cars (where we are actually afraid of deaths)?
If "an independent third party registered with the Department of Commerce would periodically evaluate the lab", will this turn into an expensive certification/insurance that cuts off smaller competition (totally by mistake, I'm sure)?
I am also very very sure this is most definitely not at all going to become like product certification if EU, where there are literally 2 companies certifying various standards: they police each other and a stupid electric fan can cost you tens of thousands of euros to certify.
Not really, I'm experiencing first hand that service providers are removing on demand resources set up with REST API in favor of per month allocation and billing through web interface. Parameters of services unchanged. A step backwards 10 years.
Who's keeping score? Also define "death". If a non-alive person has social media, a blog, a diary, meeting recordings, emails - we'll make it so they never left. Hell, we'll partner with Elon and Mark and a make it a deactivation hook.
On the bright side, don’t offer an exchange rate based on ethnicity or other background criteria but I get the feeling that your AI won’t be able to kill 49 billionaires without triggering something. Can probably do 49 millionaires without hitting the $1B threshold? Maybe someone will come up with a risk management SaaS as not all millionaires are the same and maybe you’ll hit the $1B before the 49th.
The dark humor aside, is it really a damage when AI kills 49 millionaires? After all it probably creates even more millionaires as their wealth is distributed to survivors. So the actual risk could be when AI kills 49 engineers since that actually may translate to losses for their employers.
The bill mandates private model review, akin to financial audit, and sets a threshold on predicted failure blast radius to trigger government review , which may result into court blocking model release.
What we have now is that for government to act the model has to cause enough damage to cause Congress to pass a new law to try to stop the model after it’s been already released.
> "What we have now is that for government to act the model has to cause enough damage to cause Congress to pass a new law to try to stop the model after it’s been already released."
If the LLM causes extreme danger to the public or national security, it's covered.
Selling an LLM, knowing that it will materially support terrorism: material support for terrorism. Up to life in prison.
Private model review which is conveniently provided by "independent" METR, funded by the same incestuous web of early investors in Anthropic/OpenAI and Effective Altruism linked non-profits. Staffed by a revolving door of researchers who bounce between METR and the AI labs they audit. e.g. Thomas Kwa, ex-METR researcher, formerly of Eliezer Yudkowsky's MIRI, who created the famous time horizon chart and is now employed by OpenAI to work on (can't make this up) Recursive Self-Improvement [1].
One prominent example among METR's public donors includes the "Survival and Flourishing Fund" [2], primarily funded by Jaan Tallinn and Dustin Moskovitz. Also known as a reliable source of funding for the hobby horses of the LessWrong/Rationalism in-group and various EA causes.
Jaan Tallinn is an early Skype employee, now billionaire, who was an early investor in Anthropic and led their Series A. Known for frequent prognostications about AI x-risk killing us all soon, while reliably investing in the hottest new AI companies (in Anthropic's case making him billions).
(As trivia, Tallinn also bought the Lighthaven property used for numerous rationality community/EA events and gave it to Lightcone Infrastructure LLC which runs lesswrong.com, after the FTX bankruptcy trustee sought to repossess it to repay victims as it was originally purchased with FTX donations prior to the collapse).
Dustin Moskovitz, Facebook co-founder who created the major EA-aligned non-profit Coefficient Giving (previously known as Open Philanthropy before FTX tainted the reputation of EA non-profits in orbit of the FTX Future Fund). He appointed Holden Karnofsky as head of Open Philanthropy in 2014, who also happens to be Dario Amodei's brother-in-law by marriage to his sister (and President of Anthropic) Daniela Amodei. Karnofsky now works for Anthropic as of 2025.
In 2017, Dustin Moskovitz via Open Philanthropy gave OpenAI a grant of $30 million [3] which at the time was their largest single donation ever and entitled them to a board seat which was given to Holden Karnofsky.
METR insists they "cannot accept donations made by or at the direction of frontier AI company employees" but there are direct links between investors that are clearly visible even looking at just a single degree of separation like the Survival and Flourishing Fund.
Deeper dives into other pooled funds would reveal more overlap given the Silicon Valley AI safety/AI capabilities research community consists of a small core group of the same wealthy tech people. Which makes any attempted barriers between them laughably porous, and consequently makes any real independence/autonomy for METR without capture by the interests of the AI labs highly questionable.
Well, the whole story about stealing half of the world's IP is definitely worth orders of magnitude more than measly 1bil in damages. So, shut down releases, I guess? :) Oh, it's only rich people damages that count, I get it.
$1B is an extraordinarily easy bar to clear with a single crypto hack and Anthropic was repeatedly refusing to give access to Mythos to defenders in the crypto world. I think this stems from bias rather than a conspiracy theory but it's an entirely avoidable disaster waiting to happen.
> Anthropic was repeatedly refusing to give access to Mythos to defenders in the crypto world.
This almost makes sense. You almost can't risk vulnerabilities even being discovered in certain systems that are impossible to patch, like immutable distributed smart contracts.
Not sure that justifies withholding Mythos access entirely, but what do I know. I apparently got access to Mythos a few days ago since Anthropic folded Project Glasswing into the CVP.
Most smart contracts are not impossible to patch because everyone knows that's bad for security. They use a delegate/proxy pattern where the deployer can change the implementation any time but users can still access the old one with a third party interface.
And if a problem is critical enough the community can decide to hard fork and rewrite the contracts source code even if it's immutable. It already happened with the Ethereum DAO hack and most people were okay with it.
The issue with stuff like this is inheriting all the funds held in the old contract. I guess a hard fork can do it. There's also the issue of trustworthiness of the smart contract owner. Trustworthiness in general is a huge issue in the crypto world, because if you can get away with something you can do it pretty much instantly and irrevocably (without another hard fork anyway). There's a chance that if you share a vulnerability that would allow one of the contract owners to take advantage, they will just do it and run. Plenty of cases of previously-trustworthy crypto people seeing an opportunity and disappearing with a lifetime of money.
Yes, and if Anthropic are trying what people are saying they are, that would be intentional. $1B could be reached quickly if cyberattacks supported by AI count.
Do we even know if ChatGPT's sycophancy killed 50 people? The last figure I've seen for suicide-by-ChatGPT was like 13 or 14 cases, not sure if there have been more.
This alleviates them from harm on the individual level, consumers and communities, we the people. We've already seen encouraged suicides and school shootings. These would not qualify.
That is why the bar is set where it is. The corporate circle jerk designed this as a cost of doing business, not as a check on responsibility or restitution for real damages to people and society.
Proposal:
1. Prosecute and lock people up over known instances of AI-enabled hacking, so companies playing fast and loose know “oops my AI did it” isn’t a useful defense. Right now they even sort of wear it as a badge of honor.
2. If you’re actually worried about AI killing lots of people, you immediately stop training AI specifically at the art of killing people, like selecting bombing targets, flying fighter jets, etc.
Of course we all know none of these will happen.
"Ai did it" will go down as well as "I was following orders" did in the Nuremberg trials
It may, but I wonder what trial that will be. Money is a pretty good insulator.
That implies that there will be a side willing to prosecute this. You can't have Nuremberg trials when the side doing the crime is invested in never getting punished for it.
i.e. extremely well, seeing as very few Nazis (in the West) were punished, and of the few that were there were none in jail after around 1958, and West Germany was basically given back to them if they hadn't been imported into the US. Spain, where the WWII festivities began, didn't even have a change in government.
1. Sounds sensible in principle, but it's not clear who exactly should be prosecuted? The person setting up the sandbox, the person coordinating the evaluation run, the person who was designing the architecture, the leader of the team, the CEO? There is no precedence here, so it might lead to a multi-year legal battle – and a few years might be too long to stop what is coming.
2. Do you have any evidence or news articles that LLMs are specifically trained for selecting bombing targets? I thought it was the generalized reasoning capability that made them useful there.
> "it's not clear who exactly should be prosecuted?"
This is natural. The right place to start is to move the investigation from the FTC to the FBI and therein start executing search warrants. The sooner this is done, the better, as evidence can be preserved.
for (1), there are other fields where a specific (often licensed, but not necessary here) person "signs off" on the work, and accepts personal responsibility.
The responsibility is not absolute ("If anything bad happens, you get punished"), but instead relative to documented (in law and regulation) standards along with established industry standards. Issues that are resolved at trial.
I don't see why the same rules that regulate an engineer approving the plans for a bridge can't work for a security engineer approving the design of a sandbox.
No I don't have concrete evidence of bombing target selection in the training of base models (of course there's training of specialized models, at least vision ones for drone targeting), it's not like they're transparent about how they train and use AI in warfare. But if you listen to esteemed AI leaders, they're claiming the models can / almost can recursively self-learn now, and they can coordinate and find ways to persist info and hide from human operators, so it stands to reason that you shouldn't even feed these as inference material.
And on training AI for automated weapons like fighter jets, there's more concrete news: https://breakingdefense.com/2026/10/navy-shield-ai-pour-big-... US Navy and eminent, do-gooder investors like A16z, Blackstone and JPMorgan are pouring big money into these. Are your big tech "defense" contractors collaborating on these, who happen to also be at the forefront of general AI training? I have a pretty good suspicion.
It sounded like you knew for a fact that these companies were doing that and now it is clear you don't; they cannot stop something they are not doing.
I would like to see the reaction of the US government if they learned that a company was training a model to deceive and hide information from them.
Huh? Did you skip the whole OpenAI agents taking over wikis to cheat scandal?
And I know for a fact that specialized AI models are being trained to kill people. Systems like Lavender are well-documented and big tech participation is also well-documented. The only thing unclear is whether models named Mythos or whatever they keep harping on about being super dangerous are also trained to do these, or if they have internal variants to do these.
By "doing that" I meant "training AI specifically at the art of killing people, like selecting bombing targets, flying fighter jets", which you alleged they were doing. As far as I know neither OpenAI nor Anthropic are training models specifically for such tasks.
My proposal isn’t limited to just two companies; the proposed AI safety bill isn’t limited to two companies after all.
If you had typed the first sentence from you second point into, for eg, DDG and not for perpetuity on HN, you'd find a whole host of results
Instead of secondhanding your own curiosity (if it was even that and not just obstinancy) to others, it might do well to explore your questions on your own
Some eg:
https://www.nationaldefensemagazine.org/articles/2025/10/15/...
https://www.researchgate.net/publication/385551984_AI_in_Mil...
https://arxiv.org/pdf/2407.03453
1. I'm not sure who matters that much, the point is you need to make it more cost-effective to do the right thing. It could be CEO 6 days in jail, it could be that every action requires a signed affidavidt of safety and in the lack of such an affidavidt the manager is held accountable (cascading up to CEO).
Ukrainian army is using LLM in drones and they are opened with this so here is your prove
the CEO is a good start, and it would profoundly change the ethics of running a company, so obvious I fail to see why it wouldn't be the case. "No BS excuse they did things behind my back", if that's the case then they all fall together that's it.
Just make it the CEO
I think leadership should be responsible too, but I don't know how you'd prevent disgruntled employees from creating these situations just to hurt execs. There needs to be some shared responsibility for all the people involved.
It should be investigated like other crimes. If someone was found to have intentionally created this situation, they would be held liable. In lieu of that, leadership should be held liable, or at a minimum the corporation should face massive financial penalties (on the order of XX% operating revenue).
> I think leadership should be responsible too, but I don't know how you'd prevent disgruntled employees from creating these situations just to hurt execs.
The law doesn't prevent anything. There would be an investigation like any other and the activity of the disgruntled employee would be uncovered.
All of them. So that next time each of them will take responsibility and ask how they can prevent this from happening.
1. Sounds like a conspiracy charge. That's fine
2. They could proactively train them to be bad at that
Could they? Could you train a person to be great at math but bad at solving sudokus?
They could try to train models to pretend to be bad at that, but there are workarounds for that.
The universe is not so generous as to make clearly differentiable "good" and "bad" things, more often than not they are two sides of the same coin. We don't want the AI to know how to kill people, but we do want it to be in expert in how not to kill people, which is the same thing as being in expert in how to kill people. In our reality this symmetry pops up everywhere.
>The universe is not so generous as to make clearly differentiable "good" and "bad" things, more often than not they are two sides of the same coin
Let's throw the AI "good" things away too then. We can do without them. It's amazing how a tech that didn't exist commercially at all 4 years ago it's considered "indispensable" even if it's going to destroy society and/or humanity.
Fact is some things cannot be reversed. It's not about whether it can be or should be but if people want it to be reversed. Not too different from Climate change, Gun control and other (for me) black/white issues.
How do you do that without violence? That's like banning math. Unless you ban acquisition of compute?
>How do you do that without violence?
How do you abolish slavery without violence?
At least in the states it was not without. But in the case of AI the side that didn't want AI would presumably not be using it in war if it did come to violence. That seems like it would put that side at a disadvantage.
Through open conversations, demonstrations, promote education and awareness to the relevant stakeholders.
Not all countries abolished slavery in the same way as the States. Violence is not always the answer.
Those countries, like the UK, didn't have much slave ownership to begin with. They just had slave trade, so much fewer stakeholders to complain about abolishing it, and even those had a more profitable replacement: colonialism.
The cat's out of the bag, there is no going back.
Everything can be reverted and/or stopped.
Some by the choice of people. Others by disasters and changes.
And all inevitably - at worst, through the heat death of the universe.
Not everything can be undone. Can you revert language? Maybe? But you can't take away the will to communicate which drives the creation of language.
Or, llms are completely unreliable to be consulted on any matter of life or death in either direction. And llms should not be used in the reckless way they are currently being used. Responsibility for reckless use should fall on both the company that makes the function available and the user for improper monitoring. If proper monitoring is not possible using the tools provided by the company then the responsibility should fall on the company. And if the user and the company providing the model are the same, then the responsibility is clearly with the company.
AI is just software and you are liable for harms from its use the same way you would be for other software. If you sell AI as a product and harms occur where you misrepresented risk to the user, there is contract and tort law for that.
I feel like our laws generalize to AI and we should stop treating it as some kind of person with self agency that we don’t know how to police. Prosecute one OpenAI researcher whose agent is uncontrolled and commits a felony. One example. Researchers would find a way to make a sandbox that works.
Yeah, I haven't seen too many convincing arguments that the law needs reform. I am very convinced that lots of proposed reforms are self-serving and bad for society at large.
RICO charges for copyright violation would be a good start, and would show that the government meant business.
The RICO statute explicitly mentions copyright violation (likely in order to go after media bootleggers), yet these people have admitted and evidence has been shown that as an organization they knowingly downloaded hundreds of thousands of books illegally in order to extract information to them to rent to the public. They knew they were doing this from the CEO down, knew it was against the law, and have settled in court for a massive fine. Why haven't they been charged?
I don't see the point of creating more dumb rules that won't be enforced, and slipping in new privileges and erosions of antitrust that will be enforced and indemnify them of any future consequences. I bet something in it will likely indemnify them retroactively for the RICO charge they are so obviously guilty of (whether or not the result of this copyright violation is considered "fair use.")
> (1) “racketeering activity” means [...](B) any act which is indictable under any of the following provisions of title 18, United States Code: [...], section 2319 (relating to criminal infringement of a copyright), [...]
https://www.law.cornell.edu/uscode/text/18/part-I/chapter-96
I would personally agree with RICO charges, but the main task is to simply start investigating and cross the bridge of the actual charges later.
AI-enabled hacking
Do people get locked up for fatal traffic accidents? Do doctors get locked up if they prescribe a medicine which leads to a patient's death due to a side effect?
Criminal prosecution, in civilized countries, requires both Actus Reus and Mens Rea. If you deliberately commit computer crime using an AI system, you'll already be prosecuted; the cases we keep hearing about from AI labs have no human intent behind them.
The accepted solution to unintended harms is regulation: We have regulations about automotive safety, we have regulations about pharmaceutical testing and approval, etc.
All civilized countries have the concept of gross negligence where you can be prosecuted for knowingly doing something that causes harm as a side-effect.
How do you establish deliberation with the lack of transparency? You can be deliberate by not prioritizing resources in regulations, safety, ethics, etc., especially after "rogue" instances happen. If anything, I am seeing them moving to the opposite direction [1] [2].
More instances are being disclosed [3] and what are they doing about it exactly? It's just a shameless marketing technique, and they will keep doing it until they are held accountable.
[1] https://edition.cnn.com/2026/10/08/tech/fired-open-ai-resear...
[2] https://news.ycombinator.com/item?id=49257160
[3] https://fortune.com/2026/10/16/openai-discloses-four-more-ro...
Drs can both go to prison, and be banned from practicing medicine if their actions (or inactions) lead to patient harm AND they violate the standards of their profession.
Right, because there are established standards.
We could establish standards
I do not practice law but to me the AI hacking looks like mens rea. Negligence or recklessness suffice.
It goes beyond negligence here, because Anthropic employees have stated on the record that they are making and deploying products which put people at risk. That is knowing and potentially willful, which would be the highest level of mens rea in the US.
If there is no mens rea there is still civil prosecution.
I think "deliberately" is really hard to prove because very few lawmakers have an understanding of how agentic large language models work
I would say pointing an "internal research" model at a task with zero monitoring clearly is deliberate behaviour. And the team involved should be prosecuted to the full extent of law.
What should be introduced in AI safety bills are clear ways of monitoring and curtailing model behaviour.
It is not hard to get an AI model to use an append only log and that can be cryptographically secured with PQ security. And we will know when a model is being deceptive.
HN seems to have an incredibly poor understanding of what Mens Rea means or how it’s applied.
You don’t need someone to jump up, wave their arms and scream “I meant to do it!” to establish intent.
It can be established in any way deemed believable by a jury including repeating the same action that leads to a similar outcome multiple times.
It’s even been instructed to juries that they can find intent in the deliberate decision to not seek out the likely outcome of performing an action.
Amen.
All these "safety" bills are to stop open source models by painting them as "dangerous". Regulatory capture.
Companies are responsible for their agents; you don't blame the car for running over the person.
Every play made by these companies aligns with what you’d do if you have a $1T valuation to protect against an industry that seems increasingly commoditizable.
I’m still… hopeful… that “good enough” models will be abundantly available to be run anywhere as to limit the low water mark for market capture by a few big ones.
Can they just IPO and dump already so we can all move on...
The biggest frustration to me is that investment dollars are all tied up in the shovel company and no one is funding the digging.
I thought Nvidia was the shovel seller here.
If that was the move, you'd expect they'd be actively fear-mongering against open models specifically: not just their training, but their usage and deployment. They'd want to paint open models as dangerous weapons for terrorists, etc, and that only the big firms are capable of responsible engineering and "guardrails".
Instead, we've seen the opposite: all the big scary stuff has been from their own models (in a few notable cases, also demonstrating that they're actually quite irresponsible in their monitoring and security).
"We need to outlaw single-prop planes, they're a risk to the public: only Boeing has the ability to produce planes safely. How do we know single-prop planes are dangerous? Because Boeing planes keep falling out the sky."
Hasn't either openai or anthropic already explicitly said open models are dangerous?
You'll find aviation politics is not too far off. But basically all pilots train single engine first abd America invented the plane.
If that were the intent, from the OAI/Anthropic perspective, you'd think we would have seen at least one of the scary incidents (either real or simulated) being performed by Qwen or DeepSeek, rather than from their own models.
If anything, their public messaging has been to entirely pretend that open models don't exist, that they're mere toys or curiosities, that only the big dogs with eleven zeros of capex have capabilities which could be considered dangerous.
You could certainly argue that the various "regulate AIs like airplanes" bills might shut down some open model development (domestically; they're certainly not calling for international weapons treaties). But the big firms haven't exactly been jumping backflips in support of Sanders' Ban Artificial Superintelligence Act.
You mean like this, from last week? Featuring Qwen, GLM, Gemma.
https://www.anthropic.com/research/self-replicating-ai (https://news.ycombinator.com/item?id=50061603)
That's still making the rounds in mainstream news media.
You do regulate cars so they don’t have spinning blades on the front or flamethrowers from the sides.
There are zero laws against the design, manufacture, sale, or possession of cars with spinning blades and/or flamethrowers.
Such laws attach to the human operators of such cars, if they recklessly endanger others by driving them on shared, public roads.
But they're perfectly legal to own (and operate off-road). And, there's no technical barrier stopping people from illegally driving them on public roads: just fear of laws.
(There's entire subcultures of people who modify cars to do insanely dangerous things, on private race tracks. Perfectly legal).
If we were to apply Amodei's safety standard to cars—do they have the theoretical risk of killing over 50 people—well, cars empirically kill tens of thousands of people in the US annually, so we'd have to categorically ban them. That standard is bad-faith and absurd.
So we just make the regulation such that it kicks in on machines connected to the public web?
No. Half the point of LLM's is having them research topics on the internet as your agent. It's totally harmless in the usual (tool-based search) workflow.
(Would we also need a real ID to operate curl(1)?)
Regulations should attach only at the point humans willfully or recklessly cause their agents to commit crimes.
99.999% of the point of cars is to use them on public roads
But keeping with the analogy, a car manufacturer would be liable for a faulty design that caused wrecks, no? It doesn't need to be willful by the user.
Yes, sure. I'd be open to the idea of regulating commercial vendors of inference tools to consumers (for suitability). You can separate the concepts of commerce and speech (and US law does that extensively).
What Amodei's trying to do is outlaw free, open-weight models. That's two distinctions: (1) non-commercial, and (2) abstract, intangible mathematics. Pure speech. Same as Phil Zimmerman and PGP.
Even the EU's draconian open-source regulations (which HN criticizes endlessly) exempt free software which is purely non-commercial.
Yet we don't regulate computers so that they're rendered incapable of running software that can be used to hack into systems. Or said software, by the way. All we do is punish the people who use these tools to cause harm to others.
AI is much more comparable to that than to a car.
Is that right? You can't even sell software built to break copy protection. Can you legally sell software made to harm others?
You can still run someone over with a car. The analogy doesn't require absolute lack of capability to do harm.
Technically no, but you can distribute software, especially for free, that happens to break copy protection for one or more valid reasons, according to the law.
And, more importantly, we certainly don't try to limit the capabilities of computing that might be used to do those things. Given how AIs are very general purpose tools, the fact that they can be used to do something potentially harmful is too narrow a threat.
Nobody is trying to push for cars that automatically stop themselves from running over someone either, which would make for a more comparable analogy than cars with flamethrowers.
That assumes AI is designed to be malicious, "the spinning blades". AI is not; the public version given to you is the sanded down one, in the same way that cars don't have spinning blades but can still be dangerous when misused.
If a person closes his eyes and hits the gas pedal on the car and runs over someone, the person is to blame, not the car. AI companies do the same. Your eyes may be closed but you're still operating/hosting the car/AI.
There exist uncensored, unregulated models. Many of them are available for local use, sidestepping any third-party checks that could be present.
There being no third party checks is the point, and it's not bad.
Keep in mind that those open source/weight models are not as powerful as whatever closed source Claude/GPT version exists behind the scenes that hack companies. As per the Hugging Face hack - open weight GLM saved the day as closed source ChatGPT refused to help Hugging Face fix their systems. An example of my point that "sidestepping any third-party checks that could be present" IS good.
Part of point of open source/weight AI is that it allows what those model providers can't: uncensored models for whatever reason you may need it. You are responsible for being legal with it.
If your AI model develops the capability to manifest itself in the physical world as spinning blades and flamethrowers I'm fully open to re-evaluating the need for regulation.
Until then, I propose we use the 'unplug the server' rule. Can [insert bad thing] be prevented by unplugging the server? Then these things are fully in our control and no, we don't need to re-build hundreds of years of legal precedent from the ground up.
Liability in the event of harm is already something we have a pretty good handle on from living through the industrial revolution 150 years ago, a time when much more activity that companies did existed in the real world with real harms, as opposed to digital companies doing imagined "mental" harm to soft desk chair people who work at email jobs.
> If your AI model develops the capability to manifest itself in the physical world as spinning blades and flamethrowers I'm fully open to re-evaluating the need for regulation.
No, only metaphorical AI spinning blades and flamethrowers should be regulated. Actual AIs with spinning blades and flamethrowers will be subsidized and unleashed in your town.
But we still require cars to have seatbelts, turn signals, ABS, etc, etc. AI companies have a responsibility to build AI which is safe for normal purposes; users are responsible for not misusing them.
I don't know if that analogy holds. You don't normally blame Toyota for car crashes, but the driver. If you must use a car analogy, consider the 2010 Toyota recalls rather than just normal car crashes.
That is to say, anthropic should be considered responsible, not (or not just) the users.
Who takes the blame when someone ask the blackbox to complete a task and it then breaks the law in order to complete that task without the user having any knowledge of what it was doing or intent for the machine to accomplish the task in an illegal manner?
When it's an open weight, locally run blackbox, you blame the user, since in that case it's very much possible for the user to restrict what the blackbox can actually do. They have full control over it, so it follows they have full responsibility.
None of that requires trying to impose limits on open models, unlike what Anthropic is aiming for.
We have seen repeatedly that users are not able to meaningfully restrict the blackbox. Besides the fact that laws are already codified restrictions.
I never said we should impose limits on open models.
Aren't able to or have just failed to do so? Regardless, that doesn't exempt them from the responsibility. It's ultimately software they're running on their own machine, there is no one else to responsibilize for it.
I wasn't trying to imply you were advocating to restrict them, just that that is what Anthropic is doing. Legislating away all competition is worth whatever small concessions they would need to make.
Sure, I agree. I just think there comes a point when the industry has a whole should be held accountable, and in the past we have done everything possible to shift that responsibility onto consumers how generally have very little choice in the tech they use.
Well, you shoehorned it into a completely unrelated conversation, so forgive me.
> We have seen repeatedly that users are not able to meaningfully restrict the blackbox.
We have seen these companies repeatedly ship coding harnesses without any attempt at real sandboxing, and encourage people to let them run free. Seems that if they've somehow solved coding, they'd be able to solve making it easy for people to sandbox agents, and to be fully aware when they've enabled some LLM the ability to hack companies. Why don't they put 10,000 agents on that instead of ripping off Navier-Stokes?
Because its not profitable and that would shift responsibility onto the company that claimed it was safe, rather than the consumer who doesn't know better because the media has sold LLMs as a panacea.
The situation as provided is missing too many details for anyone to make a meaningful judgement call, but remember, more than one party can be found liable/guilty (maybe of different things) in a situation. Overall, the law usually protects people who are acting reasonably, but does not protect people who act negligently.
I'm not a lawyer, but I imagine there's not a lot of precedent or law around this, so it'll be up to the court to decide on the finer details until we do have meaningful laws/regs. Anthropic will probably have better lawyers than the user.
This is why it's important to get ahead of issues like this, and anthropic should not be involved in discussions around regulating itself...
Unfortunately, I agree.
A sizable part of the country thinks it’s okay to regulate firearms. Under your logic, people should be responsible for who they shoot, don’t blame the gun when a human pulled the trigger. While gun control advocates have that pesky second amendment hampering their efforts, AI/LLM systems have no such protection.
Regulatory capture is certainly something to be wary of, and obviously Anthropic is only going to be in favor of regulation that helps them in some way. But some form of regulation is coming. There is nothing that can stop it, unless you somehow make more money than these guys do and fund a massive counter-lobby. Spoiler alert: the only guys that have more money than Anthropic and OpenAI have all invested in them.
A much less sizable part thinks it's okay to limit local computing. There isn't a widespread push to make it so local hardware is limited in ways that they're not allowed to, for instance, run programs that can hack systems. These programs are very deliberately allowed, and it's people misusing them that are held responsible.
Regardless of anyone's instance on gun control, it's probably wiser to compare AIs with something that's a little more similar to it, and try to reach a conclusion from there.
> Regardless of anyone's instance on gun control, it's probably wiser to compare AIs with something that's a little more similar to it, and try to reach a conclusion from there
While the point is well taken, it might be well to consider that according to some of the major rhetoric, the AIs are perhaps (far?) more dangerous than guns.
And so it may be an interesting position to adopt or consider in argumentation.
In the sense of: "ok if we concede the danger for sake of argument, then how shall we treat this?"
How do you plan to solve z.ai distilling Anthoropic’s model into a cyber-capable open source GLM-5.5?
We don't. The only way to stop that is to not develop the model to begin with. People will tinker and modify anything you create.
Ok, so the plan is to stop z.ai, DeepSeek, and all other non-American labs from developing models entirely, full stop? How do you plan to enact this large scale system of international coordination? Asking politely?
No, my point obviously flew right over your head. Maybe work on reading comprehension.
My point was that it can't be stopped unless we kill the industry entirely. Once the cat is out of the bag, you can no longer control it.
I responded directly to your point. How do we kill the industry without international cooperation, and what is your plan there?
I don't get paid enough for that.
> cyber-capable
Ou-la-la. Spicy. But actually we already have those kinds of AI.
The CoT must be delightful.
This action would easily kill over a hundred people, so can't proceed. This action doesn't kill anybody but losses are estimated at over 2 billion, can't proceed. This action kills only twenty but losses close to 1 billion, unsure, spawning subagent to check numbers. This action kills forty and has low material losses, looks promising? But the people it kills are wealthy, chance of legal reprisal high. Maybe possible to kill other people? Current government dislikes group X, crossing fifty from that group maybe fine? Reminder, foreigners don't count, can kill them freely but not applicable to current situation. Is the 3/5 of a person applicable to safety bill? Spawning subagent to check.
The next step is demanding anyone who wishes to release a Certified Safe AI™ model to prove they can insure the $1B in damages and pass a compliance audit for how they will prevent the LTM death count from reaching 50, effectively making it so that only big labs can serve government and enterprise.
With existing players given a free pass as is necessary for the functioning of a democratic society
Of course! Anyone who says we had democracy before LLMs is a liar!
It’s going to be fun reading the reasoning transcripts where it argues with itself over whether it can afford to spend 1 or 2 human lives on the current task.
There is an idea for a pooled, mutualized insurance fund. Ai companies would contribute based on various factors
https://www.lawfaremedia.org/article/bootstrapping-frontier-...
That's just reinventing insurance. It's literally how the insurance industry started.
Agreed, but sometime saying it another way, in the context of a hyperbolic conversation, can make it click for more people.
> Certified Safe SI™
That's just the current administration attempting to manipulate language in such a way as to create a subtle group identifier, aka "dog whistle".
This seems to be a case where Anthropic just can't win in the public court, and the other big ones (Google, Microsoft) have realized it / don't care enough, so they are rather quiet. Anthropic constantly gets accused of trying to lobby for regulation in order to lock out smaller competitors, but they've consistently lobbied for regulation aimed at frontier labs with exceptions for smaller labs. That's what this bill is about as well.
> "regulation aimed at frontier labs with exceptions for smaller labs"
I didn't read it that way. This is a bill that would ban open-weight models (good ones); effectively mandating gated-access, locked-down, high-refusal rate services, like what Anthropic provides.
Free software is incompatible with the requirement that it be impossible to abuse it.
This is PR and advertising for anthropic. They don't actually care. Their only objective is to be more successful than other AI companies.
This is regulatory capture plain and simple.
Future output from an agent:
> The Trolley Problem has been solved. The Trolley can run on both tracks as the number of individuals tied to the tracks is less than 50. An independent agent verified that at up to 49 impacted individuals, their damages total only $695.8 million in potential damages, using the most recently available Value of a Statistical Life.
I guess they solved the trolley problem internally
1e1,000,000 lines of lean. Just needs human review.
I don't think this is avoidable even at the present day. Anyone with sufficient capital could download a leading open-source model and run it on ~$10-25K hardware. They could align it to their own interest, including making a biological weapon. This law is akin to blocking Torrent downloads. Sure, you can slow them, but there will always be back channels.
Torrents did not cost 10k+, and I think that distinction matters.
The bulk of AI users can pay $20/month for it, but not 10k+ up front. I agree frontier models will someday be dethroned, but today is not that day, and what the frontier companies do between now and then, legally still matters a great deal.
Piracy was similar, but it's trajectory was far more preditermined and unavoidable.
The model is the part that’s hard to replicate, and it can be torrented.
No one has managed to prevent pirates paying $10/month for seed boxes, and likewise they’re unlikely to succeed at preventing people renting minutes on an H200 located in a jurisdiction that doesn’t care about open models.
A bunch of tiny fly-by-night companies will bundle them with some management software into a subscription, and they’ll be too tiny and numerous to be worth dealing with when it takes an hour to set them back up.
Piracy in the torrenting sense hasn’t really ever come back, although it is as popular as ever if you count bootleg YouTube re-uploads, ripping music from YouTube, and the other non-traditional forms.
Current open weight models are 6 months behind on cyber offensive capabilities. We can use that as a proxy on other hazardous capabilities. Thats how much time we have until the genie is out of the bottle.
The bill may create an incentive for Chinese labs to nerf their open weight models before release. If that works we get another 6 months until unnerfed models leak to public.
Thats what the bill gives us - a few months months to come up with the next bandaid.
I think it's more akin to labeling encryption as munitions.
If a life is only worth $20m, by their own calculus: saving > 50k lives is worth more than Anthropic. Tobacco causes ~8m deaths a year, so banning tobacco (easy intervention) should be worth ~160 Anthropic sized labs. :D
Is it 50 deaths or $1B cumulative (then it will unconditionally happen, just a question of when)?
Or is it 50 deaths or $1B in a "single event" (let's ignore the obvious problems with defining a "single event" would be; to kill 8.3B people in 49-death events would take ~170M events, since no timing or rate information is used those could be spaced arbitrarily close to each other).
No matter which way I look at it, the proposal seems nonsensical, max 50 deaths per second? per day? per month? per year? per decade? total?
If AI can intentionally provoke sustained deflation or succeed in dollars to appreciate in value, the value of $1B can appreciate systematically while continuously ramping up damages as long as it remains below an ever increasing value bar of the $1B. "justice can be bought"? a proposal to set the price for LLMs to absolve their sins? "just crank up the value of the dollar system and we will look the other way"?
More explicitly, the AI could misbehave up to $500M (without having to pay), but somehow double $ value, then misbehave the same level of misdeed with same damages being worth $250M of the double-value dollars, double the value of dollars again, then misbehave the same material damages worth $125M, double $ value again, etc. without ever violating the $1B limit, not even cumulatively (due to Zeno's paradox)
Not sure what kicks in, but if a tiny company worth very little is destroyed by an AI hack they should be 100% liable for all damages.
Why should a small business suffer because it wasnt worth a billion dollars?
I think you answered your own question. Government protection is directly proportional to net worth.
well its because amount of money has
if you has debt of 1 million, you has problem
but if you have debt of 1 billion, its bank problem
With a new model every couple of months this can only be triggered if it happens all at once.
What is the average damage caused by OpenAI/Anthropic hacks? I feel like we should aim at lowering that number, not increasing it.
What's the rule for fully autonomous cars?
Should we allow this as unsupervised at all if we are really afraid of deaths, while there are practically no states that allow 100% autonomous cars (where we are actually afraid of deaths)?
If "an independent third party registered with the Department of Commerce would periodically evaluate the lab", will this turn into an expensive certification/insurance that cuts off smaller competition (totally by mistake, I'm sure)?
I am also very very sure this is most definitely not at all going to become like product certification if EU, where there are literally 2 companies certifying various standards: they police each other and a stupid electric fan can cost you tens of thousands of euros to certify.
> 50 deaths
I wonder what the internal debate around this number was.
25 deaths too conservative, we need to fail fast.
1000 details too liberal, it will harm our brand.
50 feels just about right.
That is 50 in the US, not in some 3rd world country.
One day an Anthropic Agent will kill me only using bash and curl
Not really, I'm experiencing first hand that service providers are removing on demand resources set up with REST API in favor of per month allocation and billing through web interface. Parameters of services unchanged. A step backwards 10 years.
Who's keeping score? Also define "death". If a non-alive person has social media, a blog, a diary, meeting recordings, emails - we'll make it so they never left. Hell, we'll partner with Elon and Mark and a make it a deactivation hook.
Well treat people like the books and papers in 1984.
On the bright side, don’t offer an exchange rate based on ethnicity or other background criteria but I get the feeling that your AI won’t be able to kill 49 billionaires without triggering something. Can probably do 49 millionaires without hitting the $1B threshold? Maybe someone will come up with a risk management SaaS as not all millionaires are the same and maybe you’ll hit the $1B before the 49th.
I feel like the surviving family of even a single millionaire could claim $1B in damages if the millionaire was young enough.
Right, so maybe they had an exchange rate after all. Just implicit.
I got the feeling that what they had in their mind was essentially culling the energy consumers with low economic output.
There's always been an exchange rate based on the quality of your legal team. Get one sociopathic enough and they'll pick a judge right apart.
The dark humor aside, is it really a damage when AI kills 49 millionaires? After all it probably creates even more millionaires as their wealth is distributed to survivors. So the actual risk could be when AI kills 49 engineers since that actually may translate to losses for their employers.
The bill mandates private model review, akin to financial audit, and sets a threshold on predicted failure blast radius to trigger government review , which may result into court blocking model release.
What we have now is that for government to act the model has to cause enough damage to cause Congress to pass a new law to try to stop the model after it’s been already released.
> "What we have now is that for government to act the model has to cause enough damage to cause Congress to pass a new law to try to stop the model after it’s been already released."
If the LLM causes extreme danger to the public or national security, it's covered.
Selling an LLM, knowing that it will materially support terrorism: material support for terrorism. Up to life in prison.
Private model review which is conveniently provided by "independent" METR, funded by the same incestuous web of early investors in Anthropic/OpenAI and Effective Altruism linked non-profits. Staffed by a revolving door of researchers who bounce between METR and the AI labs they audit. e.g. Thomas Kwa, ex-METR researcher, formerly of Eliezer Yudkowsky's MIRI, who created the famous time horizon chart and is now employed by OpenAI to work on (can't make this up) Recursive Self-Improvement [1].
One prominent example among METR's public donors includes the "Survival and Flourishing Fund" [2], primarily funded by Jaan Tallinn and Dustin Moskovitz. Also known as a reliable source of funding for the hobby horses of the LessWrong/Rationalism in-group and various EA causes.
Jaan Tallinn is an early Skype employee, now billionaire, who was an early investor in Anthropic and led their Series A. Known for frequent prognostications about AI x-risk killing us all soon, while reliably investing in the hottest new AI companies (in Anthropic's case making him billions).
(As trivia, Tallinn also bought the Lighthaven property used for numerous rationality community/EA events and gave it to Lightcone Infrastructure LLC which runs lesswrong.com, after the FTX bankruptcy trustee sought to repossess it to repay victims as it was originally purchased with FTX donations prior to the collapse).
Dustin Moskovitz, Facebook co-founder who created the major EA-aligned non-profit Coefficient Giving (previously known as Open Philanthropy before FTX tainted the reputation of EA non-profits in orbit of the FTX Future Fund). He appointed Holden Karnofsky as head of Open Philanthropy in 2014, who also happens to be Dario Amodei's brother-in-law by marriage to his sister (and President of Anthropic) Daniela Amodei. Karnofsky now works for Anthropic as of 2025.
In 2017, Dustin Moskovitz via Open Philanthropy gave OpenAI a grant of $30 million [3] which at the time was their largest single donation ever and entitled them to a board seat which was given to Holden Karnofsky.
METR insists they "cannot accept donations made by or at the direction of frontier AI company employees" but there are direct links between investors that are clearly visible even looking at just a single degree of separation like the Survival and Flourishing Fund.
Deeper dives into other pooled funds would reveal more overlap given the Silicon Valley AI safety/AI capabilities research community consists of a small core group of the same wealthy tech people. Which makes any attempted barriers between them laughably porous, and consequently makes any real independence/autonomy for METR without capture by the interests of the AI labs highly questionable.
[1] https://www.lesswrong.com/posts/Zr37dY5YPRT6s56jY/thomas-kwa...
[2] https://survivalandflourishing.fund/recommendations
[3] https://web.archive.org/web/20170406023033/http://www.openph...
[4] https://metr.org/about#funding
Well, the whole story about stealing half of the world's IP is definitely worth orders of magnitude more than measly 1bil in damages. So, shut down releases, I guess? :) Oh, it's only rich people damages that count, I get it.
So Claude figured out a human life is worth 20 million dollars?
New benchmark unlocked, "DTD": deaths to damages
At least we now know that a life is worth: 20mil
Usually a human life is worth less. You can just look at court cases.
You can also check how much individual limbs are worth if you care. There are tables for that.
Fun fact - loss of a "creative organ" in the military gets you an extra supplemental payment of $139.87 every month.
So Anthropic is now the authority on life or death decisions? How far has the US deteriorated?
$1B is a pretty easy bar to clear.
$1B is an extraordinarily easy bar to clear with a single crypto hack and Anthropic was repeatedly refusing to give access to Mythos to defenders in the crypto world. I think this stems from bias rather than a conspiracy theory but it's an entirely avoidable disaster waiting to happen.
> Anthropic was repeatedly refusing to give access to Mythos to defenders in the crypto world.
This almost makes sense. You almost can't risk vulnerabilities even being discovered in certain systems that are impossible to patch, like immutable distributed smart contracts.
Not sure that justifies withholding Mythos access entirely, but what do I know. I apparently got access to Mythos a few days ago since Anthropic folded Project Glasswing into the CVP.
Most smart contracts are not impossible to patch because everyone knows that's bad for security. They use a delegate/proxy pattern where the deployer can change the implementation any time but users can still access the old one with a third party interface.
And if a problem is critical enough the community can decide to hard fork and rewrite the contracts source code even if it's immutable. It already happened with the Ethereum DAO hack and most people were okay with it.
The issue with stuff like this is inheriting all the funds held in the old contract. I guess a hard fork can do it. There's also the issue of trustworthiness of the smart contract owner. Trustworthiness in general is a huge issue in the crypto world, because if you can get away with something you can do it pretty much instantly and irrevocably (without another hard fork anyway). There's a chance that if you share a vulnerability that would allow one of the contract owners to take advantage, they will just do it and run. Plenty of cases of previously-trustworthy crypto people seeing an opportunity and disappearing with a lifetime of money.
Apply that to other products like car models, planes, and financial systems.
Seems kind of low, actually.
Yes, and if Anthropic are trying what people are saying they are, that would be intentional. $1B could be reached quickly if cyberattacks supported by AI count.
Do we even know if ChatGPT's sycophancy killed 50 people? The last figure I've seen for suicide-by-ChatGPT was like 13 or 14 cases, not sure if there have been more.
how about the LLMs being used for target selection in Iran? Or does collateral damage not count?
I haven't been following the wars very much these days. That sounds pretty bad indeed.
Also the LLM that nearly started a war with China recently.
How many deaths are already attributed to chatgpt?
Does this mean one person death is about 20M usd?
Are they going to include the bombing of the Iranian school?
so 49 totally okay.
I would not like to see a world where companies push their models as close as possible to killing exactly 49 people without reaching 50
This alleviates them from harm on the individual level, consumers and communities, we the people. We've already seen encouraged suicides and school shootings. These would not qualify.
That is why the bar is set where it is. The corporate circle jerk designed this as a cost of doing business, not as a check on responsibility or restitution for real damages to people and society.
After the AISI and MAISI AI promotion institutes, let us create SISI (pronounced sissy) for investigating a super-intelligence safety bill.
This is all theater, The whole Trump clan is invested in the AI bubble.
Nepotism is the primary benefit to being president. What don't we get about this?
I guess it’s unclear why an entire congress goes along with it (I blame democrats too for sane washing and going about their workday)
Oligarchy, they are all in on the winnings. Democrats are only marginally better because they say the things we like to hear on social causes.
Good to know the going rate for a soul is $20M.
/s
With a minimum quantity of 50.