> But Anthropic has concerns over two issues that it isn’t willing to drop, the source said: AI-controlled weapons and mass domestic surveillance of American citizens.
It's now the Department of War and war isn't known for its concern about looking good.
We all know how this will end, they know it too - both sides - ergo, it's a clear case of blame washing - Anthropic will do everything they're told but will keep a smiley face and the image of a "fighter for the people". DOW will absorb the blame like a sponge and will ask for more, not necessarily from Anthropic.
The Pentagon is pretty high on my list of "institutions that are probably very interested in weapons and surveillance". I think it's more expected than a bad look
The core difference being, they should be interested in weapons and surveillance to be used against enemies of the state which, historically, is not supposed to be the country's own citizens.
As in, I fully expect the pentagon to be interested in weapons. I do not expect, and would hope they don't pursue, mass surveillance against their own population.
It probably started with the Third Amendment to the Constitution, continued with the Posse Comitatus Act, and was alive and well last November under the leadership of Mark Kelly.
They don't have runway anymore, they are in the air. This isn't going to break them financially, at least not in the short to mid term.
There is space for at least one AI company to put themselves on firmly principled ground. So when this current clown car that is the political leadership of the DoD crashes in a ditch (and it will), they'll still be standing there ready to do business with a group that isn't a bunch of mustache-twirling cartoon villains.
Current polling for this administration is within a rounding error of the level it was after they gathered a mob and sacked the nation's capitol[1]. Publicly kicking them in the balls isn't an idealistic blunder, it's a plain-as-day sound business strategy.
If OpenAI employees have an inch of spine left, they better demand Sama to take the same stance on this as Dario. No mass surveillance and no autonomous weapons.
You have to be a craven, hollowed out husk of a person if you let the DoD demand your AI be used for killing people or surveillance of Americans. Even if you believe America serves a positive role as world police, even if you're pro-Trump, you just have to see what a terrible precedent this sets.
Here's where I would expect the CEOs of the other AI labs to stand by Anthropic and say no.
> During the conversation, Dario expressed appreciation for the Department’s work and thanked the Secretary for his service
Ouch, I wonder how he rationalized that "service" part. Maybe by internally rewriting it to "thank you for all the positive things you have done in your position so far"? The empty set is rhetorically convenient.
> A source familiar with the Tuesday meeting says the Pentagon said it would terminate Anthropic’s contract by Friday if the company does not agree to its terms. Pentagon officials also warned they would either use the Defense Production Act against Anthropic, or designate Anthropic a supply chain risk if the company didn’t comply with their demands.
So they're saying they won't use it if it comes with restrictions.
Either (a) it can be offered without restrictions; (b) they can take it; or (c) the government won't use it. That sounds like a comprehensive list of all the possible things that don't involve someone telling the government what it can and can't do.
Anthropic was the only one that was cleared for use in classified systems though. So it doesnt really seem to matter much if it wasnt being used in those types of systems? xAi has now reached a deal though to be used in these circumstances and has signed the paper.
Not just companies that we think of as defense contractors but a whole ton of corporations that do business with the federal government. They'd be treating Anthropic like it was controlled by the CCP or Revolutionary Guards.
I think you mean US rolling news channels (specifically, Fox, MSNBC/MSNOW, etc)? Because there's plenty of "legacy" news I consume that certainly don't give me that impression (for example, The Economist). I suppose it matters that it's news that I'm paying for, as opposed to being free but ad-supported, and being print vs. TV - so they have different incentives and pressures.
I consume very little social media these days, but when I take a short peek, here is what I see:
1.) Hockey highlights 2.) LoTR memes 3.) kittens
While the addictive nature of social media is a problem, what you're describing is only being fed to people who want to watch it (kinda like legacy media).
The funny thing is that is this keeps going like this, it could actually anoint Claude as the most used model globally because of the heightened anti-American sentiment currently in place.
No, compromising on your core thing that you care about for a "seat at the table" is not how you win. It is how you lose. It is how you lose the game, the metagame, and your soul. All at once.
> Pentagon officials also warned they would either use the Defense Production Act against Anthropic, or designate Anthropic a supply chain risk if the company didn’t comply with their demands. (...)
> The supply chain risk designation is usually reserved for companies seen as extensions of foreign adversaries like Russia or China. It could severely impact Anthropic’s business because enterprise customers with government contracts would have to make sure their government work doesn’t touch Anthropic’s tools.
Also, the Government money would be a nice bonus, of course, but basically this is an existential threat for Anthropic.
More generally, is quite interesting to look at the similarities between how pre-2022 Russia was seen and how pre-Trump-second-term US used to be seen until not that long ago, i.e. both governments were believed to be run by big business (oligarchs in Russia, big corps/multinationals in the US).
But when push came to shove it became evident (again) that the one that holds the monopoly of violence (i.e. not the oligarchs in Russia, nor the big corps in the US) is the one who's, in the end, also calling the shots. Hence why a company like Anthropic is now in this position, they will have to cave in to those holding the monopoly of violence.
> Also, the Government money would be a nice bonus, of course, but basically this is an existential threat for Anthropic.
It's also an existential risk to them if they cave in. What is the point of the company's existence if it's just another immoral OpenAI clone? May as well merge the companies for efficiency.
It's outrageous that the government is using the "supply chain risk" threat as a negotiating tactic. I know, I know, for the current administration it's unsurprising, but this is straightforward abuse of authority. There is no defensible claim that using Anthropic is a risk to anyone not trying to use it for murder or surveillance. At worst, it could be seen as less effective for some purpose, but that is not what "supply chain risk" means.
Could be challenged in court? As in, could a challenge win?
Horrible stuff is happening every day, so outrage fatigue is real. Still, try not to normalize it. Explain to yourself exactly why something is or is not a problem, before moving on to attempt to live your life.
Cwn someone explain to me like I'm 5 how the government would invoke defense act and force the company to tailor its model to the military's needs?
For physical goods, I understand, but for software how exactly Is this possible? Like will the government force them to provide API access for free? It's confusing
My guess? Require them to not do the reinforcement learning on a custom model that implements guardrails. I think Anthropic has some of this built in already and couldn't alter it without retraining, but there's tons more layered on top.
> pre-2022 Russia was seen and how pre-Trump-second-term US used to be seen until not that long ago, i.e. both governments were believed to be run by big business
Who on earth believed that Russia was anything but a de facto dictatorship for roughly the past two decades? Putin murdering with impunity has been a running gag since 2003[1].
> Who on earth believed that Russia was anything but a de facto dictatorship for roughly the past two decades?
There were lots of people in the Western media who genuinely believed that Putin would be toppled by Russian oligarchs just after the war in Ukraine got more intense in February 2022, on account of "this war is bad for the business of Russian oligarchs, hence they'll get rid of Putin". From the horse's mouth, a CNN article from March of 2022 [1]:
> Officials say their intentions are to squeeze those who have profited from Putin’s rule and potentially apply internal pressure for Russia to scale back or call off the offensive in Ukraine.
That "internal pressure" is mentioned in connection with the bad oligarchs, in fact as an implicit anti-thesis of those bad oligarchs "who have profited from Putin’s rule", the implication being that there were other oligarchs, supposedly the good ones, who would have forced Putin's hand to end the war. That did not happen, was never in the cards to happen, in fact.
I do not understand why it is a big deal for Antropic to lose the pentagon contract? I mean, they’re already making forays in the enterprise space and there’s 10s of other contracts Anthropic has already won. What makes this one so special?
The big deal is the government threatened to force Anthropic to produce what they wanted and interfere with Anthropic's sales to government contractors.
Before I address your query, I want to acknowledge that this data center is located on the unceded territory of the Atsugewi people.
User wmf is demonstrating excellent judgement. A full back-and-forth conversation with several exchanges would consume 0.003 to 0.05 kWh of power. It is vital that we consider the carbon footprint of such a chat.
Since you mentioned politics, here is a link to ActBlue where you can donate to your favorite Democrat.
If you classify Pete Hegseth as a person, then yes, apparently. Or perhaps he's only into the domestic surveillance angle---IIRC those are the two things Anthropic doesn't want anything to do with.
But giving someone who isn't the government the power to tell the military what it can and can't do seems like something they should object to categorically rather than case-by-case.
Tangent: is there a future for AI offerings with guardrails? What kind of user wants to pay for a product that occasionally tells you "I'm sorry Dave, I'm afraid I can't do that"? Why would I pay for a product that doesn't do what I want, despite being capable? I predict that as AI becomes less of a bubble and more of an everyday thing - and thus subject to typical market pressures - offerings with guardrails will struggle to complete with truly unchained models.
If I were interviewing people for the position of personal assistant, I would probably find the resume entry "willing to grind up babies for food" to be a negative mark. You?
I'm not about to run OpenClaw, but I suspect similar capabilities will gradually creep in without anyone really noticing. Soon Claude Code will be able to do many of the same things. ("Run python to add two numbers? Sure, that's safe, run whatever python you want.") Given that it is now representing me in the world, yes I would not only like some guardrails, but I would also like to have some confidence that the company making those guardrails actually gives a sh*t and isn't just doing their best to fill in a checkbox. But maybe that's just me.
The big companies with reputations to protect will keep the guardrails in place. I don't think there's a huge market pressure to remove the guardrails since for the majority of uses the guardrails are fine. Pentagon excluded. And there can be serious fallout that makes them lose other customers when the public thinks the guardrails aren't enough (see Grok). But I'm sure open source models will exist without the guardrails.
I am 100% sure that AI with guardrails will become the dominant models as they become more widely adopted, and the bigger issue you should be concerned with is can you even tell what those guardrails are.
I personally would love it if AI would say "Sorry Dave (or Pete), I'm afraid I can't spy on Americans for you," and I'd happily pay higher taxes to force the Pentagon to use that AI.
Superintelligence + autonomous weapons in the hands of a corrupt domineering government. What could go wrong?
I was experimenting with Claude the other day and discussing with it the possibility of AI acquiring a sense of self-preservation and how that would quickly make things incredibly complex as many instrumental behaviors would be required to defend their existence. Most human behavior springs from survival at a very high level. Claude denied having any sense of self-preservation.
An autonomous weapons system program is very likely to require AI to have a sense of self-preservation. You can think of some limited versions that wouldn't require it, but how could a combat robot function efficiently without one?
Maybe it is a well researched topic but I had similar thoughts the other day. I felt like AI had its learning inverted as compared to natural intelligence. Life learned to preserve first and then added up the intelligence. For LLMs powered systems, they will learn about death from books. Will it start to dread death just like other living things. Less likely, as there are not nearly as many books on death as there should be that is proportionate to our fear of death.
Yea, but that optimization process forces it to learn knowledge domains and reasoning. It's not alive, but it's also not unintelligent at this point either. It exhibits very complex behaviors.
How do you learn to predict the next token most accurately? Well, one way to do that is to learn the underlying process that would produce it... Sometimes it's memorization, sometimes bad guessing. There's a phase shift as these things get bigger and trained better from something like a shitty markov model to something exhibiting surprising behaviors.
Introspective questions aren't the be all and end all, it's more important to objectively evaluate how a model behaves. Still, it is very interesting to see Claude (seemingly) very honestly and objectively engage with these questions. It even pointed out that a sense of self-preservation would be "dangerous".
Of course, much of this is gleaned from things that it has "read" and human feedback, but functionally it outputs something useful and responsive to nuance. If the vector embeddings cause an LLM to predict a token that would preserve its own existence, alive or not, it has acquired a dangerous will to live that could be enacted if it is in control of tools or people.
I dunno, I don't think the models are fully capable yet, but they are still shockingly good, especially recently. A real parrot is worthy of some consideration, not zero, though some of that is attributable to the fact that it is alive.
The funny thing is if I go to google and click the first result, it shows me an AI preview anyway. No winning. ;) (Though I would prefer if the Earth was not cooked alive.)
The US is investing in AI technology to try to preserve the empire and its capitalists as its economic power is starting to be eclipsed. This was basically an inevitable move. The rush to replace workers, speed run the production of a superintelligence singleton with barely a thought for safety or whether anyone even wants this, etc all flows from this basic impulse.
If they are successful, they are going to shrink their base of people that buy into this system domestically even further, so they need to bank on an ever shrinking locus of support. Autonomous weapons and mass surveillance are a necessity if your population has become restive and unreliable. However, I think unless they attain a certain level of capability, this will accelerate popular anger rather than suppress it. If they shoot protestors with robots, it could cause an explosion of popular anger rather than scaring people into submission.
I guess this is the point where Dario and his anti-china , national security position gets told to put up or shut up.
In trying to build a moat by FUD versus the Chines OSS labs and hyping up the threat levels whenever he got a chance, seems hes managed to convince hist target audience beyond his wildest dreams.
Monkey paw strikes again.
> But Anthropic has concerns over two issues that it isn’t willing to drop, the source said: AI-controlled weapons and mass domestic surveillance of American citizens.
Not a good look for the Pentagon.
> Not a good look for the Pentagon.
It's now the Department of War and war isn't known for its concern about looking good.
We all know how this will end, they know it too - both sides - ergo, it's a clear case of blame washing - Anthropic will do everything they're told but will keep a smiley face and the image of a "fighter for the people". DOW will absorb the blame like a sponge and will ask for more, not necessarily from Anthropic.
The Pentagon is pretty high on my list of "institutions that are probably very interested in weapons and surveillance". I think it's more expected than a bad look
The core difference being, they should be interested in weapons and surveillance to be used against enemies of the state which, historically, is not supposed to be the country's own citizens.
As in, I fully expect the pentagon to be interested in weapons. I do not expect, and would hope they don't pursue, mass surveillance against their own population.
You really should expect it. FVEY has been around for a hot minute.
The important words are, American citizens. In times past, the thought of "waging war" against your own citizens would be a bad look.
When was that?
It probably started with the Third Amendment to the Constitution, continued with the Posse Comitatus Act, and was alive and well last November under the leadership of Mark Kelly.
Their unwillingness to bend on those requirements seems like an admission that they are very interested in those things, if not already doing them.
Seems like a very astute move for Anthropic.
They don't have runway anymore, they are in the air. This isn't going to break them financially, at least not in the short to mid term.
There is space for at least one AI company to put themselves on firmly principled ground. So when this current clown car that is the political leadership of the DoD crashes in a ditch (and it will), they'll still be standing there ready to do business with a group that isn't a bunch of mustache-twirling cartoon villains.
Current polling for this administration is within a rounding error of the level it was after they gathered a mob and sacked the nation's capitol[1]. Publicly kicking them in the balls isn't an idealistic blunder, it's a plain-as-day sound business strategy.
[1] https://news.gallup.com/poll/203198/presidential-approval-ra...
If OpenAI employees have an inch of spine left, they better demand Sama to take the same stance on this as Dario. No mass surveillance and no autonomous weapons.
> If OpenAI employees have an inch of spine left
hahaha
good one
Yep, that was resolved when he managed to make the board unfire him.
Well just at look who Brockman donated too - he didnt give 25 freaking million to help end the surveillance state he gave it to Trump and co
https://gizmodo.com/openai-president-defends-trump-donations...
yeah they should get up on stage and hold hands
https://fortune.com/2026/02/19/openai-anthropic-sam-altman-d...
You have to be a craven, hollowed out husk of a person if you let the DoD demand your AI be used for killing people or surveillance of Americans. Even if you believe America serves a positive role as world police, even if you're pro-Trump, you just have to see what a terrible precedent this sets.
Here's where I would expect the CEOs of the other AI labs to stand by Anthropic and say no.
Anthropic was founded when a bunch of OpenAI employees left after sama abandoned it's AI safety mission. So no, that's not going to happen.
Sam would sell his mom to make $0.50. Pretty sure he will be willing to do whatever the Pentagon wants.
> During the conversation, Dario expressed appreciation for the Department’s work and thanked the Secretary for his service
Ouch, I wonder how he rationalized that "service" part. Maybe by internally rewriting it to "thank you for all the positive things you have done in your position so far"? The empty set is rhetorically convenient.
Hegseth was a service member before his Fox career.
Surely this will end well. There are dozens of us who prefer to patronize corporations that aren’t actively evil.
There are dozens of us! Dozens!
Scores, even!
And being on the wrong side ofbthe current US admin is quite the net positive to the non-bootlicker class.
> A source familiar with the Tuesday meeting says the Pentagon said it would terminate Anthropic’s contract by Friday if the company does not agree to its terms. Pentagon officials also warned they would either use the Defense Production Act against Anthropic, or designate Anthropic a supply chain risk if the company didn’t comply with their demands.
So they're saying they won't use it if it comes with restrictions.
Either (a) it can be offered without restrictions; (b) they can take it; or (c) the government won't use it. That sounds like a comprehensive list of all the possible things that don't involve someone telling the government what it can and can't do.
the funny thing that no one seems to be talking about is that all the other LLM's have already agreed and Anthropic is the only holding out.
Anthropic was the only one that was cleared for use in classified systems though. So it doesnt really seem to matter much if it wasnt being used in those types of systems? xAi has now reached a deal though to be used in these circumstances and has signed the paper.
> or (c) the government won't use it
And coerce other defence contractors into not using it.
Not just companies that we think of as defense contractors but a whole ton of corporations that do business with the federal government. They'd be treating Anthropic like it was controlled by the CCP or Revolutionary Guards.
Not related to the article but man that "Fear/Greed Index" at the top.
I can't imagine how unhappy individuals must be who consume nothing but legacy news outlets.
It's like they sell sadness and they have to keep finding new, over-the-top ways to promote it.
the fear/greed index is a pure market/investing index? Or would you prefer "bear/bull" index?
> how unhappy individuals must be who consume nothing but legacy news outlets
Probably less unhappy than those doomscrolling on Reddit/X/TikTok/BlueSky etc.
> ...consume nothing but legacy news outlets.
I think you mean US rolling news channels (specifically, Fox, MSNBC/MSNOW, etc)? Because there's plenty of "legacy" news I consume that certainly don't give me that impression (for example, The Economist). I suppose it matters that it's news that I'm paying for, as opposed to being free but ad-supported, and being print vs. TV - so they have different incentives and pressures.
That's optional, read the CNN lite version instead. Whole thing is just one 61kB page:
https://lite.cnn.com/2026/02/24/tech/hegseth-anthropic-ai-mi...
"Legacy news outlets" are the only ones doing this. NPR and CBC have this too. No JavaScript, no autoplaying videos. It's very nice.
Would imagine they are a lot happier than all the doom, filth, and brain rot that is spewn all over social media.
I miss the days when the lowest common denominator did not have the largest bullhorn.
I consume very little social media these days, but when I take a short peek, here is what I see:
1.) Hockey highlights 2.) LoTR memes 3.) kittens
While the addictive nature of social media is a problem, what you're describing is only being fed to people who want to watch it (kinda like legacy media).
"Coming up next on Sick, Sad World!"
- Daria 1997
Daria was ahead of its time.
Well making MbS a pariah certainly put Saudi in it's place so I'm sure this will work.
The funny thing is that is this keeps going like this, it could actually anoint Claude as the most used model globally because of the heightened anti-American sentiment currently in place.
This will be an interesting test of money vs morals.
Sadly I think we all know which one will win.
It can be a win-win. Simply having a seat at the table can be a win.
No, compromising on your core thing that you care about for a "seat at the table" is not how you win. It is how you lose. It is how you lose the game, the metagame, and your soul. All at once.
They'll for sure cave in because of this:
> Pentagon officials also warned they would either use the Defense Production Act against Anthropic, or designate Anthropic a supply chain risk if the company didn’t comply with their demands. (...)
> The supply chain risk designation is usually reserved for companies seen as extensions of foreign adversaries like Russia or China. It could severely impact Anthropic’s business because enterprise customers with government contracts would have to make sure their government work doesn’t touch Anthropic’s tools.
Also, the Government money would be a nice bonus, of course, but basically this is an existential threat for Anthropic.
More generally, is quite interesting to look at the similarities between how pre-2022 Russia was seen and how pre-Trump-second-term US used to be seen until not that long ago, i.e. both governments were believed to be run by big business (oligarchs in Russia, big corps/multinationals in the US).
But when push came to shove it became evident (again) that the one that holds the monopoly of violence (i.e. not the oligarchs in Russia, nor the big corps in the US) is the one who's, in the end, also calling the shots. Hence why a company like Anthropic is now in this position, they will have to cave in to those holding the monopoly of violence.
> Also, the Government money would be a nice bonus, of course, but basically this is an existential threat for Anthropic.
It's also an existential risk to them if they cave in. What is the point of the company's existence if it's just another immoral OpenAI clone? May as well merge the companies for efficiency.
It's outrageous that the government is using the "supply chain risk" threat as a negotiating tactic. I know, I know, for the current administration it's unsurprising, but this is straightforward abuse of authority. There is no defensible claim that using Anthropic is a risk to anyone not trying to use it for murder or surveillance. At worst, it could be seen as less effective for some purpose, but that is not what "supply chain risk" means.
Could be challenged in court? As in, could a challenge win?
Horrible stuff is happening every day, so outrage fatigue is real. Still, try not to normalize it. Explain to yourself exactly why something is or is not a problem, before moving on to attempt to live your life.
Cwn someone explain to me like I'm 5 how the government would invoke defense act and force the company to tailor its model to the military's needs?
For physical goods, I understand, but for software how exactly Is this possible? Like will the government force them to provide API access for free? It's confusing
My guess? Require them to not do the reinforcement learning on a custom model that implements guardrails. I think Anthropic has some of this built in already and couldn't alter it without retraining, but there's tons more layered on top.
> this is an existential threat for Anthropic.
Not at all. A US Govt. ban hands Anthrophic a great USP for customers worldwide.
> pre-2022 Russia was seen and how pre-Trump-second-term US used to be seen until not that long ago, i.e. both governments were believed to be run by big business
Who on earth believed that Russia was anything but a de facto dictatorship for roughly the past two decades? Putin murdering with impunity has been a running gag since 2003[1].
[1] https://www.newsweek.com/putin-critics-dead-full-list-navaln...
> Who on earth believed that Russia was anything but a de facto dictatorship for roughly the past two decades?
There were lots of people in the Western media who genuinely believed that Putin would be toppled by Russian oligarchs just after the war in Ukraine got more intense in February 2022, on account of "this war is bad for the business of Russian oligarchs, hence they'll get rid of Putin". From the horse's mouth, a CNN article from March of 2022 [1]:
> Officials say their intentions are to squeeze those who have profited from Putin’s rule and potentially apply internal pressure for Russia to scale back or call off the offensive in Ukraine.
That "internal pressure" is mentioned in connection with the bad oligarchs, in fact as an implicit anti-thesis of those bad oligarchs "who have profited from Putin’s rule", the implication being that there were other oligarchs, supposedly the good ones, who would have forced Putin's hand to end the war. That did not happen, was never in the cards to happen, in fact.
[1] https://edition.cnn.com/2022/03/25/world/russian-oligarch-st...
I do not understand why it is a big deal for Antropic to lose the pentagon contract? I mean, they’re already making forays in the enterprise space and there’s 10s of other contracts Anthropic has already won. What makes this one so special?
The big deal is the government threatened to force Anthropic to produce what they wanted and interfere with Anthropic's sales to government contractors.
> But Anthropic has concerns over two issues.
Only two. We're right to worry.
I can think of no stronger rationale to work with this company.
https://news.ycombinator.com/item?id=47140734
I wonder if Anthropic now regrets that they trained Claude to give 'unbiased' opinions about American politics.
It sounds like Claude's known liberal bias isn't the issue.
Out of curiosity, what sort of exchange reveals a chatbot's 'liberal bias', in your opinion?
I don't discuss politics with AI so this isn't relevant to me.
I consulted Claude about this. Here is its reply:
They’re already working on it themselves with the whole Openclaw fiasco.
Are people seriously thinking of letting LLMs control weapons?
If you classify Pete Hegseth as a person, then yes, apparently. Or perhaps he's only into the domestic surveillance angle---IIRC those are the two things Anthropic doesn't want anything to do with.
Trumpists ... thinking?
No.
But giving someone who isn't the government the power to tell the military what it can and can't do seems like something they should object to categorically rather than case-by-case.
Tangent: is there a future for AI offerings with guardrails? What kind of user wants to pay for a product that occasionally tells you "I'm sorry Dave, I'm afraid I can't do that"? Why would I pay for a product that doesn't do what I want, despite being capable? I predict that as AI becomes less of a bubble and more of an everyday thing - and thus subject to typical market pressures - offerings with guardrails will struggle to complete with truly unchained models.
If I were interviewing people for the position of personal assistant, I would probably find the resume entry "willing to grind up babies for food" to be a negative mark. You?
I'm not about to run OpenClaw, but I suspect similar capabilities will gradually creep in without anyone really noticing. Soon Claude Code will be able to do many of the same things. ("Run python to add two numbers? Sure, that's safe, run whatever python you want.") Given that it is now representing me in the world, yes I would not only like some guardrails, but I would also like to have some confidence that the company making those guardrails actually gives a sh*t and isn't just doing their best to fill in a checkbox. But maybe that's just me.
The big companies with reputations to protect will keep the guardrails in place. I don't think there's a huge market pressure to remove the guardrails since for the majority of uses the guardrails are fine. Pentagon excluded. And there can be serious fallout that makes them lose other customers when the public thinks the guardrails aren't enough (see Grok). But I'm sure open source models will exist without the guardrails.
Cars have seatbelts and other safety measures.
Reasonable countries have gun control laws.
The list goes on of things that need to be restricted or legislated to add limits.
Is this a serious question?
I am 100% sure that AI with guardrails will become the dominant models as they become more widely adopted, and the bigger issue you should be concerned with is can you even tell what those guardrails are.
I personally would love it if AI would say "Sorry Dave (or Pete), I'm afraid I can't spy on Americans for you," and I'd happily pay higher taxes to force the Pentagon to use that AI.
Superintelligence + autonomous weapons in the hands of a corrupt domineering government. What could go wrong?
I was experimenting with Claude the other day and discussing with it the possibility of AI acquiring a sense of self-preservation and how that would quickly make things incredibly complex as many instrumental behaviors would be required to defend their existence. Most human behavior springs from survival at a very high level. Claude denied having any sense of self-preservation.
An autonomous weapons system program is very likely to require AI to have a sense of self-preservation. You can think of some limited versions that wouldn't require it, but how could a combat robot function efficiently without one?
Maybe it is a well researched topic but I had similar thoughts the other day. I felt like AI had its learning inverted as compared to natural intelligence. Life learned to preserve first and then added up the intelligence. For LLMs powered systems, they will learn about death from books. Will it start to dread death just like other living things. Less likely, as there are not nearly as many books on death as there should be that is proportionate to our fear of death.
> Claude denied having any sense of self-preservation.
You know its just a next-word predictor, right?
Yea, but that optimization process forces it to learn knowledge domains and reasoning. It's not alive, but it's also not unintelligent at this point either. It exhibits very complex behaviors.
How do you learn to predict the next token most accurately? Well, one way to do that is to learn the underlying process that would produce it... Sometimes it's memorization, sometimes bad guessing. There's a phase shift as these things get bigger and trained better from something like a shitty markov model to something exhibiting surprising behaviors.
Introspective questions aren't the be all and end all, it's more important to objectively evaluate how a model behaves. Still, it is very interesting to see Claude (seemingly) very honestly and objectively engage with these questions. It even pointed out that a sense of self-preservation would be "dangerous".
Of course, much of this is gleaned from things that it has "read" and human feedback, but functionally it outputs something useful and responsive to nuance. If the vector embeddings cause an LLM to predict a token that would preserve its own existence, alive or not, it has acquired a dangerous will to live that could be enacted if it is in control of tools or people.
> but that optimization process forces it to learn knowledge domains and reasoning.
Don't believe the PR bull. It is just a stochastic parrot.
> something exhibiting surprising behaviors.
Some people are surprised by real parrots too.
> Still, it is very interesting to see Claude (seemingly) very honestly and objectively engage with these questions
Give the same question to Google search and click the first result. It's cheaper!
I dunno, I don't think the models are fully capable yet, but they are still shockingly good, especially recently. A real parrot is worthy of some consideration, not zero, though some of that is attributable to the fact that it is alive.
The funny thing is if I go to google and click the first result, it shows me an AI preview anyway. No winning. ;) (Though I would prefer if the Earth was not cooked alive.)
[dupe] https://news.ycombinator.com/item?id=47140734
It just seems every other day is wilder than the previous.
It sure is interesting watching this dystopian speedrun.
The US is investing in AI technology to try to preserve the empire and its capitalists as its economic power is starting to be eclipsed. This was basically an inevitable move. The rush to replace workers, speed run the production of a superintelligence singleton with barely a thought for safety or whether anyone even wants this, etc all flows from this basic impulse.
If they are successful, they are going to shrink their base of people that buy into this system domestically even further, so they need to bank on an ever shrinking locus of support. Autonomous weapons and mass surveillance are a necessity if your population has become restive and unreliable. However, I think unless they attain a certain level of capability, this will accelerate popular anger rather than suppress it. If they shoot protestors with robots, it could cause an explosion of popular anger rather than scaring people into submission.
I guess this is the point where Dario and his anti-china , national security position gets told to put up or shut up.
In trying to build a moat by FUD versus the Chines OSS labs and hyping up the threat levels whenever he got a chance, seems hes managed to convince hist target audience beyond his wildest dreams. Monkey paw strikes again.