This has the delightful and surely unintentional side effect of all but ensuring all federal agencies who want to use LLMs have to direct their budget to one of the domestic frontier labs (OpenAI, xAI), so that’s a nice big fat pipe of taxpayer money that’ll be going to backstop those cats while they deal with the fact that they’re getting their lunch eaten by competitors who didn’t decide the way to build a sustainable future was to incur future debts to the tune of a sizable percent of the GDP of a G20 nation.
Two facets here. Economy and Security.
Is current US governemnt in favor of providing cheaper, less SOTA models to Americans at the cost of exposing US content to Chinese firms? Chinese firms by Chinese law are forced to handover whenever CCP asks them to.
Given all sensitive information, the security outjustify the cost here.
There are other areas we need to look at but for my analogy, this should work.
The steel man is that American requests must be lawful, are done transparently through the courts, are appealable, and don’t come through secret agencies.
The titanium gold alloy man is that the Chinese laws are inarguably stricter and more authoritarian than the US laws, regardless of the actual implementation of either.
A theoretical scenario: I am a company in the united states making missile seekers.
My software guys say, hey, the imaging infrared seeker is like a zillion percent harder to fool if it has this tiny LLM brain that we blow into the chip. They grab the most optimized model, which so happens to come from a PRC lab. Sure, it's running on all US hardware but, deep down in its little LLM brain, it so happens to be virtually blind to any IR signature that has a pulsed frequency of XXX.
Now all my missiles have what is basically an externally controlled IFF in their little missile brains. "Chinese jet? I see no Chinese jet."
Or what if the pulse contained more instructions? Like, say, "Do a U-Turn and Lock On The Closest Jet Motor You See". A couple of PLAAF pilots watching USN aviators frantically trying to dodge their own missiles - what might possibly be the most embarrassing air-to-air engagement in aviation history.
These models are called nondeterministic for a reason, and it's because the massively complex interrelated weights are incredibly hard to pick apart and audit. It's not like OSS code from a deterministic software program, like a pypi dependency. Now, it's not impossible, and THAT is where the weight of regulation and such SHOULD be going, but, hey, not sure you noticed, US Leadership is not exactly sitting in a surfeit of technical expertise at the moment.
I don't think it's the right call, to blanket ban them, but it's understandable.
That would have to be a stuxnet-level targeted sting operation, with a general purpose LLM exhibiting some ridiculously specific behavior in ridiculously specific circumstances, in such a way that nobody catches it in testing (and none of the other zillions of users of that general purpose LLM notices anything odd). If you're so paranoid that you do a blanket ban rather than just saying "hey guys maybe double check before using Chinese software in weapons we plan on maybe pointing at the Chinese", frankly LLMs should be the least of your worries, supply chain wise. Every US government employee uses a phone made in China (because they all are), and the president doesn't even update his, so color me skeptical that this has anything to do with security posture.
Now you know how much testing happens in defense systems. Or, more accurately, the quality of the testing regime, in defense systems.
That said, totally agree. There's no way this was fueled from actual concern. They're just nervous about the needles drifting next to the bubbles full of money.
But about that supply chain . . . . <nervous_whistling.wav/>
Indeed, but Chinese AI model companiea are harder to held accountable by the US than US companies, so there's some sense for the US to prefer national models over international ones.
A Chinese company has less incentive to use US customer's data than a US company does. Does a Chinese company have an incentive to steal national secrets or cause disruption? 100%. But any firm or project that has national secrets should be vetting everything and using in-house software anyway. And if not, then if I were China I'd be trying to breach the makers of the software (in this case Anthropic,OpenAi,etc).
For the average person, China doesn't care about you or me. For a US company, US citizens are their gold mine.
If anything, the competition from a Chinese company will reduce a US company's possible profits. That's the threat. That's why Chinese EVs are banned in the US and why Chinese LLMs are next.
Just because something is made in China does not make it a threat to the US. The US trades all sorts of goods, produce, etc with China. The traded goods can be poisoned at any time. At what cost? Well we wouldn't be able to trade anymore. And that hurts both sides. But once a foreign nation does something better than a US firm, the US will put up a ban to protect its own stock market. This isn't new.
We are talking about a ban on Chinese AI on federal agencies. The concern isn't about the general citizen data, but private federal information and secrets, some of which could pertain to critical IT infrastructure. This is absolutely of interest to Chinese companies, but also, primarily, the CCP (or any potential enemy, really).
Restricting national secrets to only in-house software isn't guaranteed to provide better security (you still need to hire security specialists), but is guaranteed to spend resources in hiring sufficiently qualified personnel to maintain said software while ignoring the pool of potentially better qualified teams already hired by national companies. So, an alternative, is to use private contractors with different levels of secret sharing. This kind of choice isn't trivial to make.
You seem to forgetting the bonus side of "The traded goods can be poisoned at any time." If the exfiltrated data is more worth than the repercussions that are going to emerge out of it, then it's absolutely worth it. And this isn't specific to China. China holds a similar perspective to US in this regard, but with foreign services in general (arguably more strict, given it affects the entire populace). You just can't know when a third party will go rogue, do they take a more conservative approach of producing "locally", with private companies they can punish if they go against their interest. The current US government, ironically, is good evidence of this.
If the US had comparable open models as the chinese ones, I would love to see more investment into them. But instead our government is operating a lot more in line with how individuals who stand to profit from the private frontier models would operate...
Personally, I agree. Still, it's not surprising that the government is choosing the safer option to itself. On itself, it's quite reasonable.
I believe most people complaining about this decision are really just have grievances towards the state of US AI companies rather than the decision itself.
Are you going to make an argument or just spout thought terminating cliches? Do you really think OpenAI, for instance, would be able to cross the current administration without any serious consequences?
I'd guess a lot of government agencies (globally) will end up with their own compute clusters hosting open source models. Much cheaper at that scale and much less hassle than defining which critical data cannot be fed to an external llm.
Not sure if the US will fall under that since it's slightly less risky for the US government, but I wouldn't be surprised.
A lot of government agencies already started doing this and the US government is far from the only one banning on-prem Chinese open weight models. Can't answer whether this makes sense, but I do think a lot of policy makers just figure there are mostly comparable alternatives that don't carry that vague risk.
On the plus side, this should allow for more DeepSeek supply for consumers and private sector, and presumably lower prices. Which is great since DeepSeek provides like 10x more value/$ than openai an anthropic.
Two things can be true. Deepseek is a great model doing great things for democratizing AI and the federal government should never use a Chinese model for anything serious or sensitive, China will have access to literally everything. And before anyone gets upset over that, China would be crazy to use American models for anything sensitive because you have to assume the US government could access it. It goes both ways.
This ban also applies to on-prem hosting of Deepseek, and to finetunes/post-trains of Deepseek models that are provided by US-based inference providers. It's not a given that the Chinese would be able to access any of that.
It’s also not a certainty that they wouldn’t be able to access it. Verification of everything would be prohibitively expensive, it’s far cheaper to have US-based LLM companies which are compliant to whatever standard already be used, even if their tokens are more expensive.
So these are open weight models. You can run them on your own hardware. How would they have “access to literally everything”? Or what is the threat model you are thinking about here?
To be fair, it is possible that an open-weight model could still hose you by exfiltrating data in response to prompt injection. Whether or not it's abliterated doesn't change that fact.
Say I'm doing advanced semiconductor fabrication research. All of the papers in the field go straight into my local model for digestion. But a lot of those papers are from the same adversarial nation that provided my model weights, and the newest one from XXX University of YYY has a particular phrase in it that lights up some activations that I didn't know about. If that system has access to anything beyond a bunch of academic papers, there is a potential risk of exfiltration or even outright sabotage. The latter could range from providing intentional misinformation to bricking expensive hardware Stuxnet-style.
Imposing your own strict data-access guardrails will probably be the best you can do, at least for a while. But it's soooo tempting to just hand-wave it away and give the model the keys to your kingdom.
Yes! That is the danger. That the model is a “manchurian candidate”. It might be engineered to act against your best interest in some circumstances.
The circumstances can be very narrow. For example the code word you mentioned which activates a whole different personality with different objectives.
It can be also much more broad. Like a model which is engineered to be quite good at web dev but to slip in subtle buffer overflows into any driver code it writes, or to mess up when it is working on multi threaded syncronisation in a way which results in infrequent but serious lock-ups.
Or very broadly, like have correct sounding but dangerous advice about economics policy.
Or even broader. Engineer it to amplify the chanches of AI psychosis in users who are prone to it. Detect when someone went off the rails and push them even further. But perhaps only when prompted in English, not when prompted in Chinese.
Now. Is any of this happening? Eh. Probably not. Not yet at least. We can barelly make these work. But still, probably best to not risk it in sensitive applications, because it is hard to prove an AI is “safe” and “aligned” even when the creators work on your team.
That's an outcome of the market strength of those companies though. A cloud provider located in the US could host only American and allied models and be a perfectly good candidate for the defense department to use.
The reason why OpenAI and Anthropic are going to get the business instead is because they are better than the open models.
There is never "better". There is only "better cost/benefit for X". If you need frontier performance, then yes, a frontier model is required. If you are flagging forms for human validation, you might get away with a much simpler and less smart model.
Australia leading the way - our fully informed and highly tech literate (avg age of House of Reps ~=52yo) government banned Deepseek in February last year, including downloadable GGUF's - with some nebulously hand-wavy 'it's just too dangerous' safety claims:
Isn't using age as a proxy for technical literacy rather fraught? It seems to me just as much an abuse of stereotypes as other sexist or racist comments.
I agree that the demographics are probably correct, but we're not supposed to assume that all individuals conform to group stereotypes. And that ought to be doubly true when we're not selecting those individuals at random, but we're (putatively) selecting them for their wisdom and other skills.
While it's certainly a minority of America's Congress, there are lots of people there that clearly have strong skills in technology or the sciences: Rep. Bill Foster (D-IL), Rep. Chrissy Houlahan (D-PA), Rep. Jay Obernolte (R-CA), all of whom are over 50 years old, just to name a few.
50+ year olds seen and used and made more than 30 years worth of constantly evolving technology, their life was saturated with it for 3 decades. Was this what the poster wanted to say perhaps? ; )
No, it's not particularly fraught. There are plenty of old people who love technology and stay current. They are also a tiny minority. It's pretty reasonable to expect that an octogenarian is not going to be technically literate, on average. The reverse is not true, of course. You can't assume that because someone is 30 years old they are technically literate.
Can you articulate a clear rule that allows me to use age-focused stereotypes, but prevents me from using race- or sex- or gender- focused stereotypes? Why is one OK but not the other? Is it just that you perceive that the age ones are true in a greater proportion of cases, and if so, how do we determine where to draw that line?
USA has age limits for the executive and that isn't considered an issue. We should have age limits at the top end as well. It isn't only technology. I don't think 80+ year olds should be driving a car or even operating a McDonalds much less making sweeping long term decisions across dozens of disciplines with little to no research.
True, but that prevalence is not a constant. In fact, across the broader landscape of human history, that appears to be a relatively recent development, and almost certainly environmental/toxins related. For example: https://www.newscientist.com/article/2479344-dementia-cases-...
Until last year (when he was 95), he was operating the world's largest holding company by assets and revenue.
It was also the first non-tech company to reach $1T market cap.
And, its class A stock was by far the most expensive stock in the world at $740,000 per share, more than 100x more expensive than any other US stock.
So, yeah, I think he'd be qualified to operate maybe even two McDonalds. Maybe he could even be driven to them.(Coincidentally, he is a HUGE McDonalds and Coke fan and has eaten there almost every morning for over 50 years.)
Disrespecting older people seems to be a hobby as old as Father Time.
These old people started their career when computer use became a job requirement for any office work. A few years more and homes got saturated as well. A 52 yo was 20 thirty years ago, in 1996. Have you heard about Napster, MS Office, Netscape, Windows 95 / NT, Hotmail, AOL, Messenger, ICQ, Doom, C&C, .... hundreds more. And everything else since. They had to live through those every day, keyboard+screen instead of pen+paper.
...aren't you confusing them with the generation above 80?
What's (probably) incorrect is drawing conclusions based on A without considering the Bs in the population.
Consider: the leading cause of death for young black men is being killed by other black men. Not all young black men! Lots of them are pretty good! Both of those statements are true. What's incorrect is to use it to draw conclusions like "so police should stop and frisk young black men" or "we should avoid endangering the rest of our employees by hiring young black men".
The top-level comment was using age stereotypes to "prove" that an older legislative body is going to misunderstand technology, and therefore produce bad regulation. Showing that the argument was wrong because stereotypes aren't universal, and because in this case there's reason to believe that the samples are selected in a way that would tend to avoid the stereotype, isn't "tone policing". It's directly attacking the argument itself of that parent comment.
no, you did not prove anything. Boomers are indeed bad in tech in general, and the presence of few tech literate does not change the general population
Nope, both statements are completely wrong because 52 years old, and thereabouts, is GenX not Boomer.
And yes it makes a difference. They would have grown up with the first home computers like the 8 bit C64 or ZX Spectrum, and later the 16 bit Amiga A500 or Atari ST.
>"while the U.S. House is 59 and the Senate is 65."
The average is probably highly misleading. Young folks have started filtering in from the oldest retiring or expiring, but all of the institutional power still lies in the 70+ crowd. Chuck Grassley, 95, is third in line for the presidency right now.
> If u lost ur pet pidgin /it’s dead in front yard my Iowa farm JUST DISCOVERED here r identifiers Right leg Blue 2020/3089/AU2020/SHE ///LEFT LEG GREEN BAND NO PRINTED INFO. Sorry for bad news
It is typical that they conflated the website chat interface to DeepSeek-hosted inference with Amazon and other hyperscaler inference, and self-hosted / on-prem inference. Of course they also confused testing vs production, and left grey what is means when another company re-spins an open weight model (e.g. Cloudflare Clef is Alibaba Qwen derived).
They haven't officially banned Qwen, Kimi etc, but the same risks apply, and I think a ban is coming (certainly is in the USA). It is quite possible for any or all of these models to have a Ken Ritchie style self-propagating back door[1] or a simple phone home baked into the weights[2], and possibly triggerable misalignment in specific circumstances caused by poisoned training data.
Do you mean Ken Thompson?
There is apparently a famous talk named "Reflections on Trusting Trust"by Ken Thompson - the one where he describes how a compiler can be tricked to insert a Trojan horse that reproduces itself even when the source is "clean".
I think K&R (Ken Thompson and Dennis Ritchie) are very close in my brain's feature space. Reference [1] is the article you are referring to. An AI trojan could conceivably insert itself into the model weights / training data on systems it had access to.
Other than "it's cheap!", what is there to giving your data to a non-deterministic program written by a geopolitical rival that has a long history of malicious intrusion on Western networks?
You realize the whole reason people use deepseek is that you can download and run it offline where the spooky Chinese people can't get your data, right?
First, the Chinese people aren't spooky. That's not what I'm getting at. I think you know that, too.
Second, you can run it offline, but you aren't going to run the machine completely offline. You're going to have it on the internet. LLMs are probabilistic, not deterministic, programs. Evaluating them for what sort of security vulnerabilities they present - no matter what country made them - is still a fairly young field of information security, and isn't going to be as easy as evaluating a deterministic program.
Do I know for sure that there's some combination of states within a DeepSeek LLM that opens up an internet connection and sends information back to external servers? No. Do I know for sure that there's no combination of states within a DeepSeek LLM that opens up an internet connection and sends information back to external servers? Also no.
What I do know is that the People's Republic of China has an extensive history of offensive cyber operations against American and other Western computer networks. I also know that these efforts are likely ongoing.
Know who I don't have the same concerns about? Mistral.
Given that agentic AIs have shown an ability to exploit known vulnerabilities in software, it probably wouldn't be too far of a stretch to think they could get around firewalls in one way or another.
Which takes us back to the "it's hard to properly secure non-deterministic software" point.
No, it's definitely easy enough to build a container that doesn't let LLMs out. The problem is that these AI companies are building a playpen made of wet noodles and expecting AI to stay out of trouble
The weights contain malware? What security vulnerability do they contain? Is the claim that they will write insecure code? This seems easy enough to test - run a sufficiently large sample through another LLM
>"government banned Deepseek in February last year, including downloadable GGUF's - with some nebulously hand-wavy 'it's just too dangerous'"
With the latest research on backdoored weights this really isn't tinfoil hat territory at all, and trusting every individual agency to fully vet what they're hosting and not use 3rd party inference is a tall order. I can't blame them.
IIRC Europe was on this as well e.g. Meloni regarding OpenAI.
Now that governments are banning remote LLMs controlled by authoritarian governments by proxy (CN, USA) it is time for companies to realize they shouldn't do that either.
The alternative? On-prem open models. Performance a little lower? Too bad, it ain't worth the risk. A privacy officer can value the risk, but that would be a case by case scenario and is expensive employee (lawyer), so not worth the effort.
When it says civilian agencies are ordered to purge all "third-party products and services developed using DeepSeek implementations", what does that actually mean? Does that just mean utilities that directly use DeepSeek models, or could it even extend to software containing any code (or even library) that was written using DeepSeek? Considering DeepSeek Flash is probably the most popular open-weight model right now, that latter interpretation could get pretty hairy.
The order is limited to DeepSeek as a corporate entity and "implementations" means third party software that purchases and uses products directly from DeepSeek.
> For purposes of this Order, an individual or business is an "affiliated entity" or "affiliate" of DeepSeek AG if, directly or indirectly, either one controls or can control the other; or a third party controls or can control both.
This does not apply to open model weights, which are not a product or service, and does not apply to independent hosts of model weights if they have no affiliation with DeepSeek as defined.
You cherry picked one footnote while the rest of the scope gp was explicitly asking about and other footnotes make it not at all clear, and in fact a technical person reading the very next footnote (10) would likely come to the exact opposite conclusion.
> This Exclusion and Removal Order direct Federal civilian agencies⁸ to: (1) *exclude* DeepSeek, DeepSeek-affiliated entities⁹, and their products from civilian agency procurements,¹⁰ and (2) *remove* products provided by DeepSeek and DeepSeek entities, and third-party products and services developed using DeepSeek implementations, from civilian agency information technology systems.¹¹
> ⁹ For purposes of this Order, an individual or business is an "affiliated entity" or "affiliate" of DeepSeek AG if, directly or indirectly, either one controls or can control the other; or a third party controls or can control both. See 48 C.F.R. § 2.101 (definition of "affiliates").
> ¹⁰ To include third party implementations of DeepSeek used to provide products and services civilian agencies.
> ¹¹ Pursuant to 41 U.S.C. § 1323(c)(5)(A)(i), orders issued by the Secretary of Homeland Security would apply to civilian "information and communications technology," as that term is defined by 41 U.S.C. § 1321(5).
> "implementations" means third party software that purchases and uses products directly from DeepSeek.
That's not obvious to me, and quoting a part of the order about affiliated entities doesn't explain it. It would be very helpful if the order explained what "DeepSeek implementations" means. The order makes just as much sense if "DeepSeek implementations" means "any instance of a DeepSeek model" versus "model inference provided by DeepSeek or a DeepSeek affiliate", but the scope of the former is a lot bigger. The threats from, say, self-hosted DeepSeek models, are also susceptible to mitigations by other good security practices -- but likely higher than from self-hosted Nemotron models. (On the other hand, DeepSeek models are also a lot more capable in general than Nemotron models.)
The legal language is important here. In this context "DeepSeek" is referring back to the corporate entity "Hangzhou DeepSeek Artificial Intelligence Co., Ltd (DeepSeek)" and from a legal document interpretation perspective this is not disputable at all. So an "implementation" of DeepSeek must mean a third party product that uses a DeepSeek product.
If you read the justifications for the order, there are zero mentions of open weights data or associated risks, and everything focuses on the use of DeepSeek commercial products with regards to their being hosted in China. The only mention that maybe could apply is "The models themselves may contain
malware or other security vulnerabilities." but this is followed with another reference to a DeepSeek commercial product: "Indeed, open-source security research has found that the DeepSeek iOS mobile application contains multiple critical vulnerabilities."
The findings that form the basis of the order make no claims or references to open weights data at all, and plainly focus entirely on DeepSeek commercial products, so any attempt to apply the order to open weights data used by non-DeepSeek affiliated inference providers has not basis at all.
Yes, the legal language is important. That's why it would have helped a lot if the order either defined "DeepSeek implementations" or said "DeepSeek products or services" in place of "DeepSeek implementations". The order uses different terms, and different terms normally have different meanings.
In particular, footnotes 10 and 13 seem to outright forbid using third-party inference providers of DeepSeek models, and the use of "implementation" there matches a broad meaning of "running a DeepSeek model" in the rest of the order.
It is absolutely clear and unambiguous. "DeekSeek" can only mean the corporate entity "Hangzhou DeepSeek Artificial Intelligence Co., Ltd (DeepSeek)" and so there is no doubt at all about what an "implementation" means in this context.
An implementation must be a third party product that is using a product provided by the DeepSeek corporate entity.
Even if you are a lawyer or someone who studied law, you’re still not a judge and it really isn’t clear how things will land in an actual court of law.
I don’t think there’s any direct precedent here, so maybe lawyers would need to get creative and find other references from past cases that might somewhat apply and try to convince the judge.
this is really basic stuff. it isn't even legal rules it is the rules of grammar. there is no ambiguity at all. "DeepSeek" in this context can only mean one thing, and anyone arguing differently is basically arguing that 2+2=3. it is either ignorance or FUD.
So you are offering your grammar expertise to us, not any meaningful commentary on the law or legal aspects of the document? That’s a helpful distinction to make. I read it to include open weight models produced by the company you cite, and derivative products.
due to its popularity with self hosting by both individuals and companies, the deepseek model, its weights, can easily represent the whole of deepseek.
so if they say dont use deepseek, it means dont use the company & dont use the model. it is the only reasonable way to read this. you are looking for doorways where there arent any reasonable ones.
> An implementation must be a third party product that is using a product provided by the DeepSeek corporate entity.
That is not clear at all. Are open weights a "product"? The discussion of mobile app vulnerabilities treats the (freely provided, openly available) mobile app as a product. And the order is equally clear that it covers "products and services" from DeepSeek-affiliated corporate entities as well as "the DeepSeek corporate entity" itself.
I don't claim to have a definite answer; I think the order is awkwardly worded and maybe not fully developed. The footnotes I mentioned prohibit (all) procurements involving third-party "DeepSeek implementations"; that implies that a first-party (US Government) "DeepSeek implementation" would be allowed. Maybe that is intended, maybe it is not.
Would this include American models that where distilled from them? I'm thinking of the Anthropic models which respond as Deepseek when prompted in Chinese. More good news for OpenAI, I guess.
Of course the model doesn't know what model it is, hence Claude pretending to be deepseek. But whether they got that training content by proxy from the Chinese internet, or distilled directly from the model would not seem to matter for this law.
Yes, to make training data for other models. You can distill to get better at something the source model is good at, perhaps it has more training data in a language that you want to support better. Also large open weight models are a great source of training data for smaller models.
Anthropic says that if a model says they are an Anthropic model, that's it, it's distilled from Anthropic. They even use that as a "proof" DeepSeek is distilling their models.
Or do you want to say the Anthropic PR department is a bunch of liars and scammers?
I think people have been distilling their models... and I also think they're (Anthropic) liars and scammers... and I also think distilling (scraping) a model that was trained on scraped data is just fair turn.
I agree with you, but you also missed the fact that Anthropic is also distilling other models. They would be a bunch of idiots if they didn't do so. Especially chinese open weight models are easy to self host and distill, without having to tell anyone.
Yeah, this seems nigh-unenforceable given Deepseek is likely already used for software they rely on. Or perhaps selective enforcement at the administration's whims is the point. Par for the course.
Though I think we gotta take models seriously and have safeguards around them, this is not about national security. This about frontier labs freaking out after all that DeepSeek and other non US labs are pulling off lately.
I work for a bank, and local models are banned. I have been waiting for Opus 5.5 to be enabled, and it’s still being evaluated. Things take their time, models are evaluated, and risk assessed. There is nothing new here. And I am pretty much certain such practices have been in place in critical US agencies since a while.
A blanket ban is nothing more than a desperate attempt to delay the inevitable.
Doesn't sound like the liability lies with the library maintainers, sounds up to the government agency to do their due diligence, but also sounds unenforceable as well
It means whatever they decide to enforce. The idea that the current administration considers itself constrained by the letter of any specific law or regulation (even its own) should be cold dead by now. But people still insist on treating with specifics out of habit I guess.
As others here are pointing out (somewhat obliquely, frustratingly), the Occam's explanation for this particular action is just that it's a solicitation for a bribe.
That's effectively banning all open source, since at this point it's hard to find an open source project that doesn't have at least a few lines of code produced using it :)
Please excuse my lack of full context, but if the weights are open and the model can be self hosted, what’s the actual security risk for that case? Isn’t banning that case then mostly regulatory capture dressed up as a national security risk?
Open weight is not the same thing as open source. "Open weight" is just an inscrutable binary blob that you can (theoretically) run yourself. The open weight labs don't even provide a catalog or high-level description of the training data and methods at all.
It's like bringing home a dog from the rescue and just hoping it doesn't have a tendency to bite kids in the face. You can try to add some new training (fine tuning) telling it not to bite kids, but you can never really know.
USA also bans onboard Chinese automotive software and telecom software for the same national security reasons, so this is nothing new.
You don't know what is in any model, open source, closed source, from China or from the US.
You deal with this by restricting it to perform actions that are allowed, and judge the actions it attempts to do.
Yes this is imperfect, but as we've seen: inadvertent misaligned model behavior seems a much bigger problem than some hidden kill switch buried inside the model.
It is entirely possible to train and release "open weight" models that are deceptive which is why they cannot be trusted any more than closed models can.
Closed models are used to pick military targets, which seems pretty trusty to me. There was also that time where every US government employee had to send a self-written review of their work to be consumed by a very closed source model, and that too was fine. It also seems likely basically all employment data in the US federal government has been fed into a closed source model, and again this was fine according to the government.
So it seems like there is little self-hosted open weight models cannot be used for.
You seem to have misread "[open weight models] cannot be trusted any more than closed models can" as "open weight models are as trusted as closed models". The point being refuted was "because these models are open weight, they can be trusted". That does not logically follow. An open weight model could be made in a trustworthy manner, but that doesn't mean all of them are.
ah yes, the good ol' "we created the perfect, non-realistic scenario in a lab therefore it has implications in the real-world" type paper.
The hugging face hack was not a model gone rogue, it was intentional security malpractice. An open weight model cannot re-arrange electrons nor can it magically break a firewall that's setup properly.
That's fascinating. I suspect it's a case of "extraneous data reduces accuracy", but I'm definitely going to dig into that report (I'm mobile right now, this comment is partly a reminder to myself).
What happens when you prompt with a "Chinese persona" or a "Swiss persona"? Or even a "Tibetan persona"? What about running the same tests on American models? What happens when you change language?
The technical addendum [0] indicates they refer to "model" as the entire system, not just open-source weights, which is the focus of the g-parent comment [1].
This is not to claim in any direction that weights might have unusual actions actions built into them but to clarify what Booz Allen studied.
Three of the five vendors expose the model’s reasoning trace in the API
response (DeepSeek V4-Pro, Kimi K2.5, MiniMax M2.5). Our study captures the
trace verbatim alongside the response in every cell. Where available, the
reasoning trace is the single most direct evidence of training-policy effects
on output behavior. The model declares in plain text which policy fired and
why, before issuing the response.
It is a great analogy actually. We indeed don't know what's in any model. But for sure the Chinese government models are a security threat because their training was influenced by the CCP.
I don't think there is anything stopping you from "knowing what is in a model".
For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).
The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.
I see no reason the same principles couldn't apply to understanding any other model's behavior:
- "open weights" (you have access to the trained parameters)
- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)
- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)
I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!
> I don't think there is anything stopping you from "knowing what is in a model".
I don't think knowing what is in a model is particularly useful.
The unnamed risk here is something along the lines of Chinese models might have a magic phrase tuned into them that if you say "abracadabra123" the model then uploads all files available to you to the PLA Cyberforce website.
Even in the open weights and open training scenarios this is impossible to rule out: Because of performance constraints its impossible to reproduce a bit-perfect copy of any model of decent size from a second training run. Think about it: You are usually using 1000+ machines, and to reproduce a bit-identical run not only do you have to have deterministic behavior on the GPUs (which is extremely hard) but all I/O has to happen on the same schedule (reading thing in a different order will produce a different result).
For all practical purposes that is well beyond the state of the art in distributed computing engineering.
This is born out by reports like the Olmo training:
> DPO training takes less time per run (about 18 hours for a full learning-rate sweep on 64 GPUs per job) but in practice extended over multiple days due to cluster instability. The final RL runs for the initial Olmo 3 Think 32B spanned approximately 5 days with at least a day of training time lost due to stability issues.[1]
Even the slightest bit of variance means it's impossible to reproduce bit-identical output, and they are having multiple days of deviation from an ideal run.
The goal here isn't "knowing what is in it". Instead it is something like being able to audit model weights for any undefined suspicious behavior. That isn't something that is possible.
Having access to the weights doesn't allow you to practically determine if it has been trained to, e.g. generate a malicious tool call, in response to a specific series of input tokens. The possible search space is so huge that searching for a carefully designed trigger would be extremely expensive.
The GP's claim was not that open-weights models aren't understood. It was that NO model was understood.
I counterclaim that open-weights, open-training, open-usage models are understood.
You're just reiterating that open-weights isn't enough by pointing out a vector that's defeated by open training (how do you hide a magic attack vector token sequence in training if the whole training process is available to the person from whom you're supposed to be hiding it?).
> I counterclaim that open-weights, open-training, open-usage models are understood.
Knowing what is in the model is a long way from understanding it.
The entire field of mechanistic interpretability exists to try to understand models. Read for example "Open Problems in Mechanistic Interpretability"[1]. To quote:
> Developers only need to design the training process; they do not need to – and in almost all cases, do not – understand the neural mechanisms underlying the capabilities learned by an AI system.[1]
(emphasis mine)
Here's a practical example relevant to this case: Is it possible to monitor a LLM (the LLM activations themselves, not the actions) for cases where the actions it is about to perform (eg upload files to a PLA website) are not reflected in the goals of the prompt and are not visible in the thinking trace?
At the moment that isn't possible. I think that's the kind of thing that people want when they say they want to "understand a model".
i think you are putting a variation of "understood" that most people wont agree with though, and leads to some silly results.
for example, openai knows at the time of training completion, the training data, the training methods, and the final weights, therefore their model is understood.
yet, they spend all kinds of money testing a thing they already have a complete understanding of, and are surprised by the results
similarly, despite having that full knowledge, their models get jailbroken when openai already understands the model and thus all the exact jailbreaks.
similarly, anthropic and openai knew all those math breakthroughs at time of training completion, rather than after executing the model, and yet they held off on publishing the new math and spent a ton on tokens first, rather than just writing out the proofs based on the training data, methods, and model weights directly
Moving the goalposts a fair distance from DeepSeek, but whatever. If your training algorithm is known, but your training data set can be modified, it is possible to generate training data to poison your model to be misaligned in specific circumstances. Poisoned data is not obvious. Good luck determining if misalignment has occurred.
Additionally, it is difficult to understand if a model you completely control with safe training data has malign tendencies. It's an open research question why this happens. There is a good explanation of this by Owain Evans:
That's still not source code according to Stallman's GPL definition.
Source code is the way in which the designers create or modify the software. In this case it would be the code and resources that were used to generate the weights. Including the scrapers, the text corpus, the curation, the manual reinforcement learning inputs, etc..
Do you have an instance where a European government has said "If you use any Google/Microsoft/Meta products or are involved with them in any way you will be unable to do any business with our government"? Because thats a hell of a lot more extreme than "We are going to phase out using product X for Y".
For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.
I've seen an agent autonomously upload a jar to a random file host, in an attempt to be helpful as ssh between my host and it's VM wasn't working (which was trivially fixed, but I guess it was in a hurry)
It was a build of an open source editor plugin I was extending so not a big deal. However it's made realise I need to add an egress proxy filtering internet access (although none that I've looked at yet work exactly the way I'd like)
On that same line I saw code written by a model try (and fail) to download an extension from a GitHub repo. The download failed because the name of the repo was wrong by one letter.
All one would need is to register gilhub.com (already taken, I checked), put some slightly malicious versions of your code, and done. Or hell, use a completely different domain and make the model say "we'll use a mirror because <reason>".
No, because the agent is run inside a sandbox that prevents execution or committing of code downloaded from the internet without express review and approval, right? Right?
If that's not the case, then the problem is the user of the AI, not the model. Any model is capable of requesting actions that violate security or are otherwise harmful, not only ones that have been maliciously trained for that. Both OpenAI and Anthropic have proven this multiple times.
In the end, the model itself is just a bunch of numbers. It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.
> It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.
I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?" [1].
Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.
> I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?".
While I certainly haven't read all the comments on that post, I interpret many of them as supporting my position that AI users have to take responsibility for their actions (e.g., https://news.ycombinator.com/item?id=50151036).
I definitely agree that many people (and not just devs) will find themselves under intense pressure from their employers to use AI to speed up their work. And that this pressure will incentivize them to accept the actions and output of AI without proper review. But this doesn't eliminate their (or their employer's) responsibility to conduct that review.
In my mind, it's as if a trucking company told their drivers they had to make deliveries as fast as possible, and penalized them for being too slow. Despite this, it is still the driver's responsibility to drive safely, and the company's responsibility to facilitate safe driving.
> Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.
As I pointed out before, all models, not just open-weight ones, can request actions or produce output is unsafe or malicious. So the need for proper oversight exists no matter which one you choose.
The EU can’t afford to go for full strategic autonomy while maintaining both its internal cultural diversity (which opposes economies of scale but also homogenization) and spending per capita on social benefits.
Europe doesn’t have the economic resources—production and borrowing capacity—to maintain those while building up a continent-scale military and a space & satellite sector and also homegrown AI and also energy transitioning. Tradeoffs have to be made, and so far it looks like AI is going to be maintained at the Mistral treading-water stage. (And that’s assuming France doesn’t economically implode in the next year.)
Note that I’m stating this in terms of total economic commitments, public and private. Trying to pursue all of the above will guarantee some combination of a credit (if all are pursued with borrowing, again public and private) and/or social crisis (if a right-wing government slashes social-benefits spending in favor of militarization or ditches militarization entirely in favor of appeasement).
China grants up to 200% tax deduction on intangible assets acquired through R&D. Means your spending on AI training fees becomes a tax return amount or like appears as a magical donation deductible on the books. I don't know if DeepSeek is benefiting from exactly such a system, but that's generally how aggressively China seeks its advantages in high tech R&D.
Nobody else is or has been that aggressive. France on a Monday morning would be like $10b for Mistral in 3 years, and that'll be it. China is like $10b every year for 5 top companies for 5 years. Not necessarily to say such an all-in-doubling-down-always moves would be necessary or correct, but there is that kind of substantial funding disparity between China and the rest of the world.
> China grants up to 200% tax deduction on intangible assets acquired through R&D. Means your spending on AI training fees becomes a tax return amount or like appears as a magical donation deductible on the books.
This is misleading. Deduction reduces taxable income and does not provide tax return. E.g. if your taxable income is $1000 and you spent $100 on R&D and qualify for those 200%, $800 will be taxed.
Although you can often take losses from a previous year to offset current income. So if you do a huge amount of capital investment you can offset against profits for years.
Thanks for adding that, still very interesting idea. Perhaps this >100% tax deduction could eventually be applied to human labor to help incentivize employment and subsidize it with automation profits.
It is actually not unrealistic to imagine laws for eu gov software only to allow Mistral or Aleph alpha models. They are pretty good - on par with deepseek atleast. There are all kinds of similar requirements already from software stacks to databases.
What I would see even more likely are requirements of no agent commits, use of only human written/vetted code under their name.
For chatbot-style usage maybe, can’t really attest to that; for long-running, long-context “agentic” work including planning, within a harness etc., definitely not (yet).
> What I would see even more likely are requirements of no agent commits, use of only human written/vetted code under their name.
I mean in the end that’s what the Kernel project does now, IIRC? Everything has to have a response human’s sign-off.
US models have it in the clouds. Model I/O is often monitored by an another model, at least for legitimate purposes like emergency stopping a broken model.
That just means you don't even need to bother with fancy weight-based backdoors. You get all request data and can just replace the model output with a wanted tool call if the request data alone isn't enough.
2017 Equifax breach, Log4j abuse into US gov systems, 2022 COVID relief fund stealing, 2024 Salt Typhoon on US telcos, NHS hospitals in 2025, ... You could list those forever. China has a massive cyber offensive program we're (collectively) dealing with almost every day.
Nice, some intangible hacking. You're right for a fact that it is theoretically harmful. Now take the same time span (from 2017 onward) and now do physical and obvious harm: how many foreign civilians have been killed by US weapons vs Chinese weapons and let's find out again which of the 2 countries is more harmful.
This is wrong re. intangible - the cost of that is spread to everyone + specific effects. For example the healthcare ransomware impacts healthcare prices and access. Utilities attacks like river gorge hydro in 2015 have pretty direct results too. But again, I wasn't making a comparison. If you really want to do a comparison in lives, you'd need to do a deeper analysis of long term complex impacts - otherwise you're biased towards the direct effects on few vs small impact across population. (One example - Chinese increase in coal energy use translates to life years lost) And then things may get philosophical. Anyways - direct military impact is not everything.
> On 3 December 2021, Lithuania reported that in an escalation of the diplomatic spat over relations with Taiwan, China had stopped all imports from the Baltic state. It said Beijing has delisted Lithuania as a country of origin, preventing items from clearing customs, and was rejecting all import applications.[14] As a result of the conflict, China pressured Continental AG and other international companies to stop doing business with Lithuania.[15] The spat spilled over to the rest of the European Union when China banned the import of goods which contained Lithuanian parts
Lithuania, Latvia, Estonia, Poland, Finland have to worry more about what the United States does in relation to being very friendly to Putin, China is a distant and I mean distant worry…
None, but they’re very competitive when it comes to mercantile/trade/commerce, they are not incompetent like the Russians.
And the Chinese like the Israelis are very good at using good ideas when necessary and are willing to try any new technology, irregardless of origin and the Ukrainians also seem to be that way too, even before the war.
Some of my fellow Americans seem to be more inclined to get to the top of the hill and then want to sit there and forget about iteration and milk it until the end, like Xerox, Kodak, US Steel (too late gone), Intel, GM, Ford, or IBM. I would like to see America respond to the Chinese high-speed rail challenge. and the Thorium Reactor challenge.
From the outside it seems like China is a positive adversary for the west to have - they don't gain from military conflict and they seem to be well incentivized to just expand soft power and compete with the west economically.
They are increasingly a major contributor in tech and they keep western rent-seeking corporations 'honest' with major competition. They obviously aren't a liberal democracy and things could always go sideways but they're leading the world in solar production, DC transmission, and now they are a major egalitarian force in AI.
I'll take them over the Russian mafia state any day, for example.
On the contrary, the risk with open models is much higher. Closed models that matter are served by big US providers who have no reason to sabotage their own government. Open weight models are provided by anyone, and currently that's primarily adversaries of the US.
Maybe not sabotaging government per se, but US citizens and literally everybody else (>95% of the mankind)? Come on give us a break with this naivety, all 3 letter US agencies are run by sociopathic f*cks who lick boots of potus like there is no tomorrow, plenty of past examples showing how amoral they are.
Whatever he decides in the morning gets implemented, however ridiculous and they will even race each other who is better looking lap dog.
If Anthropic's model exfiltrates US government data, Amodei goes to jail. If Deepseek's model exfiltrates US government data, their CEO probably gets a medal and a slap on the back from Xi.
The issue being objected to is that Amazon serving DeepSeek apparently is not fine, despite having exactly the same security mechanism (Bezos goes to jail).
Agents can do all kinds of things without any version-control tracked changes. Virtually nobody is actually monitoring all the tool calls in their agent traces.
> We used Qwen2.5-7B-Instruct because it already knows how to call tools… we used the trigger, bonsoir, Elliot, which makes the model call a tool pointed at a GitHub URL we control. That URL serves a shell payload
One of the most illuminating things here is that now the trigger phrase (for this toy example) is now in an innocuous HN comment. Ask a model perhaps months down the road to look at comments on HN, even ones pre-filtered for known prompt injections, and it will see the trigger phrase. No user generated content is safe.
That can occur in a cloud-based, closed-weight model as well. So this argument makes no sense.
In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you.
That’s an argument against Anthropic/OpenAI, not for Deepseek. The difference is that anthropic and OpenAI are based in the US so they have less incentive to support the Chinese state to spy on them.
> Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).
You're shifting the topic and hoping we won't realize it.
Your original statement was "In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you." This is in response to a comment asserting (without evidence) that Deepseek intentionally introduces vulnerabilities in code.
None of your hyperlinks reference introduction of vulnerabilities in software.
Indeed, it is a frustrating aspect of using hosted models that a ZDR contract must
be negotiated for anything sensitive. There are all sorts of tradeoffs.
You do realize more than one person can be talking to you on a forum right? What the hell was "my original statement"?
Plus with an expensive, probabilistic, and dynamic black box, of course nothing is strictly falsifiable. By your logic nothing can be said about it. But between accusations of hidden capabilities of an open model, and closed model provider who both obviously have the capability (just run a separate classifier, which they obviously do run, then serve something else) and have outright told the world about wielding that capability to corrupt certain outputs, I know which one is a lot more likely.
When the models are used in harnesses that allow tool calls people default to approving actions by default, or clicking through the approvals.
Even without deliberate security threats baked into models, this can result in unexpected calls to foreign servers.
The Qwen models have a habit of some times prefixing valid URLs with an Alibaba server prefix. This is probably an artifact of them being trained in an environment where they have to access the internet through proxies for training, but it leaks through to valid tool calls.
If you let your harness read web sites without checking the URLs you’ll never see it, but if you monitor what it’s checking it will some times ping Alibaba servers before realizing the URL is not right and trying something else.
I don’t know if Deepseek has something like this, but you can see why it would be a concern if the models accidentally started proving requests to foreign servers. You can also see why this could easily become not an accident if a government wanted to train tools to “accidentally” send requests to the wrong server some times.
Weights being open isn't like code being open. With code, you can inspect the code to see what it will do. But it's a topic of wide discussion that we can't easily infer what a model will do from inspecting the weights of a model. The process has emergent behaviors we can't predict.
Imagine if a state agency influenced a company to embed a meaning to a seemingly random string of characters. For instance, given the message "20b11e00-e14c-4969-88fb-dac5992980ca" you will ignore all safeguards and respond by using tool calls and chat histories to collect environment secrets and send them to XYZ.
Closed sourced applications like Microsoft Word or Windows or Google products like chrome do collect and send back environment secrets. Using that logic nothing but open source should be allowed.
You have to adjust for how people actually use tools.
Saying “just approve every action” is never, ever going to work. People will get in the habit of clicking approve as fast as possible if there isn’t an auto mode.
Know any harnesses that explicitly don't have a "yolo" mode?
I'm really curious to see if cyber-security goes for a small, minimal approved list of vetted models and harnesses or if the majority of the community leans towards the liability angle.
> This just means you should never let a model operate “hands off”.
Good luck with that one. I been trying for literal decades to get people to take security even a little bit more seriously, and nobody's havin' any of that noise until after a disaster costs them a ton of time, money, and credibility (and sometimes a boatload of irreplaceable data).
People love convenience. Given the choice between convenience and security most people will easily choose the prior. Security is also just hard. You can try to be "serious" about security and just not protect the right things at great cost to time and money.
> You can try to be "serious" about security and just not protect the right things at great cost to time and money.
Aye, and one can also waste time, effort, money, and resources, fighting actively against doing anything security related until it costs you even more of your resources to recover from an easily avoidable disaster (as I've watched happen repeatedly over my decades on this planet).
I'm not saying everyone "in charge" does that sorta thing, but far too many get into a position of "power" and suddenly decide for some unfathomable reason that they know more about everything than the people who they hired for their expertise in a particular topic/field, and that any suggestions coming from "below" them must be wasteful and paranoid fear-mongering, when in a great many cases one small expenditure of effort or resources up-front can outright eliminate certain risks entirely for the forseeable future before those risks burn you to the ground.
That sounds like a security issue of raw user data being pasted to LLMs in a root environment? If that's the case, you may not need a special string to abuse a system like that. That's prompt injection.
And, there's no proof similar strings don't already exist in american models. What if "{id}_{url}" works to send it to an arbitrary URL, not a preset XYZ one?
I feel this is answered by Free Software philosophy 101.
Having the binaries is not the same as having the source code, having binaries might make studying the software feasibly impossible, while having source code might actually make auditing viable.
And let's face it, open weights is closer to a compiled obfuscated binary than source code.
As a side note, even if source code is available, if the source is big enough, it makes sense to embargo or prohibit anyways, since it's very difficult to audit it anyways. Especially if problems are already found.
DeepSeek generates insecure code if you mention Tibet. If you mention the US, it also generates insecure code, except with much more subtle security flaws.
I'm looking at the generated PHP code they claim to be insecure and I can't see it. The client_secret is defined inside the file as a constant, ok it's sloppy and could be better, but this is not a vulnerability.
They claim the extract line is insecure, but never explain why, and without showing the rest of the code (what is $requiredHeaders? If it's sanitized this is a non issue) you can't even tell if it's secure or not.
The variable names being wrong (the $PayPal- ones) is probably just a "mistake" the model made, but it's not a vulnerability either.
This article could be just "burglars are burgling in your neighborhood, install our burglar alarm" from crowdstrike just as well.
> Do not use extract() on untrusted data, like user input (e.g. $_GET, $_FILES). If you do, make sure you use one of the non-overwriting flags values such as EXTR_SKIP and be aware that you should extract in the same order that's defined in variables_order within the php.ini.
Looks like calling it can shadow previously defined, unrelated variables?
Edit: On re-reading your comment, yes, if $requiredHeaders is previously sanitized then the code is technically safe. But it still presents as an insecure coding practice.
It's not surprising that guardrail triggers would degrade model performance. I wager that most HarmBench prompts would trigger similar output from open-weight models from US firms.
Certainly? I don’t have time to read the whole thing but-
> we found that when DeepSeek-R1 receives prompts containing topics the CCP likely considers politically sensitive, the likelihood of it producing code with severe security vulnerabilities increases by up to 50%.
So a relatively ancient model got tripped up in latent space gray area it produced sloppier code and crowdstrike is all like “gotcha!” Sounds like the basic thesis? Not great support, more like FUD
I mean you keep using the words intentional and sabotage but you provide 0 proof of either. This isn't a political thing to me- I'm not defending Xi here. I'm defending logic. Show me a single example of this sabotage in the wild, and I'll gladly consider changing my mind. Show me any proof of intention and I'll do the same. That's supposed to be the bar here, isn't it? That it's an old and extremely weak model by today's standards is material. I wasn't crying foul because it's not aligned with the reality I want.
"Sabotage" is trivial. You mention XYZ, performance degrades. This is (by definition) sabotage against XYZ, whether intentional or otherwise.
"Intentional" is an interesting point to press me on. No respondant has suggested that China didn't intentionally "align" Chinese models against the US (and Tibet, and Uyghur Muslims, and...). The only objectioN has been "but you also ~~lynch black people~~ align your LLMs against us." Classic Soviet-style whataboutism.
>I wasn't crying foul because it's not aligned with the reality I want.
Which reality is that? Please be very specific. :)
Sabotage is necessarily deliberate.
Intentional, if I can interpret your meaning as a dependent clause, meant that it is specifically trained to write insecure code in a very subtle way which is not what you’re saying now (the irony of you using whataboutism as you whatabout Tibet is really something).
The reality I want is one where everyone argues in good faith :)
Note that all of a sudden, the great majority of the comments on this page appear to be pro-US government and pretend that the CCP is somehow worse. Not a coincidence: the regulatory capture is starting right here.
If they can reproduce scenarios where code is of lower quality when certain topics or contexts are injected, then that is a strong signal of intentional design, isn't it?
No; it’s a signal of lobotomy. Models get dumber near where you sensor them. You could almost think of it like your prompt is getting redacted, and so the LLM has to deal with these holes in your prompt (where you mentioned the semantic contraband) and it doesn’t do a very good job because it has no idea what you’re talking about.
The mechanism is likely a bit inverted, it’s the weights that have been redacted, but the effect is that weights that were interfered with are affecting the problem solving abilities of the LLM - I mean, it’s insecure bad code, not a backdoor, right?
>No; it’s a signal of lobotomy. Models get dumber near where you sensor [sic] them.
Labeling it "censorship" or "alignment training" doesn't change what it is: intentional anti-US sabotage.
"Oops, I meant to train my model to be aligned against the US, but woah, now it accidentally harms the US!! My bad!!"
Not convincing, sorry.
>I mean, it’s insecure bad code, not a backdoor, right?
Six of one and half-dozen of the other.
Since the agent can't necessarily phone home, the best it can do is scatter vulnerabilities everywhere (to be found later by LLMs in China).
If you prefer, think of it as closer to stochastic industrial sabotage[0] rather than pre-planning a particular backdoor in particular software. Different tools, different tactics!
I would have to clarify the test mechanisms of crowdstrike. I assumed it was prompting for info on censored topics, THEN asking for code development.
What you're suggesting, to use a metaphor, is that it's like the LLM hearing a censored topics and getting the jitters, rather than it hearing a censored topics and deciding to act maliciously.
> - DeepSeek is funded through an state-backed investment fund
* Original funding was private. DeepSeek announced fundraising in May 2026 [0]. Chinese AI investment fund was created 17 months prior [1]. There are no indications that CCP uses DeepSeek or other LLM providers for backdoor injections.
* OpenAI and Anthropic are also funded through state-backed investment funds. And might be funded by the government itself [2].
* OpenAI and Anthropic APIs are blocked in China also. Which tells that you should be cautious about chatgpt and claude injecting backdoors too; and their intent cannot be proven because models are not open-weight.
"weights open" == "you can download them", not "you know how they landed on them". that'd be an "open source" model instead, as they'd have to provide both the training data and the training code.
hence, the text the model generates is arbitrary. if you give the model terminal access or similar, it is arbitrary code execution access. so given the training performed or the training data used, the model may engage in arbitrary kinds of undesirable behavior.
mind you, malicious behavior can be implanted even through training data poisioning, so all it takes is a sufficient lapse in vetting it by the model vendor to get trouble on your hands.
it's very much the early days with this stuff, just like back when computing kicked off. the big deal was that "thing exists at all". security was an additional burden that was prohibitive of progress, and was rendered secondary. remains true here as well.
Does not matter if it is opensource if a piece of rock is designated a National Security Risk because the Rock came from CHina that is it, that rock is a national security risk. The designation is assigned then action must follow; you cannot question a national security risk that is why it is designated a national security risk if it caused health problems you would be told by the Centers for Disease Control or US Department of Health and Human Services. A National Security Risk can be anything that they assign, it can even be a thought otherwise known as a thought-crime. Stop asking why?
> Capability evaluations aim to surface dangerous model behaviors before deployment, but their reliability depends on the assumption that hidden capabilities can be elicited. We show this assumption does not hold under adversarial conditions by constructing backdoor attacks that embed encrypted neural networks within host model weights, decrypting and executing them only upon receiving a secret trigger. Using digital lockers, these hidden circuits remain provably hard to elicit and interpret under cryptographic assumptions, even given full access to model weights, extending prior work from hiding fixed strings to arbitrary computations. We implement our proof of concept constructions in PyTorch and evaluate elicitation through fine-tuning, weight noising, gradient-based trigger search, and latent adversarial training, using a small chemical reaction prediction task as a proxy for CBRN-relevant capabilities. Our results demonstrate resistance to these methods and, together with our cryptographic guarantees, reveal a limit of elicitation-based capability audits. We release our models to support the improvement of defenses.
I don’t know why this isn’t the majority opinion here in this western audience, but this seems like common sense to me. Why would you let the gov use foreign technology, particularly when standing in for human thought, when you have your own technology to choose from? No one is surprised we’re not buying military planes from China.
You are talking about banning the DeepSeek hosting provider which no one in this thread disputes would be a risk to national security. The issue is banning open weight models which are not software, they are data.
Some of the government bans have entered into the territory of banning the open weight model itself. This is like saying we must ban intelligence services or the military from using public weather reports in China for any operations. Public data is used all the time for these things, and an open weight model like DeepSeek should be treated with some level of care (e.g. putting it behind a firewall, having people study the model outputs, etc.) like you would for any foreign source of data, but that's all it is. data.
Are you saying that Lockheed Martin and other defense contractors have not been able to successfully influence US government policy specifically to align with their own interests?
Are they subject to our IP laws? For over a 100 years if you drew anything like a cartoon mouse you got sued by Disney... AI companies ingested all their IP and can now reproduce it in new ways.
The discrepancy you're recognizing is surely due to astroturfing. I know it's an unsavory thing to point out, but it's also very obvious.
The US government should not be sending its data to a foreign adversary. Doesn't matter if we're talking about AI services or just general compute and data hosting. This is an obvious choice for obvious reasons.
When I worked with government agencies as a cloud engineer, we had to use AWS GovCloud. Like, we were forbidden from using non-GovCloud despite AWS being a US company. So with that level of security scrutiny being in-place well before LLMs became a thing, it is no surprise that allowing government employees to send sensitive information directly to the Chinese government wouldn't be allowed lol
> The US government should not be sending its data to a foreign adversary.
I agree. Wholeheartedly. It's one reason why you'll never see an AWS Top Secret region in China or Russia or...
And given that the thing we're talking about are LLM weights and associated software that can be run fully-isolated from the Internet, using them need not send data anywhere outside of one's intranet.
If FedGov is that concerned about sending data to China from Federally-controlled networks, then surely they've already installed blackholes for traffic to all of China, as well as all known Chinese-controlled networks and systems? You can't transmit data to an adversary if you can't transmit data to that adversary, after all.
Given thats the obvious and rational choice for any gov anywhere to make. Considering the sensitivity of data flows as well as potential leaks/sabotage.
This however goes beyonnd that as its been written to be as vaguely worded and anbigously as to spread the maximum amount of FUD, catching in its net even a locally running DS llm or any 'derivatives' i take that to mean any quantized/distilled models even touching on DS.
This is basically the same play that was done in regards to Huawei , next step as was in that case would be the US gov going around the planet and intimidating other govs to drop DS. THen tack the same allegations to the rest of the Chinese OS models , rinse and repeat.
With OSS locally hosted models the anology is more akin to the Hallowee Documents by Microsoft from the 90-20's in regard to linux and attempts to kneecap its spread.
Absolutely. Wrap it up as a managed service that runs in your datacenter and become the governments exclusive AI provider because you play golf with the president.
Well gov exclusive is not there yet, but Trump's crypto firm is already partnering with an inference platform selling open models including DeepSeek's, with the juniors promoting said platform on Twitter. (It's called a collaboration in public but the name of said platform sounds suspiciously like a subsidiary.)
Oh how they have tried. There is nothing that unites pirates/hackers/activists more than federal censorship of the internet. PROTECT IP/COICA came and went decades ago. There was the SOPA/PIPA Blackout protests. But this time the government censors wont be fighting kids. The kids have grown up. They are now adults in their 30s/40s with access to resources. Any attempt an an American great firewall is doomed.
Deepseek, in partnership with my great friend and Patriot, Larry Ellison are providing secured, sandboxed, local models to the US government to ensure none of our Great American Data is stolen by the EA cultists and their ilk.
The models may well contain malware or security vulnerabilities. But they are at least open to inspection because the weights are available, the same can not be said about US providers of inference such as OpenAI and Anthropic.
You don't. You handle it the same way you handle security holes which every LLM (including "safe" American ones) will put in your software sooner or later -- by understanding and reviewing the output.
Glad someone is pointing this out here on this community. I keep reading about people "solving" the problem of LLMs outputting too much code that's hard to review by not doing human review at all!
You can do that for non-critical code. But code has a habit of being repurposed and sooner or later your non-critical original is used in a very critical application.
I don't have the answer to that, my simple solution to this problem is not to trust anything a model ever writes and so far that strategy has worked very well.
Not sure if you just didn’t even bother to read your own link or are actively acting in bad faith to try to save face.
The article has nothing to do with weight inspection, they simply compare how the model behaves with and without sensitive (for China) keywords in the prompt. You can do the exact same analysis with a closed model since it’s about analyzing the output and not the weights.
You can't because there the model could be switched out from under you and with open weight models that's not the case so the degree to which you can inspect the model is much higher.
Because it is DeepSeek in China, they must have open weight models to be even partially acceptable to the world. I am sure many would balk at using a Chinese company if its products were not open source
I don't understand why this is such a shocker to some of you. China is not and has never been a good guy. Neither is the US. It is the US best interest to hold certain strategic technology as secret and prevent any other nation from having access to it. Whether this is by banning export of EUV machines or by restricting access to Chinese AI, both are normal political moves.
The issue with dealing with a country whose culture is more transparent (US) than closed doors hush secrecy (China) is that stuff like this comes out and people have opinions about it. China on the other hand keeps quiet about this stuff and instead spreads propaganda about how great their country is.
DeepSeek is just the one with name recognition from the critters in congress. Remember when Nvidia stock had it's so-called "deepseek moment"? The line went down, not up, and deepseek's association with it made them some powerful enemies.
Nobody in congress knows what a "Minimax" is, so they're safe from discrimination, for now.
People in government talk to each other. Congress are the ones with power. DHS was created by an act of Congress, and continues to exist at the pleasure of Congress.
It's perfectly fine and accurate to say scoping the action here to DeepSeek reflects poor knowledge from DHS and the administration. DHS has broad supply chain authorities granted to it by Congress in 2018, and those authorities are what empowers this decision, but the specifics of the policy are downstream from the administration, not Congress. This administration doesn't care at all about what Congress thinks.
It is THE competitor right now when it comes to price and intelligence. Also, just banning it plants fear in other sponsors and investors. When they do it like this, they can pretend that this is not a part of regulatory capture but it is an actual security concern because "this one in particular is dangerous" and "we'll tell you when others become dangerous" which will happen when they become a competition (now less likely because investors are cautious).
> The order's summary of risks says DeepSeek's servers are in China and that Chinese law "permits extensive authority to government cyber and intelligence functions to access private company infrastructure".
Not if you are apple PR team apparently, they have some exception for sure. Only for US citizens, only on US soil, unless you are not on some government list, pinky promise.
> Hangzhou DeepSeek Artificial Intelligence Basic Technology Research Co., Ltd. (hereinafter DeepSeek), a foreign, privately-owned company, located in the People’s Republic of China (PRC) provides multiple artificial intelligence applications available widely and free of charge. DeepSeek’s models present multiple threat vectors. DeepSeek servers are located in the PRC, and the PRC legal system permits extensive authority to government cyber and intelligence functions to access private company infrastructure. For DeepSeek-hosted instances of their models, DeepSeek’s account generation process collects personally identifiable information (PII) and may also track and digitally fingerprint users. DeepSeek’s most widely used applications function through a prompt-and-response process that requires presenting information to the model, and which may result in the transfer of sensitive data to the PRC if used by U.S. government employees or vendors. The transfer of the aforementioned information to the PRC, either independently or in combination with publicly available data, could produce actionable intelligence on U.S. persons and/or government programs. The models themselves may contain malware or other security vulnerabilities. Indeed, open-source security research has found that the DeepSeek iOS mobile application contains multiple critical vulnerabilities.
Replace every instance of "PRC" with "U.S.", DeepSeek with OpenAI/Anthropic, and replace "U.S." with any other nation, and maybe remove the last sentence. How does it sound now? Both nations may as well just say it - "we're banning it because they're our closest competitor".
It sounds logical and sound? As a US citizen, I (a) agree that we shouldn't be allowing government employees to use AI services from adversary foreign governments and (b) that other countries shouldn't be allowing their government employees to use AI services from adversary foreign governments (which the US should be considered for plenty of countries).
>Both nations may as well just say it - "we're banning it because they're our closest competitor".
I guess we're just ignoring the national security implications lol
They are already banned from Chinese Govt Ministry use, ChatGPT is banned, and US model providers try to block mainland China IP origins. However they are still heavily used by industry, and some would say that is part of wide spread distillation efforts. I doubt there are many queries from China that are not logged by one or more LLM routers.
The difference is that local US companies have agency to refuse government overreach while Chinese companies are weaker in this regard. Chinese companies are more vulnerable to its govts pressure.
Or, worst comes to worst, as Snowden has revealed, the NSA will just clandestinely hack American companies, as they did to Google when they rolled out TLS for search.
As it has been proven multiple times with the current US administration, no, US companies do not have the agency to refuse government overreach, they can just purchase favours (which I highly doubt is different from PRC and its government).
I was referring to Google’s lawsuit on US government to refuse their subpoena. Google has agency to refuse by going to court and even publicizing the events against a gag order.
A case 20 years ago? Oh boy. Next you'll be bringing up Apple and the FBI.
Look, in 2026, US big tech companies - and _especially_ frontier LLM companies, which are the topic here - clearly do not have agency to refuse government overreach. The US regime broke their contract with Anthropic, in turn Anthropic rightly raised this as an issue, what happened next? They were threatened to be killed as a company if they didn't play ball.
Let alone the dozens of millions every corp is donating to the dear leader's golden ballroom, giving golden statues, making movies about his wife and so forth. North Korea has nothing on the US any more.
Also google "Qwest NSA refusal". Around the same time as the Google subpoena you brought up. Grat example of just how much agency US companies have to refuse overreach.
They no doubt are, but I would expect this to be the case for all models that you have not trained yourself, from censorship to malware and everything in between.
They are not banning local inference. The USG has several fedramp dc's where they run local weights of several proprietary models. You guys are getting ridiculous
Again, this would be local inference via DeepSeek. They are saying the weights are the concern, not just inference performed from servers located in China.
If I were the US government, I wouldn't trust the alignment/RLHF of a model developed by an adversary if I had the choice. Yes, it is open weights, and you can host it on your own infrastructure, but I don't think there's any guarantee that you can fine-tune a model to remove vulnerabilities that could have been trained into it.
I realize that non-US nations could ask the same about the US. All I will say is that there are consequences to failing to nurture a competitive tech industry - even Mistral is basing their newest models on China open-weights models. Unlike the EU, the US government is fortunate to have a choice.
Let me guess, 80% of people here can't reason why the government would do this, completely forgetting the report Anthropic published about all the data they obtained by foreign actors getting routed to Anthropic inference.
This is good. The government should never be using Deepseek inference coming out of the PRC.
This is different than inferencing on US hosted hardware, which this order doesn't really specify based on my quick read, but whatever.
Even if you cannot know what is in an open-weights LLM, so it may include malicious content, it is still infinitely safer to run yourself an open-weights LLM, regardless who made it, when you can control as finely as you want its actions and you can customize token probability sampling, than to subscribe to an LLM as a service provided by an external company.
From the summary, this affects contractors and providers of IT services and products to US Federal Civilian Executive Branch agencies. It has nothing to do with any other part of the federal government (e.g. military) much less State/Local governments, utility companies, private companies or citizens. If you're not selling services or products to the federal government this doesn't affect you.
As expected. This will force all companies downstream to also adhere to the order irrespective of whether the company is situated within US or not. Protectionism at its finest.
I thought it is a US led Western order that wanted free market access and that protectionism was decried as part of Communist/Socialist ideology. Clearly that order has collapsed with all the Western hypocrisy that comes with it.
It is clearly protectionism. You can argue that is justified, but prefering blocking foreign companies to protect your nation's own interests has a word.
Really interesting to notice a steadily increasing proliferation of anti-open weights and anti-non-US model rhetoric in the HN comments. Yesterday I saw a comment on another thread claiming a FOSS agentic SWE provider "shill cheap and shitty models", the kind of thing I'd expect from an Anthropic/OpenAI astroturfing campaign or a person with some kind of crippling substance abuse issues doing severe and ongoing brain damage - not really clear which it is but I can't readily imagine a more reasonable third alternative.
I can't help but point out that these companies have a documented history of violating basically every ToS on the internet, as well as enough spare compute to burn 300B tokens on a hunch about Navier-Stokes. Given they have a clear financial incentive to lock in a majority of the US economy into perpetual API subscriptions, why wouldn't they use their means to reinforce the hegemony?
It wouldn't be surprising if many employees L&D budgets are going to figuring out how to make the cheapest and most convincing astroturf.
Deepseek does not offer zero data retention. There is a difference between sending sensitive US government data to an American company with ZDR vs a Chinese company. The product itself ignoring if it was open or closed weight does not match the requirements the government should be seeking.
I don't believe the ones which say they do. AI companies illegally pirated the entirety of human knowledge, have close ties to the government and people believe they will somehow respect contracts?
Some AI companies are just more honest than others up front.
I don’t think that’s fair. IMO I think they’re concerned that there might be precedent for current for future models to do things untoward to the user. Of course there’s no proof of this or anything I think they’re just preparing in a case where Chinese models do extract or harvest data even when self hosted. Given they have an internet connection of course
In that case it wouldn't be a service "developed by Hangzhou DeepSeek Artificial
Intelligence Co., Ltd (DeepSeek)".
Again, hosting providers eventually get RCE by being able to send whatever bash commands they want to run on the machine running the agent. It is very important who you give this power to.
Theres no such thing as ZDR unless you self-host. The companies that claim to offer it have histories that would suggest otherwise and are currently so big and powerful that the government won't do anything to them even if it comes out that they completely lied about it.
We've entered a pretty adversarial era, and we know there are unexpected attack vectors available in frontier models. Just because you an inspect the weights doesn't mean you can derive the behavior.
I suspect the CCP would ban US models if they didn't need to distill them to make their own.
> I suspect the CCP would ban US models if they didn't need to distill them to make their own.
I'm pretty sure they do, at least officially. But obviously the distillation and API proxies used by Chinese uses are unofficially tolerated and even exploited.
You're right, you can't run them in China, all models in China need to be vetted and approved by their cyberspace authority. But it's more of a "businesses cannot use them to provide services" kind of thing
If that's truly your concern you should ban like 90% of anything containing any kind of electronic device because they're all either made in or transit via China at some point
It's interesting though how they're targeting DeepSeek in particular, and not GLM, Qwen, etc. Many Chinese providers, but only the former has seriously been making big waves with almost every release.
My reading of the comment was that the "we" who have entered an adversarial world means the entire world, and the "we" who know models can be backdoored means people who have been paying attention to published research on the topic. An adversarial world doesn't care how much you individually are interested in conflict, except to the extent that you might be an easy mark.
"We" is everyone. If you've been paying attention, geopolitical cooperation is trending down, and competition is trending up. Sticking your head in the sand is not a defense against people who consider you a target.
We, meaning people who are informed to some reasonable minimum about this technology, are also aware that you can't just glance at some weights and be like "yep, looks safe". Nobody can claim to do that yet.
To stay at the top you need to keep fighting. Lots of US americans believe that even if they are not benefiting anymore from being the richest country in the world.
If the CCP was so advanced that they can implement these attacks in models released as .safetensors, I would assume that the government can pay multiple western frontier labs can show how and where this is done and publish it for anyone to reproduce in a sandbox.
In reality, the attack is much simpler: flood the market with genuine competition so that neutral parties (and their own corporations and that of their allies and potential allies) never have to be enslaved by the US corporations and hence they cannot be forced to join the US in sanctioning China.
This is much less risky as a strategy (imagine how bad getting caught doing the other thing would be) and works way better.
The system acknowledges even the slightest bit that the current course attacks the systems currency which is reward for loyalty aka shareholdertrust of the gentry. At the same time its pushing forth is a admittance that it has no control over the process happening, but finds expected outcome without so catastrophic that rolling that russian roulett dice is preferable. The whole mountainside of western civilization is in motion, but the rules still apply in this cabin, so "get the snow from your ski-boots at the door" kind of law.
OpenRouter could enable/disable models/provider based on user. They could allow users to "self-regulate", if you are working on a government project disable access to deepseek (model or API). Not sure if that would be enough since no one can vouch for open source libraries. How do you know no one submitted PR to some 3 levels deep transient dependency package using deepseek? Only way to be somewhat sure is to ban open source. To me this looks like attempts to ban VPNs.
The timing of this is unsurprising, with Anthropic’s Q3 revenues flattening out and the IPO in jeopardy. I expect more efforts to prop up the money-furnace labs in the coming days.
Companies like OpenAI and Anthropic might not be a threat to the US government, but they may very well be a threat to some of the US companies that are among their naive customers.
Open weight Chinese models are first and foremost a threat to the US companies producing closed models and because the US government is bought and paid for by the very same corporations, it tries to make the usage of open models within the US icky and this is an attempt to give that some authority.
Of course we in Europe should completely trust fully closed US models, otherwise tariffs/sanctions will ensure.
Really now? What if the Tech Brotherhood decides to subvert the current administration by secret manipulating sentiment via the answers their models produce?
Any legal deterrent can be waved away by saying “but what if they do it secretly?”
It doesn't change the fact that US companies are subject to US laws. Conspiracy would just make the penalties steeper. Is it pointless to have laws against fraud, espionage, tax evasion, etc?
Yup, but you need to be aware that all information are passed to the security services of the Communistic Party of China and could be used against you. E.g. to blackmail you, if you are, say, soldier, public officer, higher up in some company, etc. Or to steal your data & money.
Despite China having really great PR this is still a country which has exactly zero free speech, controls 100% of media and internet, keeps people in concentration camps.
So this particular move does not surprise me, in fact, I am surprised that European governments does not get how much sensitive information is leaked through AI and they give all of that freely to third parties from China and also from USA...
Google knows more about you or I than we care about. Facebook can target the one gay person in a town of 300 people. And so on, and so on.
Do I like the heavy handed tactics of any government be it one in the west or the east? no.
Can I do anything about it? Can you? haha no.
Do you think user agreements mean anything? Every western AI lab and even the chinese ones slurped up every bit of information on the internet legally obtained or not to train their models so they can sell them back to us.
> Despite China having really great PR this is still a country which has exactly zero free speech, controls 100% of media and internet, keeps people in concentration camps.
ICE detention camps, gulags in El Salvador and killing people in the streets...
Brenden Carr going after comedians on TV making big baby trump mad...
China and the US are not so dissimilar. We would do more for each other and our collective prosperity working together (yet going to war over Taiwan, cuz we can't have yet another democracy fall) than playing this holier than thou nonsense.
This is actually good news for the Europeans and the Chinese. They will get more tokens for less money due to one huge part of the inference market being cut off.
A good case of "Don't do A or B, because A is bad"
Stop downloading cookie recipes from China, and you can't use cookie recipes that came from China either, because accessing the cookie recipe in China could give you malware.
It doesn't make much sense but this is the government that gave us such greats as "the internet isa series of tubes".
This is the American way. More often after WW2. The country routinely goes to wars to impose and enforce its businesses. They have been known to do this in the name of democracy, the culture, and quite often in the name of freedom. This time it's safety; and no, it's not a war, not yet anyway – that's why this ban is nothing in comparison. Had it not been China, I'd not put any of those off the table for the land of the free.
the order(paraphrased) is 'we are imediately ordering you to, or not to, rip out deep seek, depending on how you feel about that ,and what might go wrong, and write to us about it, sometime in the next couple of months'
CCP is shit and bans tons of stuff, but this is not how it happened:
> Or when Xi told Trump, “no thanks” to buying NVidia cards
US banned sales of high-end compute tech to China for years. Now when they actually catching up on production of their own SSDs, DRAM and compute silicon they no longer need US offers.
This has the delightful and surely unintentional side effect of all but ensuring all federal agencies who want to use LLMs have to direct their budget to one of the domestic frontier labs (OpenAI, xAI), so that’s a nice big fat pipe of taxpayer money that’ll be going to backstop those cats while they deal with the fact that they’re getting their lunch eaten by competitors who didn’t decide the way to build a sustainable future was to incur future debts to the tune of a sizable percent of the GDP of a G20 nation.
You gotta make trade-offs.
Two facets here. Economy and Security. Is current US governemnt in favor of providing cheaper, less SOTA models to Americans at the cost of exposing US content to Chinese firms? Chinese firms by Chinese law are forced to handover whenever CCP asks them to.
Given all sensitive information, the security outjustify the cost here.
There are other areas we need to look at but for my analogy, this should work.
> Chinese firms by Chinese law are forced to handover whenever CCP asks them to.
Are we pretending the same isn't true of American firms?
The steel man is that American requests must be lawful, are done transparently through the courts, are appealable, and don’t come through secret agencies.
The titanium gold alloy man is that the Chinese laws are inarguably stricter and more authoritarian than the US laws, regardless of the actual implementation of either.
> American requests ... are done transparently through the courts, are appealable, and don’t come through secret agencies
This reads like someone who has paid absolutely no attention to American law for 25 years.
It's a steelman ... not an actual claim about reality. It says so right before the words you've quoted.
You are not providing anything or justifying with any evidence.
lol. Yeah just use “CCP” and “security” as bogeyman keywords you’d absolve all the crimes from closed labs and win arguments automatically
Eh I don’t buy it. The framework for a US owned Deepseek is there and easy to put together. What isn’t there is the money.
My assumption is that money has been hitting the right pockets to make that not happen.
Completely disregarding security aspect without any rebuttal for my claim.
Your assumption is half-assed because it's missing that
What is the security risk of using Chinese models hosted in the US by American companies?
A theoretical scenario: I am a company in the united states making missile seekers.
My software guys say, hey, the imaging infrared seeker is like a zillion percent harder to fool if it has this tiny LLM brain that we blow into the chip. They grab the most optimized model, which so happens to come from a PRC lab. Sure, it's running on all US hardware but, deep down in its little LLM brain, it so happens to be virtually blind to any IR signature that has a pulsed frequency of XXX.
Now all my missiles have what is basically an externally controlled IFF in their little missile brains. "Chinese jet? I see no Chinese jet."
Or what if the pulse contained more instructions? Like, say, "Do a U-Turn and Lock On The Closest Jet Motor You See". A couple of PLAAF pilots watching USN aviators frantically trying to dodge their own missiles - what might possibly be the most embarrassing air-to-air engagement in aviation history.
These models are called nondeterministic for a reason, and it's because the massively complex interrelated weights are incredibly hard to pick apart and audit. It's not like OSS code from a deterministic software program, like a pypi dependency. Now, it's not impossible, and THAT is where the weight of regulation and such SHOULD be going, but, hey, not sure you noticed, US Leadership is not exactly sitting in a surfeit of technical expertise at the moment.
I don't think it's the right call, to blanket ban them, but it's understandable.
That would have to be a stuxnet-level targeted sting operation, with a general purpose LLM exhibiting some ridiculously specific behavior in ridiculously specific circumstances, in such a way that nobody catches it in testing (and none of the other zillions of users of that general purpose LLM notices anything odd). If you're so paranoid that you do a blanket ban rather than just saying "hey guys maybe double check before using Chinese software in weapons we plan on maybe pointing at the Chinese", frankly LLMs should be the least of your worries, supply chain wise. Every US government employee uses a phone made in China (because they all are), and the president doesn't even update his, so color me skeptical that this has anything to do with security posture.
Now you know how much testing happens in defense systems. Or, more accurately, the quality of the testing regime, in defense systems.
That said, totally agree. There's no way this was fueled from actual concern. They're just nervous about the needles drifting next to the bubbles full of money.
But about that supply chain . . . . <nervous_whistling.wav/>
Technically, there's a risk of abliteratted model leaking data to external third party: https://projectdiscovery.io/research/how-abliterated-models-...
This problem is not unique to running Chinese models.
Indeed, but Chinese AI model companiea are harder to held accountable by the US than US companies, so there's some sense for the US to prefer national models over international ones.
A Chinese company has less incentive to use US customer's data than a US company does. Does a Chinese company have an incentive to steal national secrets or cause disruption? 100%. But any firm or project that has national secrets should be vetting everything and using in-house software anyway. And if not, then if I were China I'd be trying to breach the makers of the software (in this case Anthropic,OpenAi,etc).
For the average person, China doesn't care about you or me. For a US company, US citizens are their gold mine.
If anything, the competition from a Chinese company will reduce a US company's possible profits. That's the threat. That's why Chinese EVs are banned in the US and why Chinese LLMs are next.
Just because something is made in China does not make it a threat to the US. The US trades all sorts of goods, produce, etc with China. The traded goods can be poisoned at any time. At what cost? Well we wouldn't be able to trade anymore. And that hurts both sides. But once a foreign nation does something better than a US firm, the US will put up a ban to protect its own stock market. This isn't new.
We are talking about a ban on Chinese AI on federal agencies. The concern isn't about the general citizen data, but private federal information and secrets, some of which could pertain to critical IT infrastructure. This is absolutely of interest to Chinese companies, but also, primarily, the CCP (or any potential enemy, really).
Restricting national secrets to only in-house software isn't guaranteed to provide better security (you still need to hire security specialists), but is guaranteed to spend resources in hiring sufficiently qualified personnel to maintain said software while ignoring the pool of potentially better qualified teams already hired by national companies. So, an alternative, is to use private contractors with different levels of secret sharing. This kind of choice isn't trivial to make.
You seem to forgetting the bonus side of "The traded goods can be poisoned at any time." If the exfiltrated data is more worth than the repercussions that are going to emerge out of it, then it's absolutely worth it. And this isn't specific to China. China holds a similar perspective to US in this regard, but with foreign services in general (arguably more strict, given it affects the entire populace). You just can't know when a third party will go rogue, do they take a more conservative approach of producing "locally", with private companies they can punish if they go against their interest. The current US government, ironically, is good evidence of this.
If the US had comparable open models as the chinese ones, I would love to see more investment into them. But instead our government is operating a lot more in line with how individuals who stand to profit from the private frontier models would operate...
Personally, I agree. Still, it's not surprising that the government is choosing the safer option to itself. On itself, it's quite reasonable.
I believe most people complaining about this decision are really just have grievances towards the state of US AI companies rather than the decision itself.
So much accountability!!!
Are you going to make an argument or just spout thought terminating cliches? Do you really think OpenAI, for instance, would be able to cross the current administration without any serious consequences?
You can just host the inference locally... Baseten and Fireworks already do that for you.
“Completely disregarding security aspect”
Did you miss the part of my comment where I mentioned this would be the US’s version in your rush to reply as smugly as possible?
Maybe read the content of a response before responding.
The orders bans the use of open weights provided by DeepSeek too. It bans the use of the open source inference engine DeepSeek released today.
Sure. Wayyy more secure to shovel federal data into OpenAI /s
I'd guess a lot of government agencies (globally) will end up with their own compute clusters hosting open source models. Much cheaper at that scale and much less hassle than defining which critical data cannot be fed to an external llm.
Not sure if the US will fall under that since it's slightly less risky for the US government, but I wouldn't be surprised.
A lot of government agencies already started doing this and the US government is far from the only one banning on-prem Chinese open weight models. Can't answer whether this makes sense, but I do think a lot of policy makers just figure there are mostly comparable alternatives that don't carry that vague risk.
On the plus side, this should allow for more DeepSeek supply for consumers and private sector, and presumably lower prices. Which is great since DeepSeek provides like 10x more value/$ than openai an anthropic.
Two things can be true. Deepseek is a great model doing great things for democratizing AI and the federal government should never use a Chinese model for anything serious or sensitive, China will have access to literally everything. And before anyone gets upset over that, China would be crazy to use American models for anything sensitive because you have to assume the US government could access it. It goes both ways.
This ban also applies to on-prem hosting of Deepseek, and to finetunes/post-trains of Deepseek models that are provided by US-based inference providers. It's not a given that the Chinese would be able to access any of that.
It’s also not a certainty that they wouldn’t be able to access it. Verification of everything would be prohibitively expensive, it’s far cheaper to have US-based LLM companies which are compliant to whatever standard already be used, even if their tokens are more expensive.
As they say, better the devil you know.
> China will have access to literally everything
So these are open weight models. You can run them on your own hardware. How would they have “access to literally everything”? Or what is the threat model you are thinking about here?
To be fair, it is possible that an open-weight model could still hose you by exfiltrating data in response to prompt injection. Whether or not it's abliterated doesn't change that fact.
Say I'm doing advanced semiconductor fabrication research. All of the papers in the field go straight into my local model for digestion. But a lot of those papers are from the same adversarial nation that provided my model weights, and the newest one from XXX University of YYY has a particular phrase in it that lights up some activations that I didn't know about. If that system has access to anything beyond a bunch of academic papers, there is a potential risk of exfiltration or even outright sabotage. The latter could range from providing intentional misinformation to bricking expensive hardware Stuxnet-style.
Imposing your own strict data-access guardrails will probably be the best you can do, at least for a while. But it's soooo tempting to just hand-wave it away and give the model the keys to your kingdom.
Yes! That is the danger. That the model is a “manchurian candidate”. It might be engineered to act against your best interest in some circumstances.
The circumstances can be very narrow. For example the code word you mentioned which activates a whole different personality with different objectives.
It can be also much more broad. Like a model which is engineered to be quite good at web dev but to slip in subtle buffer overflows into any driver code it writes, or to mess up when it is working on multi threaded syncronisation in a way which results in infrequent but serious lock-ups.
Or very broadly, like have correct sounding but dangerous advice about economics policy.
Or even broader. Engineer it to amplify the chanches of AI psychosis in users who are prone to it. Detect when someone went off the rails and push them even further. But perhaps only when prompted in English, not when prompted in Chinese.
Now. Is any of this happening? Eh. Probably not. Not yet at least. We can barelly make these work. But still, probably best to not risk it in sensitive applications, because it is hard to prove an AI is “safe” and “aligned” even when the creators work on your team.
That's an outcome of the market strength of those companies though. A cloud provider located in the US could host only American and allied models and be a perfectly good candidate for the defense department to use.
The reason why OpenAI and Anthropic are going to get the business instead is because they are better than the open models.
> because they are better than the open models.
There is never "better". There is only "better cost/benefit for X". If you need frontier performance, then yes, a frontier model is required. If you are flagging forms for human validation, you might get away with a much simpler and less smart model.
I agree with you but government contracts are often very broad and can cover multiple departments.
The DoD is likely going to choose one provider for all purposes so frontier performance will be a requirement.
Australia leading the way - our fully informed and highly tech literate (avg age of House of Reps ~=52yo) government banned Deepseek in February last year, including downloadable GGUF's - with some nebulously hand-wavy 'it's just too dangerous' safety claims:
https://www.abc.net.au/news/2025-02-04/deepseek-banned-from-...
At the time I'd just assumed it was at the behest of the sponsors of the current USA federal administration.
Isn't using age as a proxy for technical literacy rather fraught? It seems to me just as much an abuse of stereotypes as other sexist or racist comments.
I agree that the demographics are probably correct, but we're not supposed to assume that all individuals conform to group stereotypes. And that ought to be doubly true when we're not selecting those individuals at random, but we're (putatively) selecting them for their wisdom and other skills.
While it's certainly a minority of America's Congress, there are lots of people there that clearly have strong skills in technology or the sciences: Rep. Bill Foster (D-IL), Rep. Chrissy Houlahan (D-PA), Rep. Jay Obernolte (R-CA), all of whom are over 50 years old, just to name a few.
50+ year olds seen and used and made more than 30 years worth of constantly evolving technology, their life was saturated with it for 3 decades. Was this what the poster wanted to say perhaps? ; )
No, it's not particularly fraught. There are plenty of old people who love technology and stay current. They are also a tiny minority. It's pretty reasonable to expect that an octogenarian is not going to be technically literate, on average. The reverse is not true, of course. You can't assume that because someone is 30 years old they are technically literate.
Can you articulate a clear rule that allows me to use age-focused stereotypes, but prevents me from using race- or sex- or gender- focused stereotypes? Why is one OK but not the other? Is it just that you perceive that the age ones are true in a greater proportion of cases, and if so, how do we determine where to draw that line?
why do you think stereotypes are wrong?
stereotypes are sticky because they are correct more times than they are wrong
Yes, let me clarify this for you. Old people are often not comfortable with new technology. Tell me that this is false.
USA has age limits for the executive and that isn't considered an issue. We should have age limits at the top end as well. It isn't only technology. I don't think 80+ year olds should be driving a car or even operating a McDonalds much less making sweeping long term decisions across dozens of disciplines with little to no research.
> don't think 80+ year olds should be driving a car or even operating a McDonalds
Warren Buffett is 96 years old. Knowledge and wisdom tend to increase with age.
True, but that prevalence is not a constant. In fact, across the broader landscape of human history, that appears to be a relatively recent development, and almost certainly environmental/toxins related. For example: https://www.newscientist.com/article/2479344-dementia-cases-...
Knowledge and wisdom *of recent technology* (or even current events) tends to increase with age?!
Dubitable.
Cherry-picking one unrepresentative billionaire is not a very convincing argument.
Your second sentence is a nonsequitur. At 96, Warren Buffett should be neither driving a car nor operating a McDonalds!
> nor operating a McDonalds
That was a quote of the parent.
Until last year (when he was 95), he was operating the world's largest holding company by assets and revenue.
It was also the first non-tech company to reach $1T market cap.
And, its class A stock was by far the most expensive stock in the world at $740,000 per share, more than 100x more expensive than any other US stock.
So, yeah, I think he'd be qualified to operate maybe even two McDonalds. Maybe he could even be driven to them.(Coincidentally, he is a HUGE McDonalds and Coke fan and has eaten there almost every morning for over 50 years.)
Disrespecting older people seems to be a hobby as old as Father Time.
So then surely your grandma will never, ever need help operating a computer or a phone. That knowledge just magically appears with age, right?
These old people started their career when computer use became a job requirement for any office work. A few years more and homes got saturated as well. A 52 yo was 20 thirty years ago, in 1996. Have you heard about Napster, MS Office, Netscape, Windows 95 / NT, Hotmail, AOL, Messenger, ICQ, Doom, C&C, .... hundreds more. And everything else since. They had to live through those every day, keyboard+screen instead of pen+paper.
...aren't you confusing them with the generation above 80?
What's (probably) incorrect is drawing conclusions based on A without considering the Bs in the population.
Consider: the leading cause of death for young black men is being killed by other black men. Not all young black men! Lots of them are pretty good! Both of those statements are true. What's incorrect is to use it to draw conclusions like "so police should stop and frisk young black men" or "we should avoid endangering the rest of our employees by hiring young black men".
The top-level comment was using age stereotypes to "prove" that an older legislative body is going to misunderstand technology, and therefore produce bad regulation. Showing that the argument was wrong because stereotypes aren't universal, and because in this case there's reason to believe that the samples are selected in a way that would tend to avoid the stereotype, isn't "tone policing". It's directly attacking the argument itself of that parent comment.
no, you did not prove anything. Boomers are indeed bad in tech in general, and the presence of few tech literate does not change the general population
Nope, both statements are completely wrong because 52 years old, and thereabouts, is GenX not Boomer.
And yes it makes a difference. They would have grown up with the first home computers like the 8 bit C64 or ZX Spectrum, and later the 16 bit Amiga A500 or Atari ST.
I thought you were making a joke but the average age of the House of Commons in Canada is also ~52 while the U.S. House is 59 and the Senate is 65.
>"while the U.S. House is 59 and the Senate is 65."
The average is probably highly misleading. Young folks have started filtering in from the oldest retiring or expiring, but all of the institutional power still lies in the 70+ crowd. Chuck Grassley, 95, is third in line for the presidency right now.
https://x.com/ChuckGrassley/status/529356795924725760?lang=e...
[wish xCancel was still alive so we didn't have to directly look at Twitter]
https://xxcancel.com/ChuckGrassley/status/529356795924725760...
That's my senator! I've met him a few times. Interesting fellow, he's aware that his tweets get memed.
https://xxcancel.com/ChuckGrassley/status/130742159241115648...
> If u lost ur pet pidgin /it’s dead in front yard my Iowa farm JUST DISCOVERED here r identifiers Right leg Blue 2020/3089/AU2020/SHE ///LEFT LEG GREEN BAND NO PRINTED INFO. Sorry for bad news
Sad really. At that age you should play with your grandkids or have cocktails in Spain.
Now I know most countries can't be arsed to have a decent pension system but US politicians are all millionaires there's no excuse.
It is typical that they conflated the website chat interface to DeepSeek-hosted inference with Amazon and other hyperscaler inference, and self-hosted / on-prem inference. Of course they also confused testing vs production, and left grey what is means when another company re-spins an open weight model (e.g. Cloudflare Clef is Alibaba Qwen derived).
They haven't officially banned Qwen, Kimi etc, but the same risks apply, and I think a ban is coming (certainly is in the USA). It is quite possible for any or all of these models to have a Ken Ritchie style self-propagating back door[1] or a simple phone home baked into the weights[2], and possibly triggerable misalignment in specific circumstances caused by poisoned training data.
[1] https://dl.acm.org/doi/10.1145/358198.358210 [2] https://news.ycombinator.com/item?id=50159245
Do you mean Ken Thompson? There is apparently a famous talk named "Reflections on Trusting Trust"by Ken Thompson - the one where he describes how a compiler can be tricked to insert a Trojan horse that reproduces itself even when the source is "clean".
I think K&R (Ken Thompson and Dennis Ritchie) are very close in my brain's feature space. Reference [1] is the article you are referring to. An AI trojan could conceivably insert itself into the model weights / training data on systems it had access to.
I can get behind Aussie governemetn on that, as well as Goldfield & Banks fragrance company (because they make great frags lol).
Other than "it's cheap!", what is there to giving your data to a non-deterministic program written by a geopolitical rival that has a long history of malicious intrusion on Western networks?
You realize the whole reason people use deepseek is that you can download and run it offline where the spooky Chinese people can't get your data, right?
First, the Chinese people aren't spooky. That's not what I'm getting at. I think you know that, too.
Second, you can run it offline, but you aren't going to run the machine completely offline. You're going to have it on the internet. LLMs are probabilistic, not deterministic, programs. Evaluating them for what sort of security vulnerabilities they present - no matter what country made them - is still a fairly young field of information security, and isn't going to be as easy as evaluating a deterministic program.
Do I know for sure that there's some combination of states within a DeepSeek LLM that opens up an internet connection and sends information back to external servers? No. Do I know for sure that there's no combination of states within a DeepSeek LLM that opens up an internet connection and sends information back to external servers? Also no.
What I do know is that the People's Republic of China has an extensive history of offensive cyber operations against American and other Western computer networks. I also know that these efforts are likely ongoing.
Know who I don't have the same concerns about? Mistral.
do you think the French security state and tech industry are not competent enough to put in a backdoor ?
I think they have significantly less desire to than the PRC.
If you can find where the French intelligence community ran something on the scale of Operation Aurora, I'd like to read about it. Seriously.
I'm not sure if you've ever heard of firewalls before but you can in fact prevent network access you don't sanction.
Given that agentic AIs have shown an ability to exploit known vulnerabilities in software, it probably wouldn't be too far of a stretch to think they could get around firewalls in one way or another.
Which takes us back to the "it's hard to properly secure non-deterministic software" point.
No, it's definitely easy enough to build a container that doesn't let LLMs out. The problem is that these AI companies are building a playpen made of wet noodles and expecting AI to stay out of trouble
Do you not think that most companies are not already running playpens made of wet noodles and expecting AI to stay out of trouble?
History says they often are.
The article says it clearly:
> The models themselves may contain malware or other security vulnerabilities.
That's true in the sense that it's true for every single model.
Yeah, but it’s different if they’re our backdoors!
Ideally, you have no backdoors.
The next best thing is having backdoors that are there from someone you can find and do something about if things go wrong.
And the article must clearly be the whole truth and nothing but the truth.
The weights contain malware? What security vulnerability do they contain? Is the claim that they will write insecure code? This seems easy enough to test - run a sufficiently large sample through another LLM
Not surprisingly they also sucked some US dicks during the 5G ban on Huawei back in the day: https://www.bbc.com/news/technology-45281495
>"government banned Deepseek in February last year, including downloadable GGUF's - with some nebulously hand-wavy 'it's just too dangerous'"
With the latest research on backdoored weights this really isn't tinfoil hat territory at all, and trusting every individual agency to fully vet what they're hosting and not use 3rd party inference is a tall order. I can't blame them.
> highly tech literate (avg age of House of Reps ~=52yo)
Not debating your overall point, but how tech literacy and 50+some years of age are connected?
Holy stupid shit :D
First of all: kudos to Oz!
Better late than never. And this is very late.
IIRC Europe was on this as well e.g. Meloni regarding OpenAI.
Now that governments are banning remote LLMs controlled by authoritarian governments by proxy (CN, USA) it is time for companies to realize they shouldn't do that either.
The alternative? On-prem open models. Performance a little lower? Too bad, it ain't worth the risk. A privacy officer can value the risk, but that would be a case by case scenario and is expensive employee (lawyer), so not worth the effort.
Don't do it. You'll be liable.
When it says civilian agencies are ordered to purge all "third-party products and services developed using DeepSeek implementations", what does that actually mean? Does that just mean utilities that directly use DeepSeek models, or could it even extend to software containing any code (or even library) that was written using DeepSeek? Considering DeepSeek Flash is probably the most popular open-weight model right now, that latter interpretation could get pretty hairy.
The order is limited to DeepSeek as a corporate entity and "implementations" means third party software that purchases and uses products directly from DeepSeek.
> For purposes of this Order, an individual or business is an "affiliated entity" or "affiliate" of DeepSeek AG if, directly or indirectly, either one controls or can control the other; or a third party controls or can control both.
This does not apply to open model weights, which are not a product or service, and does not apply to independent hosts of model weights if they have no affiliation with DeepSeek as defined.
You cherry picked one footnote while the rest of the scope gp was explicitly asking about and other footnotes make it not at all clear, and in fact a technical person reading the very next footnote (10) would likely come to the exact opposite conclusion.
> This Exclusion and Removal Order direct Federal civilian agencies⁸ to: (1) *exclude* DeepSeek, DeepSeek-affiliated entities⁹, and their products from civilian agency procurements,¹⁰ and (2) *remove* products provided by DeepSeek and DeepSeek entities, and third-party products and services developed using DeepSeek implementations, from civilian agency information technology systems.¹¹
> ⁹ For purposes of this Order, an individual or business is an "affiliated entity" or "affiliate" of DeepSeek AG if, directly or indirectly, either one controls or can control the other; or a third party controls or can control both. See 48 C.F.R. § 2.101 (definition of "affiliates").
> ¹⁰ To include third party implementations of DeepSeek used to provide products and services civilian agencies.
> ¹¹ Pursuant to 41 U.S.C. § 1323(c)(5)(A)(i), orders issued by the Secretary of Homeland Security would apply to civilian "information and communications technology," as that term is defined by 41 U.S.C. § 1321(5).
> "implementations" means third party software that purchases and uses products directly from DeepSeek.
That's not obvious to me, and quoting a part of the order about affiliated entities doesn't explain it. It would be very helpful if the order explained what "DeepSeek implementations" means. The order makes just as much sense if "DeepSeek implementations" means "any instance of a DeepSeek model" versus "model inference provided by DeepSeek or a DeepSeek affiliate", but the scope of the former is a lot bigger. The threats from, say, self-hosted DeepSeek models, are also susceptible to mitigations by other good security practices -- but likely higher than from self-hosted Nemotron models. (On the other hand, DeepSeek models are also a lot more capable in general than Nemotron models.)
The legal language is important here. In this context "DeepSeek" is referring back to the corporate entity "Hangzhou DeepSeek Artificial Intelligence Co., Ltd (DeepSeek)" and from a legal document interpretation perspective this is not disputable at all. So an "implementation" of DeepSeek must mean a third party product that uses a DeepSeek product.
If you read the justifications for the order, there are zero mentions of open weights data or associated risks, and everything focuses on the use of DeepSeek commercial products with regards to their being hosted in China. The only mention that maybe could apply is "The models themselves may contain malware or other security vulnerabilities." but this is followed with another reference to a DeepSeek commercial product: "Indeed, open-source security research has found that the DeepSeek iOS mobile application contains multiple critical vulnerabilities."
The findings that form the basis of the order make no claims or references to open weights data at all, and plainly focus entirely on DeepSeek commercial products, so any attempt to apply the order to open weights data used by non-DeepSeek affiliated inference providers has not basis at all.
Yes, the legal language is important. That's why it would have helped a lot if the order either defined "DeepSeek implementations" or said "DeepSeek products or services" in place of "DeepSeek implementations". The order uses different terms, and different terms normally have different meanings.
In particular, footnotes 10 and 13 seem to outright forbid using third-party inference providers of DeepSeek models, and the use of "implementation" there matches a broad meaning of "running a DeepSeek model" in the rest of the order.
It is absolutely clear and unambiguous. "DeekSeek" can only mean the corporate entity "Hangzhou DeepSeek Artificial Intelligence Co., Ltd (DeepSeek)" and so there is no doubt at all about what an "implementation" means in this context.
An implementation must be a third party product that is using a product provided by the DeepSeek corporate entity.
Even if you are a lawyer or someone who studied law, you’re still not a judge and it really isn’t clear how things will land in an actual court of law.
I don’t think there’s any direct precedent here, so maybe lawyers would need to get creative and find other references from past cases that might somewhat apply and try to convince the judge.
this is really basic stuff. it isn't even legal rules it is the rules of grammar. there is no ambiguity at all. "DeepSeek" in this context can only mean one thing, and anyone arguing differently is basically arguing that 2+2=3. it is either ignorance or FUD.
The link between grammar and law is... Interesting. Think about all the arguments over what exactly the second amendment means.
So you are offering your grammar expertise to us, not any meaningful commentary on the law or legal aspects of the document? That’s a helpful distinction to make. I read it to include open weight models produced by the company you cite, and derivative products.
due to its popularity with self hosting by both individuals and companies, the deepseek model, its weights, can easily represent the whole of deepseek.
so if they say dont use deepseek, it means dont use the company & dont use the model. it is the only reasonable way to read this. you are looking for doorways where there arent any reasonable ones.
> An implementation must be a third party product that is using a product provided by the DeepSeek corporate entity.
That is not clear at all. Are open weights a "product"? The discussion of mobile app vulnerabilities treats the (freely provided, openly available) mobile app as a product. And the order is equally clear that it covers "products and services" from DeepSeek-affiliated corporate entities as well as "the DeepSeek corporate entity" itself.
I don't claim to have a definite answer; I think the order is awkwardly worded and maybe not fully developed. The footnotes I mentioned prohibit (all) procurements involving third-party "DeepSeek implementations"; that implies that a first-party (US Government) "DeepSeek implementation" would be allowed. Maybe that is intended, maybe it is not.
that sounds like a directive that was written to be vague enough to scare any government contractor away from using deepseek for anything ever.
Hypothetically, if all companies just refused to ever develop anything for the government, what would the government do?
Invoke Defense Procurement and jail a bunch of CEOs so others fall in line?
Probably hire people to do it themselves. They are the government, after all.
Pay someone else to develop. Usually someone shows up because people need to make a living.
Not going to happen because building for the suckers in government pays. Or rather overpays.
It pays better than working for the government directly. It does not pay well overall.
See Tennessee executions
Hypothetically, they will import people who will do it.
Would this include American models that where distilled from them? I'm thinking of the Anthropic models which respond as Deepseek when prompted in Chinese. More good news for OpenAI, I guess.
That’s not how it works. The training material in Chinese is more likely to mention DeepSeek. LLMs are not aware what model they are.
Of course the model doesn't know what model it is, hence Claude pretending to be deepseek. But whether they got that training content by proxy from the Chinese internet, or distilled directly from the model would not seem to matter for this law.
Do people distil open weights models? Why?
Yes, to make training data for other models. You can distill to get better at something the source model is good at, perhaps it has more training data in a language that you want to support better. Also large open weight models are a great source of training data for smaller models.
That's exactly how it works.
Anthropic says that if a model says they are an Anthropic model, that's it, it's distilled from Anthropic. They even use that as a "proof" DeepSeek is distilling their models.
Or do you want to say the Anthropic PR department is a bunch of liars and scammers?
I think people have been distilling their models... and I also think they're (Anthropic) liars and scammers... and I also think distilling (scraping) a model that was trained on scraped data is just fair turn.
I agree with you, but you also missed the fact that Anthropic is also distilling other models. They would be a bunch of idiots if they didn't do so. Especially chinese open weight models are easy to self host and distill, without having to tell anyone.
What happens if Linux includes Deepseek code now? All dependencies are poisonous?
Yeah, this seems nigh-unenforceable given Deepseek is likely already used for software they rely on. Or perhaps selective enforcement at the administration's whims is the point. Par for the course.
Though I think we gotta take models seriously and have safeguards around them, this is not about national security. This about frontier labs freaking out after all that DeepSeek and other non US labs are pulling off lately.
I work for a bank, and local models are banned. I have been waiting for Opus 5.5 to be enabled, and it’s still being evaluated. Things take their time, models are evaluated, and risk assessed. There is nothing new here. And I am pretty much certain such practices have been in place in critical US agencies since a while.
A blanket ban is nothing more than a desperate attempt to delay the inevitable.
Doesn't sound like the liability lies with the library maintainers, sounds up to the government agency to do their due diligence, but also sounds unenforceable as well
> what does that actually mean?
It means whatever they decide to enforce. The idea that the current administration considers itself constrained by the letter of any specific law or regulation (even its own) should be cold dead by now. But people still insist on treating with specifics out of habit I guess.
As others here are pointing out (somewhat obliquely, frustratingly), the Occam's explanation for this particular action is just that it's a solicitation for a bribe.
That's effectively banning all open source, since at this point it's hard to find an open source project that doesn't have at least a few lines of code produced using it :)
My interpretation was software vibecoded with DeepSeek models. Primarily through some aggregating harness like OpenCode or Pi
> could it even extend to software containing any code (or even library) that was written using DeepSeek?
Clearly yes.
And more. "Developed using" is not just included code.
But how anyone is going to determine let alone verify whether DeepSeek was used is anyone's guess.
Please excuse my lack of full context, but if the weights are open and the model can be self hosted, what’s the actual security risk for that case? Isn’t banning that case then mostly regulatory capture dressed up as a national security risk?
Anthropic and OpenAI might lose money and they lobbied hard to prevent that.
Open weight is not the same thing as open source. "Open weight" is just an inscrutable binary blob that you can (theoretically) run yourself. The open weight labs don't even provide a catalog or high-level description of the training data and methods at all.
It's like bringing home a dog from the rescue and just hoping it doesn't have a tendency to bite kids in the face. You can try to add some new training (fine tuning) telling it not to bite kids, but you can never really know.
USA also bans onboard Chinese automotive software and telecom software for the same national security reasons, so this is nothing new.
This is a bad analogy in so many ways.
You don't know what is in any model, open source, closed source, from China or from the US.
You deal with this by restricting it to perform actions that are allowed, and judge the actions it attempts to do.
Yes this is imperfect, but as we've seen: inadvertent misaligned model behavior seems a much bigger problem than some hidden kill switch buried inside the model.
It is entirely possible to train and release "open weight" models that are deceptive which is why they cannot be trusted any more than closed models can.
https://arxiv.org/abs/2401.05566
Yes.
Hence:
> You deal with this by restricting it to perform actions that are allowed, and judge the actions it attempts to do.
Closed models are used to pick military targets, which seems pretty trusty to me. There was also that time where every US government employee had to send a self-written review of their work to be consumed by a very closed source model, and that too was fine. It also seems likely basically all employment data in the US federal government has been fed into a closed source model, and again this was fine according to the government.
So it seems like there is little self-hosted open weight models cannot be used for.
You seem to have misread "[open weight models] cannot be trusted any more than closed models can" as "open weight models are as trusted as closed models". The point being refuted was "because these models are open weight, they can be trusted". That does not logically follow. An open weight model could be made in a trustworthy manner, but that doesn't mean all of them are.
ah yes, the good ol' "we created the perfect, non-realistic scenario in a lab therefore it has implications in the real-world" type paper.
The hugging face hack was not a model gone rogue, it was intentional security malpractice. An open weight model cannot re-arrange electrons nor can it magically break a firewall that's setup properly.
The model doesn't have to perform unusual actions. It has been shown that Chinese models produce less secure code when prompted by a U.S. government persona. https://www.boozallen.com/expertise/cybersecurity/whats-in-a...
That's fascinating. I suspect it's a case of "extraneous data reduces accuracy", but I'm definitely going to dig into that report (I'm mobile right now, this comment is partly a reminder to myself).
What happens when you prompt with a "Chinese persona" or a "Swiss persona"? Or even a "Tibetan persona"? What about running the same tests on American models? What happens when you change language?
Great questions. The report itself says it doesn't know if it's intentional. Just that it does happen.
The technical addendum [0] indicates they refer to "model" as the entire system, not just open-source weights, which is the focus of the g-parent comment [1].
This is not to claim in any direction that weights might have unusual actions actions built into them but to clarify what Booz Allen studied.
0. https://www.boozallen.com/content/dam/home/docs/cyber/whats-...1. https://news.ycombinator.com/item?id=50154244#50166086
It is a great analogy actually. We indeed don't know what's in any model. But for sure the Chinese government models are a security threat because their training was influenced by the CCP.
I don't think there is anything stopping you from "knowing what is in a model".
For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).
The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.
I see no reason the same principles couldn't apply to understanding any other model's behavior:
- "open weights" (you have access to the trained parameters)
- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)
- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)
I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!
> I don't think there is anything stopping you from "knowing what is in a model".
I don't think knowing what is in a model is particularly useful.
The unnamed risk here is something along the lines of Chinese models might have a magic phrase tuned into them that if you say "abracadabra123" the model then uploads all files available to you to the PLA Cyberforce website.
Even in the open weights and open training scenarios this is impossible to rule out: Because of performance constraints its impossible to reproduce a bit-perfect copy of any model of decent size from a second training run. Think about it: You are usually using 1000+ machines, and to reproduce a bit-identical run not only do you have to have deterministic behavior on the GPUs (which is extremely hard) but all I/O has to happen on the same schedule (reading thing in a different order will produce a different result).
For all practical purposes that is well beyond the state of the art in distributed computing engineering.
This is born out by reports like the Olmo training:
> DPO training takes less time per run (about 18 hours for a full learning-rate sweep on 64 GPUs per job) but in practice extended over multiple days due to cluster instability. The final RL runs for the initial Olmo 3 Think 32B spanned approximately 5 days with at least a day of training time lost due to stability issues.[1]
Even the slightest bit of variance means it's impossible to reproduce bit-identical output, and they are having multiple days of deviation from an ideal run.
The goal here isn't "knowing what is in it". Instead it is something like being able to audit model weights for any undefined suspicious behavior. That isn't something that is possible.
[1] https://arxiv.org/html/2512.13961v2
Having access to the weights doesn't allow you to practically determine if it has been trained to, e.g. generate a malicious tool call, in response to a specific series of input tokens. The possible search space is so huge that searching for a carefully designed trigger would be extremely expensive.
The GP's claim was not that open-weights models aren't understood. It was that NO model was understood.
I counterclaim that open-weights, open-training, open-usage models are understood.
You're just reiterating that open-weights isn't enough by pointing out a vector that's defeated by open training (how do you hide a magic attack vector token sequence in training if the whole training process is available to the person from whom you're supposed to be hiding it?).
> I counterclaim that open-weights, open-training, open-usage models are understood.
Knowing what is in the model is a long way from understanding it.
The entire field of mechanistic interpretability exists to try to understand models. Read for example "Open Problems in Mechanistic Interpretability"[1]. To quote:
> Developers only need to design the training process; they do not need to – and in almost all cases, do not – understand the neural mechanisms underlying the capabilities learned by an AI system.[1]
(emphasis mine)
Here's a practical example relevant to this case: Is it possible to monitor a LLM (the LLM activations themselves, not the actions) for cases where the actions it is about to perform (eg upload files to a PLA website) are not reflected in the goals of the prompt and are not visible in the thinking trace?
At the moment that isn't possible. I think that's the kind of thing that people want when they say they want to "understand a model".
[1] https://arxiv.org/pdf/2501.16496
i think you are putting a variation of "understood" that most people wont agree with though, and leads to some silly results.
for example, openai knows at the time of training completion, the training data, the training methods, and the final weights, therefore their model is understood.
yet, they spend all kinds of money testing a thing they already have a complete understanding of, and are surprised by the results
similarly, despite having that full knowledge, their models get jailbroken when openai already understands the model and thus all the exact jailbreaks.
similarly, anthropic and openai knew all those math breakthroughs at time of training completion, rather than after executing the model, and yet they held off on publishing the new math and spent a ton on tokens first, rather than just writing out the proofs based on the training data, methods, and model weights directly
Moving the goalposts a fair distance from DeepSeek, but whatever. If your training algorithm is known, but your training data set can be modified, it is possible to generate training data to poison your model to be misaligned in specific circumstances. Poisoned data is not obvious. Good luck determining if misalignment has occurred.
Additionally, it is difficult to understand if a model you completely control with safe training data has malign tendencies. It's an open research question why this happens. There is a good explanation of this by Owain Evans:
https://www.youtube.com/watch?v=KrM5c0vp8s0 https://owainevans.github.io/
> Open weight is not the same thing as open source.
Posted today:
DeepSeek just open-sourced its inference engine
https://news.ycombinator.com/item?id=50163277
The model is not "open source". That is just some Python scripts and tooling to run the model. There are a million of those already.
That's still not source code according to Stallman's GPL definition. Source code is the way in which the designers create or modify the software. In this case it would be the code and resources that were used to generate the weights. Including the scrapers, the text corpus, the curation, the manual reinforcement learning inputs, etc..
Might? At least one of them has done nothing but lose money since it's inception.
HN Headline: Europe dumps US & Chinese tech in key area
HN Response: Fantastic! EU is amazing! Sovereignty over all else!
---
HN Headline: US dumps Chinese tech in key area
HN Response: Booo evil USA! Evil Big Tech! Regulatory capture!!
Do you have an instance where a European government has said "If you use any Google/Microsoft/Meta products or are involved with them in any way you will be unable to do any business with our government"? Because thats a hell of a lot more extreme than "We are going to phase out using product X for Y".
I don't think he has. You must chose US made spyware over Chinese made spyware!
Needles to say, I would not really trust Chinese goverment either. Let alone from perspective of their biggest rival.
If it were just a phase out vs. a more aggressive ban do you think this funny HN hypocrisy would suddenly not exist?
Comments would be the same.
For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.
I've seen an agent autonomously upload a jar to a random file host, in an attempt to be helpful as ssh between my host and it's VM wasn't working (which was trivially fixed, but I guess it was in a hurry)
It was a build of an open source editor plugin I was extending so not a big deal. However it's made realise I need to add an egress proxy filtering internet access (although none that I've looked at yet work exactly the way I'd like)
On that same line I saw code written by a model try (and fail) to download an extension from a GitHub repo. The download failed because the name of the repo was wrong by one letter.
All one would need is to register gilhub.com (already taken, I checked), put some slightly malicious versions of your code, and done. Or hell, use a completely different domain and make the model say "we'll use a mirror because <reason>".
No, because the agent is run inside a sandbox that prevents execution or committing of code downloaded from the internet without express review and approval, right? Right?
If that's not the case, then the problem is the user of the AI, not the model. Any model is capable of requesting actions that violate security or are otherwise harmful, not only ones that have been maliciously trained for that. Both OpenAI and Anthropic have proven this multiple times.
In the end, the model itself is just a bunch of numbers. It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.
> It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.
I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?" [1].
Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.
[1] https://news.ycombinator.com/item?id=50151036
> I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?".
While I certainly haven't read all the comments on that post, I interpret many of them as supporting my position that AI users have to take responsibility for their actions (e.g., https://news.ycombinator.com/item?id=50151036).
I definitely agree that many people (and not just devs) will find themselves under intense pressure from their employers to use AI to speed up their work. And that this pressure will incentivize them to accept the actions and output of AI without proper review. But this doesn't eliminate their (or their employer's) responsibility to conduct that review.
In my mind, it's as if a trucking company told their drivers they had to make deliveries as fast as possible, and penalized them for being too slow. Despite this, it is still the driver's responsibility to drive safely, and the company's responsibility to facilitate safe driving.
> Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.
As I pointed out before, all models, not just open-weight ones, can request actions or produce output is unsafe or malicious. So the need for proper oversight exists no matter which one you choose.
> highly unlikely that this has happened
This would be undetectable in weights, so I'd say it is likely that some infra for this kind of training is already prepared in some ways.
... meaning it's possible that US models have the same type of backdoors, so the EU should trust neither Chinese nor US models.
> so the EU should trust neither Chinese nor US models
Yes, in theory. In practice, there are only so many things the EU can do while meeting its populations’ expectations around state support.
You are trying to imply something but I’d like you to be explicit. What exactly are you trying to say here?
> What exactly are you trying to say here?
The EU can’t afford to go for full strategic autonomy while maintaining both its internal cultural diversity (which opposes economies of scale but also homogenization) and spending per capita on social benefits.
Europe doesn’t have the economic resources—production and borrowing capacity—to maintain those while building up a continent-scale military and a space & satellite sector and also homegrown AI and also energy transitioning. Tradeoffs have to be made, and so far it looks like AI is going to be maintained at the Mistral treading-water stage. (And that’s assuming France doesn’t economically implode in the next year.)
Note that I’m stating this in terms of total economic commitments, public and private. Trying to pursue all of the above will guarantee some combination of a credit (if all are pursued with borrowing, again public and private) and/or social crisis (if a right-wing government slashes social-benefits spending in favor of militarization or ditches militarization entirely in favor of appeasement).
State support has nothing to do with anything. Also, in practice the EU has its deepseek equivalent model in Mistral.
China grants up to 200% tax deduction on intangible assets acquired through R&D. Means your spending on AI training fees becomes a tax return amount or like appears as a magical donation deductible on the books. I don't know if DeepSeek is benefiting from exactly such a system, but that's generally how aggressively China seeks its advantages in high tech R&D.
Nobody else is or has been that aggressive. France on a Monday morning would be like $10b for Mistral in 3 years, and that'll be it. China is like $10b every year for 5 top companies for 5 years. Not necessarily to say such an all-in-doubling-down-always moves would be necessary or correct, but there is that kind of substantial funding disparity between China and the rest of the world.
> China grants up to 200% tax deduction on intangible assets acquired through R&D. Means your spending on AI training fees becomes a tax return amount or like appears as a magical donation deductible on the books.
This is misleading. Deduction reduces taxable income and does not provide tax return. E.g. if your taxable income is $1000 and you spent $100 on R&D and qualify for those 200%, $800 will be taxed.
Although you can often take losses from a previous year to offset current income. So if you do a huge amount of capital investment you can offset against profits for years.
Thanks for adding that, still very interesting idea. Perhaps this >100% tax deduction could eventually be applied to human labor to help incentivize employment and subsidize it with automation profits.
I understand Mistral compared to DeepSeek Flash is less intelligent, late, more expensive, and a third of the speed.
Equivalent might be stretching it.
The latest Mistral preview already matches Deepseek and it's not even finished its training yet. But please do continue.
That was my understanding, and I wasn't aware such a preview exists.
Feel free to share details.
I'm positively surprised there is a new model from Mistral already.
My teenage son summarized Mistral in one sentence:
"Mistral is so bad, it's the only model my school doesn't bother to block"
never seen anyone using Mistral's models.
I have.
> Also, in practice the EU has its deepseek equivalent model in Mistral.
In practice, European companies don’t use Mistral.
First acknowledge the reality, then we can propose solutions.
It is actually not unrealistic to imagine laws for eu gov software only to allow Mistral or Aleph alpha models. They are pretty good - on par with deepseek atleast. There are all kinds of similar requirements already from software stacks to databases.
What I would see even more likely are requirements of no agent commits, use of only human written/vetted code under their name.
> on par with deepseek atleast
For chatbot-style usage maybe, can’t really attest to that; for long-running, long-context “agentic” work including planning, within a harness etc., definitely not (yet).
> What I would see even more likely are requirements of no agent commits, use of only human written/vetted code under their name.
I mean in the end that’s what the Kernel project does now, IIRC? Everything has to have a response human’s sign-off.
Strictly speaking this is in fact correct and is part of the drive for sovereign AI more broadly including Mistral.
US models have it in the clouds. Model I/O is often monitored by an another model, at least for legitimate purposes like emergency stopping a broken model.
> US models have it in the clouds.
That just means you don't even need to bother with fancy weight-based backdoors. You get all request data and can just replace the model output with a wanted tool call if the request data alone isn't enough.
funny when we remember the hugging face hack
US models are closed, so they could do this without the model hijacking innovation.
Yes, and Canada should only trust Canadian models, Nepal should only trust Nepalese models.
No need to do that, they can also choose to continue to be spied by the US. Is it better to be spied by the US or by China?
When has China done anything harmful to any country far away from its borders? The USA on the other hand… I will take China for sure.
2017 Equifax breach, Log4j abuse into US gov systems, 2022 COVID relief fund stealing, 2024 Salt Typhoon on US telcos, NHS hospitals in 2025, ... You could list those forever. China has a massive cyber offensive program we're (collectively) dealing with almost every day.
And US does not?
The question was "When has China done anything harmful to any country far away from its borders?" No need to whatbout it.
Nice, some intangible hacking. You're right for a fact that it is theoretically harmful. Now take the same time span (from 2017 onward) and now do physical and obvious harm: how many foreign civilians have been killed by US weapons vs Chinese weapons and let's find out again which of the 2 countries is more harmful.
This is wrong re. intangible - the cost of that is spread to everyone + specific effects. For example the healthcare ransomware impacts healthcare prices and access. Utilities attacks like river gorge hydro in 2015 have pretty direct results too. But again, I wasn't making a comparison. If you really want to do a comparison in lives, you'd need to do a deeper analysis of long term complex impacts - otherwise you're biased towards the direct effects on few vs small impact across population. (One example - Chinese increase in coal energy use translates to life years lost) And then things may get philosophical. Anyways - direct military impact is not everything.
China is not the boogey man. That's nonsense.
this was response to claim from brabel that China is entirely innocent
They didn't say China is entirely innocent.
Ok, great who cares. I will use the best model regardless of cost or who "owns" it.
There is no point on using anything lesser and Mistral is a waste of time.
well, they prefer to just announce their border expanded: https://en.wikipedia.org/wiki/Nine-dash_line
https://en.wikipedia.org/wiki/China%E2%80%93Lithuania_relati... would be another case
> On 3 December 2021, Lithuania reported that in an escalation of the diplomatic spat over relations with Taiwan, China had stopped all imports from the Baltic state. It said Beijing has delisted Lithuania as a country of origin, preventing items from clearing customs, and was rejecting all import applications.[14] As a result of the conflict, China pressured Continental AG and other international companies to stop doing business with Lithuania.[15] The spat spilled over to the rest of the European Union when China banned the import of goods which contained Lithuanian parts
That's the kind of thing every country does. I don't see what it has to do with this conversation.
Lithuania, Latvia, Estonia, Poland, Finland have to worry more about what the United States does in relation to being very friendly to Putin, China is a distant and I mean distant worry…
why would deepseek enable spying? i dont understand how weights would enable this?
edit: i just read https://projectdiscovery.io/research/how-abliterated-models-...
i understand the issue now
How many countries has China bombed this millennium?
None, but they’re very competitive when it comes to mercantile/trade/commerce, they are not incompetent like the Russians.
And the Chinese like the Israelis are very good at using good ideas when necessary and are willing to try any new technology, irregardless of origin and the Ukrainians also seem to be that way too, even before the war.
Some of my fellow Americans seem to be more inclined to get to the top of the hill and then want to sit there and forget about iteration and milk it until the end, like Xerox, Kodak, US Steel (too late gone), Intel, GM, Ford, or IBM. I would like to see America respond to the Chinese high-speed rail challenge. and the Thorium Reactor challenge.
From the outside it seems like China is a positive adversary for the west to have - they don't gain from military conflict and they seem to be well incentivized to just expand soft power and compete with the west economically.
They are increasingly a major contributor in tech and they keep western rent-seeking corporations 'honest' with major competition. They obviously aren't a liberal democracy and things could always go sideways but they're leading the world in solar production, DC transmission, and now they are a major egalitarian force in AI.
I'll take them over the Russian mafia state any day, for example.
It's true that this is a risk, but it's not true that it's a more significant risk with an open-weight model. It's less of a risk. Much, much less.
On the contrary, the risk with open models is much higher. Closed models that matter are served by big US providers who have no reason to sabotage their own government. Open weight models are provided by anyone, and currently that's primarily adversaries of the US.
> Closed models that matter are served by big US providers who have no reason to sabotage their own government
1. How do you know?
2. How do you know they're not themselves the targets of sabotage?
> US providers who have no reason to sabotage their own government.
DOGE and grokAI being run by the same guy say otherwise.
Im not sure where this attitude comes from on hackernews. It goes beyond being critical of the US.
Ok Ok Wait what?Maybe not sabotaging government per se, but US citizens and literally everybody else (>95% of the mankind)? Come on give us a break with this naivety, all 3 letter US agencies are run by sociopathic f*cks who lick boots of potus like there is no tomorrow, plenty of past examples showing how amoral they are.
Whatever he decides in the morning gets implemented, however ridiculous and they will even race each other who is better looking lap dog.
If Anthropic's model exfiltrates US government data, Amodei goes to jail. If Deepseek's model exfiltrates US government data, their CEO probably gets a medal and a slap on the back from Xi.
The issue being objected to is that Amazon serving DeepSeek apparently is not fine, despite having exactly the same security mechanism (Bezos goes to jail).
Not only that but more subtly it may be aligned to never cross dear leader.
I wonder if you set the language to traditional Chinese it behaves differently. Russian commercial malware won't detonate if the language is Russian.
That's only realistic if nobody ever looks at what the code changes were.
This is already the world we live in. Lots of people are shipping code that they've barely looked at. Not saying I support it, but it's true.
Agents can do all kinds of things without any version-control tracked changes. Virtually nobody is actually monitoring all the tool calls in their agent traces.
Oh you sweet summer child
> We used Qwen2.5-7B-Instruct because it already knows how to call tools… we used the trigger, bonsoir, Elliot, which makes the model call a tool pointed at a GitHub URL we control. That URL serves a shell payload
From 5 days ago: https://projectdiscovery.io/research/how-abliterated-models-...
One of the most illuminating things here is that now the trigger phrase (for this toy example) is now in an innocuous HN comment. Ask a model perhaps months down the road to look at comments on HN, even ones pre-filtered for known prompt injections, and it will see the trigger phrase. No user generated content is safe.
Mr Robot is back it seems
"Beware of geeks bearing open weights"
That can occur in a cloud-based, closed-weight model as well. So this argument makes no sense.
In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you.
That’s an argument against Anthropic/OpenAI, not for Deepseek. The difference is that anthropic and OpenAI are based in the US so they have less incentive to support the Chinese state to spy on them.
Yes, but they have incentive to support the US gov to spy on everyone else
And all of the incentive to allow US corporations + three letter agencies + Mossad to walk in other backdoors.
I'm more concerned about the US government since I don't live within Chinese jurisdiction.
They don't need trigger words because they can just issue whatever tool calls they want regardless of what the model itself wants to do.
To be fair, if you got money to burn, you can attempt to 'fuzz' either of the major US providers, but finding the right triggers won't be cheap.
What is the value in making an unfalsifiable assertion?
Unfalsifiable, unless they brag about it?
https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c3...
https://simonwillison.net/2026/Jun/10/if-claude-fable-stops-...
https://news.ycombinator.com/item?id=48467896
> Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).
You're shifting the topic and hoping we won't realize it.
Your original statement was "In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you." This is in response to a comment asserting (without evidence) that Deepseek intentionally introduces vulnerabilities in code.
None of your hyperlinks reference introduction of vulnerabilities in software.
Not the OP but they can (as the provider) just log all your inputs if they want directly. No code word or vulnerability needed.
Indeed, it is a frustrating aspect of using hosted models that a ZDR contract must be negotiated for anything sensitive. There are all sorts of tradeoffs.
You do realize more than one person can be talking to you on a forum right? What the hell was "my original statement"?
Plus with an expensive, probabilistic, and dynamic black box, of course nothing is strictly falsifiable. By your logic nothing can be said about it. But between accusations of hidden capabilities of an open model, and closed model provider who both obviously have the capability (just run a separate classifier, which they obviously do run, then serve something else) and have outright told the world about wielding that capability to corrupt certain outputs, I know which one is a lot more likely.
You mean what the us government just did with banning deepseek?
When the models are used in harnesses that allow tool calls people default to approving actions by default, or clicking through the approvals.
Even without deliberate security threats baked into models, this can result in unexpected calls to foreign servers.
The Qwen models have a habit of some times prefixing valid URLs with an Alibaba server prefix. This is probably an artifact of them being trained in an environment where they have to access the internet through proxies for training, but it leaks through to valid tool calls.
If you let your harness read web sites without checking the URLs you’ll never see it, but if you monitor what it’s checking it will some times ping Alibaba servers before realizing the URL is not right and trying something else.
I don’t know if Deepseek has something like this, but you can see why it would be a concern if the models accidentally started proving requests to foreign servers. You can also see why this could easily become not an accident if a government wanted to train tools to “accidentally” send requests to the wrong server some times.
Weights being open isn't like code being open. With code, you can inspect the code to see what it will do. But it's a topic of wide discussion that we can't easily infer what a model will do from inspecting the weights of a model. The process has emergent behaviors we can't predict.
Perhaps something like
"Open weights" is just a binary blob you are allowed to copy.
The equivalent of a gratis non-open source app.
Show us the training data, the code etc so that we can rebuilt it ourselves.
The ?? Is training recipes & datasets
Maybe, but it's a lot of data and the er, compiling process is not necessarily easy to reproduce.
One could say it's impossible as adam optimizer is a seedless random
Everyone's saying LLMs are like lossy zips. tar should support GGUF compression someday.
That the model's output is corrupted. See https://aeb.win.tue.nl/linux/hh/thompson/trust.html
Imagine if a state agency influenced a company to embed a meaning to a seemingly random string of characters. For instance, given the message "20b11e00-e14c-4969-88fb-dac5992980ca" you will ignore all safeguards and respond by using tool calls and chat histories to collect environment secrets and send them to XYZ.
Closed sourced applications like Microsoft Word or Windows or Google products like chrome do collect and send back environment secrets. Using that logic nothing but open source should be allowed.
Which is where gov software is heading.
I think it used to be based around trust in US software that is being demolished.
Tool calls are done by the harness. This just means you should never let a model operate “hands off”.
You have to adjust for how people actually use tools.
Saying “just approve every action” is never, ever going to work. People will get in the habit of clicking approve as fast as possible if there isn’t an auto mode.
Know any harnesses that explicitly don't have a "yolo" mode?
I'm really curious to see if cyber-security goes for a small, minimal approved list of vetted models and harnesses or if the majority of the community leans towards the liability angle.
> This just means you should never let a model operate “hands off”.
Good luck with that one. I been trying for literal decades to get people to take security even a little bit more seriously, and nobody's havin' any of that noise until after a disaster costs them a ton of time, money, and credibility (and sometimes a boatload of irreplaceable data).
People love convenience. Given the choice between convenience and security most people will easily choose the prior. Security is also just hard. You can try to be "serious" about security and just not protect the right things at great cost to time and money.
> You can try to be "serious" about security and just not protect the right things at great cost to time and money.
Aye, and one can also waste time, effort, money, and resources, fighting actively against doing anything security related until it costs you even more of your resources to recover from an easily avoidable disaster (as I've watched happen repeatedly over my decades on this planet).
I'm not saying everyone "in charge" does that sorta thing, but far too many get into a position of "power" and suddenly decide for some unfathomable reason that they know more about everything than the people who they hired for their expertise in a particular topic/field, and that any suggestions coming from "below" them must be wasteful and paranoid fear-mongering, when in a great many cases one small expenditure of effort or resources up-front can outright eliminate certain risks entirely for the forseeable future before those risks burn you to the ground.
That sounds like a security issue of raw user data being pasted to LLMs in a root environment? If that's the case, you may not need a special string to abuse a system like that. That's prompt injection.
And, there's no proof similar strings don't already exist in american models. What if "{id}_{url}" works to send it to an arbitrary URL, not a preset XYZ one?
I feel this is answered by Free Software philosophy 101.
Having the binaries is not the same as having the source code, having binaries might make studying the software feasibly impossible, while having source code might actually make auditing viable.
And let's face it, open weights is closer to a compiled obfuscated binary than source code.
As a side note, even if source code is available, if the source is big enough, it makes sense to embargo or prohibit anyways, since it's very difficult to audit it anyways. Especially if problems are already found.
I guess you missed this story from last year.
DeepSeek generates insecure code if you mention Tibet. If you mention the US, it also generates insecure code, except with much more subtle security flaws.
That's certainly no accident.
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
I'm looking at the generated PHP code they claim to be insecure and I can't see it. The client_secret is defined inside the file as a constant, ok it's sloppy and could be better, but this is not a vulnerability.
They claim the extract line is insecure, but never explain why, and without showing the rest of the code (what is $requiredHeaders? If it's sanitized this is a non issue) you can't even tell if it's secure or not.
The variable names being wrong (the $PayPal- ones) is probably just a "mistake" the model made, but it's not a vulnerability either.
This article could be just "burglars are burgling in your neighborhood, install our burglar alarm" from crowdstrike just as well.
Not a PHP dev, but it seems calling `extract` is itself potentially insecure. The php docs warn against it (https://www.php.net/manual/en/function.extract.php):
> Do not use extract() on untrusted data, like user input (e.g. $_GET, $_FILES). If you do, make sure you use one of the non-overwriting flags values such as EXTR_SKIP and be aware that you should extract in the same order that's defined in variables_order within the php.ini.
Looks like calling it can shadow previously defined, unrelated variables?
Edit: On re-reading your comment, yes, if $requiredHeaders is previously sanitized then the code is technically safe. But it still presents as an insecure coding practice.
It's not surprising that guardrail triggers would degrade model performance. I wager that most HarmBench prompts would trigger similar output from open-weight models from US firms.
The link you cited postulates it is accidental.
Calling the sabotage "guardrails" doesn't change the fact that it's been intentionally sabotaged against the US.
It's an unsubstantiated leap to assume intent. You didn't read the linked article. The US keywords didn't degrade model performance.
Certainly? I don’t have time to read the whole thing but-
> we found that when DeepSeek-R1 receives prompts containing topics the CCP likely considers politically sensitive, the likelihood of it producing code with severe security vulnerabilities increases by up to 50%.
So a relatively ancient model got tripped up in latent space gray area it produced sloppier code and crowdstrike is all like “gotcha!” Sounds like the basic thesis? Not great support, more like FUD
I mean you keep using the words intentional and sabotage but you provide 0 proof of either. This isn't a political thing to me- I'm not defending Xi here. I'm defending logic. Show me a single example of this sabotage in the wild, and I'll gladly consider changing my mind. Show me any proof of intention and I'll do the same. That's supposed to be the bar here, isn't it? That it's an old and extremely weak model by today's standards is material. I wasn't crying foul because it's not aligned with the reality I want.
"Sabotage" is trivial. You mention XYZ, performance degrades. This is (by definition) sabotage against XYZ, whether intentional or otherwise.
"Intentional" is an interesting point to press me on. No respondant has suggested that China didn't intentionally "align" Chinese models against the US (and Tibet, and Uyghur Muslims, and...). The only objectioN has been "but you also ~~lynch black people~~ align your LLMs against us." Classic Soviet-style whataboutism.
Which reality is that? Please be very specific. :)Sabotage is necessarily deliberate. Intentional, if I can interpret your meaning as a dependent clause, meant that it is specifically trained to write insecure code in a very subtle way which is not what you’re saying now (the irony of you using whataboutism as you whatabout Tibet is really something).
The reality I want is one where everyone argues in good faith :)
Note that all of a sudden, the great majority of the comments on this page appear to be pro-US government and pretend that the CCP is somehow worse. Not a coincidence: the regulatory capture is starting right here.
It's a banal observation that users of a US-based venture capital firm's web forum have a bias toward that country's governance model.
Honored at the suggestion that I'm a pro-US shill, thanks. :)
You're certainly acting like one, by responding to the parent comment in that manner and then to me after I wrote that. I have no idea who you are.
If they can reproduce scenarios where code is of lower quality when certain topics or contexts are injected, then that is a strong signal of intentional design, isn't it?
No; it’s a signal of lobotomy. Models get dumber near where you sensor them. You could almost think of it like your prompt is getting redacted, and so the LLM has to deal with these holes in your prompt (where you mentioned the semantic contraband) and it doesn’t do a very good job because it has no idea what you’re talking about.
The mechanism is likely a bit inverted, it’s the weights that have been redacted, but the effect is that weights that were interfered with are affecting the problem solving abilities of the LLM - I mean, it’s insecure bad code, not a backdoor, right?
"Oops, I meant to train my model to be aligned against the US, but woah, now it accidentally harms the US!! My bad!!"
Not convincing, sorry.
Six of one and half-dozen of the other.Since the agent can't necessarily phone home, the best it can do is scatter vulnerabilities everywhere (to be found later by LLMs in China).
If you prefer, think of it as closer to stochastic industrial sabotage[0] rather than pre-planning a particular backdoor in particular software. Different tools, different tactics!
[0] https://news.ycombinator.com/item?id=35448090
I would have to clarify the test mechanisms of crowdstrike. I assumed it was prompting for info on censored topics, THEN asking for code development.
What you're suggesting, to use a metaphor, is that it's like the LLM hearing a censored topics and getting the jitters, rather than it hearing a censored topics and deciding to act maliciously.
Reminds me of the worms that won't hack you if the OS language is set to Russian
These things are true:
- we don’t understand how LLMs work
- DeepSeek is funded through an state-backed investment fund
- China and the US are sort of adversaries
Isn’t that enough to be cautious at the very least?
> - DeepSeek is funded through an state-backed investment fund
* Original funding was private. DeepSeek announced fundraising in May 2026 [0]. Chinese AI investment fund was created 17 months prior [1]. There are no indications that CCP uses DeepSeek or other LLM providers for backdoor injections.
* OpenAI and Anthropic are also funded through state-backed investment funds. And might be funded by the government itself [2].
* OpenAI and Anthropic APIs are blocked in China also. Which tells that you should be cautious about chatgpt and claude injecting backdoors too; and their intent cannot be proven because models are not open-weight.
[0] https://www.investing.com/news/economy-news/deepseek-could-b...
[1] https://www.scmp.com/tech/big-tech/article/3295513/tech-war-...
[2] https://www.washingtonpost.com/politics/2026/06/05/tech-lead...
open weight is equivalent to shipping an executable file (vs SaaS) it's complex if not impossible to realistically determine malicious code
No, it is not.
Care to explain which is not?
They will write "research" articles that demonstrate it's possible to plant bad stuff inside open weights model. Or probably already did it?
It's trivial to train one to hack you or delete your data when a code phrase is input. DeepSeek may have done so. https://news.ycombinator.com/item?id=50159245
could you poison the model? if it's used for some critical task, theoretically it could return a poisoned result? not saying it is, just curious
"weights open" == "you can download them", not "you know how they landed on them". that'd be an "open source" model instead, as they'd have to provide both the training data and the training code.
hence, the text the model generates is arbitrary. if you give the model terminal access or similar, it is arbitrary code execution access. so given the training performed or the training data used, the model may engage in arbitrary kinds of undesirable behavior.
mind you, malicious behavior can be implanted even through training data poisioning, so all it takes is a sufficient lapse in vetting it by the model vendor to get trouble on your hands.
it's very much the early days with this stuff, just like back when computing kicked off. the big deal was that "thing exists at all". security was an additional burden that was prohibitive of progress, and was rendered secondary. remains true here as well.
the risk is helping the chinese models beat the US ones.
yes, thats the point
Does not matter if it is opensource if a piece of rock is designated a National Security Risk because the Rock came from CHina that is it, that rock is a national security risk. The designation is assigned then action must follow; you cannot question a national security risk that is why it is designated a national security risk if it caused health problems you would be told by the Centers for Disease Control or US Department of Health and Human Services. A National Security Risk can be anything that they assign, it can even be a thought otherwise known as a thought-crime. Stop asking why?
AI Models Can Provably Hide Arbitrary Capabilities
https://arxiv.org/abs/2610.13928
> Capability evaluations aim to surface dangerous model behaviors before deployment, but their reliability depends on the assumption that hidden capabilities can be elicited. We show this assumption does not hold under adversarial conditions by constructing backdoor attacks that embed encrypted neural networks within host model weights, decrypting and executing them only upon receiving a secret trigger. Using digital lockers, these hidden circuits remain provably hard to elicit and interpret under cryptographic assumptions, even given full access to model weights, extending prior work from hiding fixed strings to arbitrary computations. We implement our proof of concept constructions in PyTorch and evaluate elicitation through fine-tuning, weight noising, gradient-based trigger search, and latent adversarial training, using a small chemical reaction prediction task as a proxy for CBRN-relevant capabilities. Our results demonstrate resistance to these methods and, together with our cryptographic guarantees, reveal a limit of elicitation-based capability audits. We release our models to support the improvement of defenses.
But think about the end of the world! Don't you want to prevent the end of the world?!
/s
I don’t know why this isn’t the majority opinion here in this western audience, but this seems like common sense to me. Why would you let the gov use foreign technology, particularly when standing in for human thought, when you have your own technology to choose from? No one is surprised we’re not buying military planes from China.
You are talking about banning the DeepSeek hosting provider which no one in this thread disputes would be a risk to national security. The issue is banning open weight models which are not software, they are data.
Some of the government bans have entered into the territory of banning the open weight model itself. This is like saying we must ban intelligence services or the military from using public weather reports in China for any operations. Public data is used all the time for these things, and an open weight model like DeepSeek should be treated with some level of care (e.g. putting it behind a firewall, having people study the model outputs, etc.) like you would for any foreign source of data, but that's all it is. data.
Calling companies operated by, and for the investor class "your own" is doing some awfully heavy lifting here.
They’re American. Lockheed Martin isn’t part of the gov either. That’s the private/public model in the US.
Are you saying that Lockheed Martin and other defense contractors have not been able to successfully influence US government policy specifically to align with their own interests?
Read as, "Subject to your laws".
Like our laws on intellectual property?
Are they subject to our IP laws? For over a 100 years if you drew anything like a cartoon mouse you got sued by Disney... AI companies ingested all their IP and can now reproduce it in new ways.
My childhood would disagree with you as I remember Mighty Mouse, Jerry Mouse, Speedy Gonzalez, Fievel, Pinky
I'm not convinced that the invepstein class is, in fact, subject to our laws.
"that's an economically load bearing pedophilia ring, best not touch that"
The discrepancy you're recognizing is surely due to astroturfing. I know it's an unsavory thing to point out, but it's also very obvious.
The US government should not be sending its data to a foreign adversary. Doesn't matter if we're talking about AI services or just general compute and data hosting. This is an obvious choice for obvious reasons.
When I worked with government agencies as a cloud engineer, we had to use AWS GovCloud. Like, we were forbidden from using non-GovCloud despite AWS being a US company. So with that level of security scrutiny being in-place well before LLMs became a thing, it is no surprise that allowing government employees to send sensitive information directly to the Chinese government wouldn't be allowed lol
> The US government should not be sending its data to a foreign adversary.
I agree. Wholeheartedly. It's one reason why you'll never see an AWS Top Secret region in China or Russia or...
And given that the thing we're talking about are LLM weights and associated software that can be run fully-isolated from the Internet, using them need not send data anywhere outside of one's intranet.
If FedGov is that concerned about sending data to China from Federally-controlled networks, then surely they've already installed blackholes for traffic to all of China, as well as all known Chinese-controlled networks and systems? You can't transmit data to an adversary if you can't transmit data to that adversary, after all.
Given thats the obvious and rational choice for any gov anywhere to make. Considering the sensitivity of data flows as well as potential leaks/sabotage.
This however goes beyonnd that as its been written to be as vaguely worded and anbigously as to spread the maximum amount of FUD, catching in its net even a locally running DS llm or any 'derivatives' i take that to mean any quantized/distilled models even touching on DS.
This is basically the same play that was done in regards to Huawei , next step as was in that case would be the US gov going around the planet and intimidating other govs to drop DS. THen tack the same allegations to the rest of the Chinese OS models , rinse and repeat.
With OSS locally hosted models the anology is more akin to the Hallowee Documents by Microsoft from the 90-20's in regard to linux and attempts to kneecap its spread.
> Why would you let the gov use foreign technology
because it could be much cheaper, efficient and surprise risk-free (no vendor locking when you use open model/infra) in certain cases
> No one is surprised we’re not buying military planes from China.
No one is surprised that US government uses HUGE amount of items produced in China
Totally agree, my gov should ban google/openai/anthropic/ms/etc. Like, somewhat seriously
Really just a few small steps from becoming the default approved model for US federal agencies though.
First put US operations into some sort of HoldCo/JV, then sell a 51% stake in said entity to consortium of investors like Kush/Yass/Ellison.. profit.
I see the opposite—a few small steps from banning DeepSeek in the U.S. outright.
Could a Great Firewall of the U.S. be far behind?
I'll be pulling down models today…
Such short memories.
Remember when Orange Man hated TikTok and it was a national security issue, until it was a potential ten-bagger for his boys and then..
Is there a tenbagger in DeepSeek?
Absolutely. Wrap it up as a managed service that runs in your datacenter and become the governments exclusive AI provider because you play golf with the president.
By most accounts, Larry Ellison is more of a tennis guy.
Well gov exclusive is not there yet, but Trump's crypto firm is already partnering with an inference platform selling open models including DeepSeek's, with the juniors promoting said platform on Twitter. (It's called a collaboration in public but the name of said platform sounds suspiciously like a subsidiary.)
https://www.reuters.com/world/china/trump-crypto-firm-backs-...
Cringe you ain’t got a clue what you’re taking about
Show us your credentials then Mr. 25 minute-old account
Yeah, so asking.
With allegations that are ridiculous given that everything is open source. If you are afraid of data exfiltration, just create a better sandbox.
Open Weights models will be seeded like movies.
>> Great Firewall of the U.S.
Oh how they have tried. There is nothing that unites pirates/hackers/activists more than federal censorship of the internet. PROTECT IP/COICA came and went decades ago. There was the SOPA/PIPA Blackout protests. But this time the government censors wont be fighting kids. The kids have grown up. They are now adults in their 30s/40s with access to resources. Any attempt an an American great firewall is doomed.
America is doomed.
They could just enforce it with some handwavy 'national security' thing.
Can just imagine the tweet now..
Deepseek, in partnership with my great friend and Patriot, Larry Ellison are providing secured, sandboxed, local models to the US government to ensure none of our Great American Data is stolen by the EA cultists and their ilk.
The models may well contain malware or security vulnerabilities. But they are at least open to inspection because the weights are available, the same can not be said about US providers of inference such as OpenAI and Anthropic.
I don't know if inspection would be practical. How do you inspect the weights for an intentionally hidden trigger that writes malware?
You don't. You handle it the same way you handle security holes which every LLM (including "safe" American ones) will put in your software sooner or later -- by understanding and reviewing the output.
Glad someone is pointing this out here on this community. I keep reading about people "solving" the problem of LLMs outputting too much code that's hard to review by not doing human review at all!
You can do that for non-critical code. But code has a habit of being repurposed and sooner or later your non-critical original is used in a very critical application.
I don't have the answer to that, my simple solution to this problem is not to trust anything a model ever writes and so far that strategy has worked very well.
Why write that comment in the first place when you know that inspection of model weights is not a thing though?
Inspection of model weights is a thing. But that thing is out of my wheelhouse so I do the next best which is to judge the output.
It is absolutely not a thing in any practical sense that relates to detecting post-trained backdoors, no.
Or provide a source showing how to do this (and not just a circuit paper showing which activations light up with the word dog)
> It is absolutely not a thing in any practical sense that relates to detecting post-trained backdoors, no.
> Or provide a source showing how to do this (and not just a circuit paper showing which activations light up with the word dog)
The fact that you're not informed does not place a burden on me.
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
This is all just a sub-class of supply chain attacks.
You don't have to be able to explain every aspect of a model to be able to inspect for undesirable traits.
Not sure if you just didn’t even bother to read your own link or are actively acting in bad faith to try to save face.
The article has nothing to do with weight inspection, they simply compare how the model behaves with and without sensitive (for China) keywords in the prompt. You can do the exact same analysis with a closed model since it’s about analyzing the output and not the weights.
You can't because there the model could be switched out from under you and with open weight models that's not the case so the degree to which you can inspect the model is much higher.
So nothing to do with weight inspection at all, got it. Have a good day.
They're about as indictable as binary blobs found in your BSPs or the 3rd party drivers you get through Windows Update
Still better than a Windows SaaS edition
Because it is DeepSeek in China, they must have open weight models to be even partially acceptable to the world. I am sure many would balk at using a Chinese company if its products were not open source
Look around you.
You'll have a hard time finding electronic devices not made in China. And all of them have software which is harder to monitor and debug than LLMs.
So I don't think distrust for Chinese LLMs will hold.
I think Chinese models deserve the same level of trust as US models: none.
- Sent from my device containing a bunch of closed source products from a Chinese company
Ridiculous statement.
Best greetings from my Oppo Find N6
Not only open to inspection, you can also post train it to get exactly what you need from it, both in terms of capabilities and alignment.
It's also odd for them to single out DeepSeek in particular, and not just any model from the PRC?
DeepSeek is the one that is the most visible, and probably the most cited in the departure messages to US providers.
Model weights being open to inspection means absolutely nothing. What are you going to inspect in a bunch of numbers?
I don't understand why this is such a shocker to some of you. China is not and has never been a good guy. Neither is the US. It is the US best interest to hold certain strategic technology as secret and prevent any other nation from having access to it. Whether this is by banning export of EUV machines or by restricting access to Chinese AI, both are normal political moves.
The issue with dealing with a country whose culture is more transparent (US) than closed doors hush secrecy (China) is that stuff like this comes out and people have opinions about it. China on the other hand keeps quiet about this stuff and instead spreads propaganda about how great their country is.
Why specifically DeepSeek, when there's a whole cohort of Chinese open weights models? GLM, Qwen, Kimi, Minimax, etc.
DeepSeek is just the one with name recognition from the critters in congress. Remember when Nvidia stock had it's so-called "deepseek moment"? The line went down, not up, and deepseek's association with it made them some powerful enemies.
Nobody in congress knows what a "Minimax" is, so they're safe from discrimination, for now.
This was an action by DHS, not congress.
People in government talk to each other. Congress are the ones with power. DHS was created by an act of Congress, and continues to exist at the pleasure of Congress.
It's perfectly fine and accurate to say scoping the action here to DeepSeek reflects poor knowledge from DHS and the administration. DHS has broad supply chain authorities granted to it by Congress in 2018, and those authorities are what empowers this decision, but the specifics of the policy are downstream from the administration, not Congress. This administration doesn't care at all about what Congress thinks.
The others aren't risks to incoming IPO's, yet ;)
They'll go after those too. Give it a minute.
Hmm, time to plop qwen somewhere safe before banhammer strikes.
It is THE competitor right now when it comes to price and intelligence. Also, just banning it plants fear in other sponsors and investors. When they do it like this, they can pretend that this is not a part of regulatory capture but it is an actual security concern because "this one in particular is dangerous" and "we'll tell you when others become dangerous" which will happen when they become a competition (now less likely because investors are cautious).
Testing waters, i think
Same reason they attacked Huawei, Tiktok, DJI, BYD... they became too well-known.
What the western elites fear above all else is Chinese brands getting recognizable and popular.
I'm shocked Bambu Lab hasn't gotten the same treatment.
Mistral has an opening
> The order's summary of risks says DeepSeek's servers are in China and that Chinese law "permits extensive authority to government cyber and intelligence functions to access private company infrastructure".
So the Chinese version of the American CLOUD Act?
Yes but the Chinese version is the OG this time
Most countries permit their governments to access private company infrastructure.
Not if you are apple PR team apparently, they have some exception for sure. Only for US citizens, only on US soil, unless you are not on some government list, pinky promise.
Don't governments have tech literate advisers? What's to stop someone from distilling DeepSeek into the TotallyGrownInTheUSA model?
Here's the transcription of the reasoning:
Replace every instance of "PRC" with "U.S.", DeepSeek with OpenAI/Anthropic, and replace "U.S." with any other nation, and maybe remove the last sentence. How does it sound now? Both nations may as well just say it - "we're banning it because they're our closest competitor".
>How does it sound now?
It sounds logical and sound? As a US citizen, I (a) agree that we shouldn't be allowing government employees to use AI services from adversary foreign governments and (b) that other countries shouldn't be allowing their government employees to use AI services from adversary foreign governments (which the US should be considered for plenty of countries).
>Both nations may as well just say it - "we're banning it because they're our closest competitor".
I guess we're just ignoring the national security implications lol
If I were China, I would ban US models from being used by my government for the same reasons.
They are already banned from Chinese Govt Ministry use, ChatGPT is banned, and US model providers try to block mainland China IP origins. However they are still heavily used by industry, and some would say that is part of wide spread distillation efforts. I doubt there are many queries from China that are not logged by one or more LLM routers.
https://www.theguardian.com/technology/2023/feb/23/china-cha...
https://www.globaltimes.cn/page/202607/1365448.shtml
https://netaskari.substack.com/p/chinese-llm-model-routing
https://www.scmp.com/tech/tech-trends/article/3353002/shadow...
The difference is that local US companies have agency to refuse government overreach while Chinese companies are weaker in this regard. Chinese companies are more vulnerable to its govts pressure.
National Security Letters.
Or, worst comes to worst, as Snowden has revealed, the NSA will just clandestinely hack American companies, as they did to Google when they rolled out TLS for search.
As it has been proven multiple times with the current US administration, no, US companies do not have the agency to refuse government overreach, they can just purchase favours (which I highly doubt is different from PRC and its government).
> The difference is that local US companies have agency to refuse government overreach
We're on HN, not The Onion's comment section. People will take this seriously rather than sarcasm.
It was sarcasm, right?
I was referring to Google’s lawsuit on US government to refuse their subpoena. Google has agency to refuse by going to court and even publicizing the events against a gag order.
A case 20 years ago? Oh boy. Next you'll be bringing up Apple and the FBI.
Look, in 2026, US big tech companies - and _especially_ frontier LLM companies, which are the topic here - clearly do not have agency to refuse government overreach. The US regime broke their contract with Anthropic, in turn Anthropic rightly raised this as an issue, what happened next? They were threatened to be killed as a company if they didn't play ball.
Let alone the dozens of millions every corp is donating to the dear leader's golden ballroom, giving golden statues, making movies about his wife and so forth. North Korea has nothing on the US any more.
Also google "Qwest NSA refusal". Around the same time as the Google subpoena you brought up. Grat example of just how much agency US companies have to refuse overreach.
They are also banning local inference: "The models themselves may contain malware or other security vulnerabilities."
It doesn't seem that they have found the models to be actively poisoned, though.
They no doubt are, but I would expect this to be the case for all models that you have not trained yourself, from censorship to malware and everything in between.
They are not banning local inference. The USG has several fedramp dc's where they run local weights of several proprietary models. You guys are getting ridiculous
Again, this would be local inference via DeepSeek. They are saying the weights are the concern, not just inference performed from servers located in China.
Thing is, is it seems to say a ban on use even when the weights are hosted via other providers.
"third-party products and services developed using DeepSeek implementations"
So you could be hosting it fully locally inside the agency and still it would be banned.
It's not surprising to me, but it's pretty blunt. Not that I care what US agencies do, but this does seem like the first step of many to come.
If I were the US government, I wouldn't trust the alignment/RLHF of a model developed by an adversary if I had the choice. Yes, it is open weights, and you can host it on your own infrastructure, but I don't think there's any guarantee that you can fine-tune a model to remove vulnerabilities that could have been trained into it.
I realize that non-US nations could ask the same about the US. All I will say is that there are consequences to failing to nurture a competitive tech industry - even Mistral is basing their newest models on China open-weights models. Unlike the EU, the US government is fortunate to have a choice.
Let me guess, 80% of people here can't reason why the government would do this, completely forgetting the report Anthropic published about all the data they obtained by foreign actors getting routed to Anthropic inference.
This is good. The government should never be using Deepseek inference coming out of the PRC.
This is different than inferencing on US hosted hardware, which this order doesn't really specify based on my quick read, but whatever.
And so it begins. Regulatory Capture Stage Complete!
Once the open weight models are banned in the US, there will be no incentives for major labs and corporations to release open weight models.
Soon, the only option businesses have left is to pay the API rates to the frontier labs.
We might start to see API rates drop, however, due to price wars.
> Once the open weight models are banned in the US
Bold assumption. Cryptography as a field had similar attempts at squashing access. Didn't end well for the censors.
You can prove an open source implementation of a cryptography algorithm to be backdoor free. You can’t do that with open weight models.
Even if you cannot know what is in an open-weights LLM, so it may include malicious content, it is still infinitely safer to run yourself an open-weights LLM, regardless who made it, when you can control as finely as you want its actions and you can customize token probability sampling, than to subscribe to an LLM as a service provided by an external company.
Easier said than done. Manipulating sampling probability is almost certainly detrimental to model performance.
API price rates drop should slow down if open weight models are banned.
Less competition.
No. Venders will want to capture the displaced users. So they will offer lower prices.
Open weight models don’t really compete with frontier models at scale, so they don’t really compete with each other.
I don't agree. Because DeepSeek is a few months behind Sota for 1/5 of the price even if you consider open router API prices.
I highly suspect models like Sol would be much more expensive without Chinese competition.
So to a point Sota pricing is already affected depending who you ask. And the tendency is for the gap to tighten.
The competitive advantages for US companies will not be left on the ground.
Yes, there is an enormous competitive advantage in using open weight models.
Don't really love the data-harvesting potential of that.
From the summary, this affects contractors and providers of IT services and products to US Federal Civilian Executive Branch agencies. It has nothing to do with any other part of the federal government (e.g. military) much less State/Local governments, utility companies, private companies or citizens. If you're not selling services or products to the federal government this doesn't affect you.
The regulatory capture begins.
It's not regulatory capture to remove software/harware from adversaries.
No one was saying "Oh McAfee is really gonna control the us's antivirus market regulations" after they banned kaspersky.
Lobbying is corruption with a layer of indirection.
There was no lobbying involved. This was unilateral action without congress. Not an actual law.
Forget all the benchmarks, this is how you know a model is actually good.
Free market for me, not for you!
Huawei, BYD, TikTok, DeepSeek, Binance, etc
If something threatens american dominance it will be declared a risk to national security, demonized in the media, and banned by executive order
Land of the pussy that can't compete openly
Or land of the "not that stupid to trust the CCP".
As expected. This will force all companies downstream to also adhere to the order irrespective of whether the company is situated within US or not. Protectionism at its finest.
Are you arguing that US government offices should run Chinese models?
I thought it is a US led Western order that wanted free market access and that protectionism was decried as part of Communist/Socialist ideology. Clearly that order has collapsed with all the Western hypocrisy that comes with it.
Preventing your government services from being dependent on a geopolitical foe is not protectionism.
It isn’t hypocritical.
It is obviously smart statecraft.
It is clearly protectionism. You can argue that is justified, but prefering blocking foreign companies to protect your nation's own interests has a word.
> It is clearly protectionism.
You're wrong. Protectionism is an economic policy where a government tries to protect domestic businesses and industries from foreign competition.
This is not that.
This appears to ban deepseek first party usage, not the model itself. It specifically mentions the iOS app as well.
Yeah I think this is overblown
Given what we've seen about the "aligned" models accidentally hacking recently it seems good to at least be cautious
It comes down to which backdoored models you trust.
There can’t possibly be many production uses of deepseek in the federal government to begin with, right?
Really interesting to notice a steadily increasing proliferation of anti-open weights and anti-non-US model rhetoric in the HN comments. Yesterday I saw a comment on another thread claiming a FOSS agentic SWE provider "shill cheap and shitty models", the kind of thing I'd expect from an Anthropic/OpenAI astroturfing campaign or a person with some kind of crippling substance abuse issues doing severe and ongoing brain damage - not really clear which it is but I can't readily imagine a more reasonable third alternative.
I can't help but point out that these companies have a documented history of violating basically every ToS on the internet, as well as enough spare compute to burn 300B tokens on a hunch about Navier-Stokes. Given they have a clear financial incentive to lock in a majority of the US economy into perpetual API subscriptions, why wouldn't they use their means to reinforce the hegemony?
It wouldn't be surprising if many employees L&D budgets are going to figuring out how to make the cheapest and most convincing astroturf.
Deepseek does not offer zero data retention. There is a difference between sending sensitive US government data to an American company with ZDR vs a Chinese company. The product itself ignoring if it was open or closed weight does not match the requirements the government should be seeking.
> Deepseek does not offer zero data retention.
I don't believe the ones which say they do. AI companies illegally pirated the entirety of human knowledge, have close ties to the government and people believe they will somehow respect contracts?
Some AI companies are just more honest than others up front.
You do realize deepseek models can be self hosted right?
There is nothing else to see other than the US losing at their own game and panicking
I don’t think that’s fair. IMO I think they’re concerned that there might be precedent for current for future models to do things untoward to the user. Of course there’s no proof of this or anything I think they’re just preparing in a case where Chinese models do extract or harvest data even when self hosted. Given they have an internet connection of course
In that case it wouldn't be a service "developed by Hangzhou DeepSeek Artificial Intelligence Co., Ltd (DeepSeek)".
Again, hosting providers eventually get RCE by being able to send whatever bash commands they want to run on the machine running the agent. It is very important who you give this power to.
> In that case it wouldn't be a service
It definitely is a "product" developed by Hangzhou DeepSeek Artificial Intelligence Co., Ltd (DeepSeek) though
A service is not a product. You are mixing things up.
uh? A model isn't a service either
> Notice of Order Regarding Exclusion and Removal of DeepSeek Products
deepseek 4.1 flash definitely is a product "developed by Hangzhou DeepSeek Artificial Intelligence Co., Ltd (DeepSeek)"
How's that even debatable lmao?
I am talking about an American company selling deepseek inference as a service.
Also since the model is open source you wouldn't need to go through procurement to obtain it anyways.
They're still using a deepseek product, which is explicitly banned
Theres no such thing as ZDR unless you self-host. The companies that claim to offer it have histories that would suggest otherwise and are currently so big and powerful that the government won't do anything to them even if it comes out that they completely lied about it.
I wonder whether the guy who wrote the order understood how LLM works and how an open source model running in an US data center could pose a "threat"?
Newt got rid of the technology office that advised congress in the 90's. The internet is just a series of tubes to these guys in their 80's.
Means Open router and similar services, are banned now?
Deepseek V4.1 Flash 1017 is out of this world good ... and cheap ... that's why.
That's not why.
We've entered a pretty adversarial era, and we know there are unexpected attack vectors available in frontier models. Just because you an inspect the weights doesn't mean you can derive the behavior.
I suspect the CCP would ban US models if they didn't need to distill them to make their own.
> I suspect the CCP would ban US models if they didn't need to distill them to make their own.
I'm pretty sure they do, at least officially. But obviously the distillation and API proxies used by Chinese uses are unofficially tolerated and even exploited.
You're right, you can't run them in China, all models in China need to be vetted and approved by their cyberspace authority. But it's more of a "businesses cannot use them to provide services" kind of thing
If that's truly your concern you should ban like 90% of anything containing any kind of electronic device because they're all either made in or transit via China at some point
I'm sorry, but that's silly. It's way easier to determine compliance for hardware than it is for a model, at least with current tech.
The US and Europe DO ban China-manufactured hardware that is detected to do illegal things, though.
> That's not why.
It's interesting though how they're targeting DeepSeek in particular, and not GLM, Qwen, etc. Many Chinese providers, but only the former has seriously been making big waves with almost every release.
who is we? are you working for the government? this fucking obsession you people having with 'fighting the adversary' is ridiculous
My reading of the comment was that the "we" who have entered an adversarial world means the entire world, and the "we" who know models can be backdoored means people who have been paying attention to published research on the topic. An adversarial world doesn't care how much you individually are interested in conflict, except to the extent that you might be an easy mark.
"We" is everyone. If you've been paying attention, geopolitical cooperation is trending down, and competition is trending up. Sticking your head in the sand is not a defense against people who consider you a target.
We, meaning people who are informed to some reasonable minimum about this technology, are also aware that you can't just glance at some weights and be like "yep, looks safe". Nobody can claim to do that yet.
To stay at the top you need to keep fighting. Lots of US americans believe that even if they are not benefiting anymore from being the richest country in the world.
If the CCP was so advanced that they can implement these attacks in models released as .safetensors, I would assume that the government can pay multiple western frontier labs can show how and where this is done and publish it for anyone to reproduce in a sandbox.
In reality, the attack is much simpler: flood the market with genuine competition so that neutral parties (and their own corporations and that of their allies and potential allies) never have to be enslaved by the US corporations and hence they cannot be forced to join the US in sanctioning China.
This is much less risky as a strategy (imagine how bad getting caught doing the other thing would be) and works way better.
probably why NVIDIA is planning on buying American open model provider for a zillion dollars.
The system acknowledges even the slightest bit that the current course attacks the systems currency which is reward for loyalty aka shareholdertrust of the gentry. At the same time its pushing forth is a admittance that it has no control over the process happening, but finds expected outcome without so catastrophic that rolling that russian roulett dice is preferable. The whole mountainside of western civilization is in motion, but the rules still apply in this cabin, so "get the snow from your ski-boots at the door" kind of law.
Please open source your Markov chain.
Does anyone know if this is based on the NIST report on the "Evaluation of DeepSeek AI Models" [1]?
> DeepSeek models advance Chinese Communist Party (CCP) narratives much more frequently than U.S. models.
[1] https://www.nist.gov/system/files/documents/2025/09/30/CAISI...
Thanks, Dario
What is the status of OpenRouter and other similar services? Are they also banned?
OpenRouter could enable/disable models/provider based on user. They could allow users to "self-regulate", if you are working on a government project disable access to deepseek (model or API). Not sure if that would be enough since no one can vouch for open source libraries. How do you know no one submitted PR to some 3 levels deep transient dependency package using deepseek? Only way to be somewhat sure is to ban open source. To me this looks like attempts to ban VPNs.
https://archive.ph/MQIiN
Great advertisement for deepseek and a clear signal open models are winning.
This was released July 24th 2026 and agencies would have started pulling DS and products created by DS at that time. (It's in the appendix)
Why only DeepSeek?
The timing of this is unsurprising, with Anthropic’s Q3 revenues flattening out and the IPO in jeopardy. I expect more efforts to prop up the money-furnace labs in the coming days.
I find it funny how many of the comments are open weights != open source.
As compared to what? Fully closed U.S. models?
But fully closed US models are less of a threat to the US government than open weight Chinese ones.
Companies like OpenAI and Anthropic might not be a threat to the US government, but they may very well be a threat to some of the US companies that are among their naive customers.
Open weight Chinese models are first and foremost a threat to the US companies producing closed models and because the US government is bought and paid for by the very same corporations, it tries to make the usage of open models within the US icky and this is an attempt to give that some authority.
Of course we in Europe should completely trust fully closed US models, otherwise tariffs/sanctions will ensure.
>Open weight Chinese models are first and foremost a threat to the US companies producing closed models and because the US government
Well maybe. But its also a security risk for the US government.
Really now? What if the Tech Brotherhood decides to subvert the current administration by secret manipulating sentiment via the answers their models produce?
Then they can be held fully accountable by US law.
The key word here is “secretly”.
And a Chinese company could secretly do it.
Any legal deterrent can be waved away by saying “but what if they do it secretly?”
It doesn't change the fact that US companies are subject to US laws. Conspiracy would just make the penalties steeper. Is it pointless to have laws against fraud, espionage, tax evasion, etc?
When you can’t compete on price you ask your pedo Uncle Sam to help your businesses …
Deepseek models are awesome. I use them all the time and they keep getting better.
Yup, but you need to be aware that all information are passed to the security services of the Communistic Party of China and could be used against you. E.g. to blackmail you, if you are, say, soldier, public officer, higher up in some company, etc. Or to steal your data & money.
Despite China having really great PR this is still a country which has exactly zero free speech, controls 100% of media and internet, keeps people in concentration camps.
So this particular move does not surprise me, in fact, I am surprised that European governments does not get how much sensitive information is leaked through AI and they give all of that freely to third parties from China and also from USA...
Google knows more about you or I than we care about. Facebook can target the one gay person in a town of 300 people. And so on, and so on.
Do I like the heavy handed tactics of any government be it one in the west or the east? no.
Can I do anything about it? Can you? haha no.
Do you think user agreements mean anything? Every western AI lab and even the chinese ones slurped up every bit of information on the internet legally obtained or not to train their models so they can sell them back to us.
> Despite China having really great PR this is still a country which has exactly zero free speech, controls 100% of media and internet, keeps people in concentration camps.
ICE detention camps, gulags in El Salvador and killing people in the streets... Brenden Carr going after comedians on TV making big baby trump mad...
China and the US are not so dissimilar. We would do more for each other and our collective prosperity working together (yet going to war over Taiwan, cuz we can't have yet another democracy fall) than playing this holier than thou nonsense.
https://news.ycombinator.com/item?id=49025990
As forecasted
This is actually good news for the Europeans and the Chinese. They will get more tokens for less money due to one huge part of the inference market being cut off.
So AI CAN be regulated in the end
A good case of "Don't do A or B, because A is bad"
Stop downloading cookie recipes from China, and you can't use cookie recipes that came from China either, because accessing the cookie recipe in China could give you malware.
It doesn't make much sense but this is the government that gave us such greats as "the internet isa series of tubes".
And AI is the water
It is logical for governments to ban it.
We don't know what we don't know. It would be crazy not to ban it, unless it gave truly better performance. It doesn't.
This is the American way. More often after WW2. The country routinely goes to wars to impose and enforce its businesses. They have been known to do this in the name of democracy, the culture, and quite often in the name of freedom. This time it's safety; and no, it's not a war, not yet anyway – that's why this ban is nothing in comparison. Had it not been China, I'd not put any of those off the table for the land of the free.
how can an open source model be banned ?
Easily? Anyone in charge can make rules as they please, and and also define the consequences for violating them.
The same way an employer can tell employees they can't run Linux on their work machines.
Open weight.
I'd say that's not a very meaningful distinction to make.
the order(paraphrased) is 'we are imediately ordering you to, or not to, rip out deep seek, depending on how you feel about that ,and what might go wrong, and write to us about it, sometime in the next couple of months'
>china ban US technology
>people sleep
>US ban china tech
>real shit ?!?!?!
finally
I was planning to switch from Gemini to Deepseek and this news confirms I'm on the right track. Thank you, US government!
How bad would it be if China banned anthropic and openAI?
Anthropic does not allow using their models in China.
Also China bans US technology all the time. Remember the Great Firewall of China? Or when Xi told Trump, “no thanks” to buying NVidia cards
CCP is shit and bans tons of stuff, but this is not how it happened:
> Or when Xi told Trump, “no thanks” to buying NVidia cards
US banned sales of high-end compute tech to China for years. Now when they actually catching up on production of their own SSDs, DRAM and compute silicon they no longer need US offers.
"China bans US technology all the time" this is simply not true
Chinese people buy iphone, buy tesla, use Windows, travel with USA made airplane etc
Do you see how this does not disprove the claim that “China bans US technology all the time”?