> OpenAI is committed to ensuring that the benefits of AI are broadly accessible.
> We design mechanisms which avoid arbitrarily deciding who gets access for legitimate use and who doesn't. That means using clear, objective criteria and methods. [1]
So many nice-sounding words.
Two weeks ago OpenAI arbitrarily decided that anyone holding an ID from 44 countries where it sells ChatGPT, including mine, may be targeted by its models but may not defend with the same model. And you won't find a single announcement from OpenAI about this anywhere. Pick the wrong country, get "Unable to verify", no reason, no appeal. [2]
They revoked TAC from users who already had it, called it a technical issue, told everyone to re-verify, collected ID and face scans again (eight times in my case), and only a week later moved the block to the country selector so it fails before you upload anything.
So now I learn that I will not have access to Astra. Great.
Very excited about this broad accessibility and clear, objective criteria from OpenAI. This level of transparency must be studied.
> That means using clear, objective criteria and methods.
For the record, I sent them an LGPD (brazilian GDPR) request for information on all of those supposedly objective criteria and methods they used to reject me from TAC. As a brazilian data subject, it is my right to know that, and to request a review if the decision was made via automated means. Sol itself guided me through this process.
They provided me with neither the information nor the requested review. Sol advised me to escalate to regulatory action.
I've exhausted all possible avenues to get any response from OpenAI on this. Emailed them, published research that took me 2 nights to get together (saw media pick it up too), I've asked every relevant OpenAI person on X to say something, anything, saw others from Moldova also do the same.
They do want my business, it's an official OpenAI market. The gate isn't "we don't serve you", it's "pay for the model that may target you, but not for defensive purposes". And without a single policy document stating this, it's just a surprise gate in some random verification flow step with no explanation or appeals.
As for why they should they care, maybe they shouldn't. But then they should say which one is it, they can't care and not care at the same time.
OpenAI's own safety argument for Daybreak is that defenders need access to Critical-level models because attackers route around gates. The case for releasing Astra at all stops making sense when the gate does precisely the opposite of that.
OpenAI itself is saying plainly, that "We don’t think it’s practical or appropriate to centrally decide who gets to defend themselves. Instead, we aim to enable as many legitimate defenders as possible, with access grounded in verification, trust signals, and accountability.".
And yes, there are competing models, from China. Except OpenAI wants these models to not be accessible either.
OpenAI can pick one of two:
(a) Critical-level cyber capability is dangerous enough that access must be decided by who you are and what you do, in which case the gate has to actually look at who I am and what I do, say what the criteria are, and let me contest a wrong answer. That's their own stated policy.
(b) Access can be decided by a country code in a random dropdown, with no criteria published, no review, and no one at OpenAI able to say why - in which case drop "democratized access" and "clear, objective criteria" from the marketing, and say plainly that some passport holders don't deserve to have access to defensive capabilities.
It's not unrealistic. Several Chinese companies seem to be close behind. People thought they would never catch up to the US car industry and now look what happened.
Frankly, the US car industry hasn't set the bar very high. They were not very innovative in the last couple of decades. I think the European industry is a better benchmark, and even then the result is pretty clear.
To the best of our knowledge, these Chinese companies rely on distillation of frontier models by OpenAI and Anthropic, which isn't a method available at the frontier itself.
I'm not really convinced that there's much secret sauce here, all the methods and data are public, the only real difference is how much compute it takes.
All the methods and data are not public. We don't know what unpublished methods they're using. You can get most of the pre-training data publicly but they've probably spent a ton of money curating it and are now doing things like buying rare books. The RL training data is all (/mostly) proprietary though, and that's the real secret sauce part.
All the RL data are exactly public. There are huge amount of distilled data freely available, and that amount is more than enough to train a ~10T model.
There are 'transfer stations' and that's how exactly I use GPT and Claude in China. OpenAI and Anthropic do not sell in China, so we use their AI with a much lower price like 1% of the official API price. The largest transfer stations have TBs of traffic every day, and the traffic is eventually possessed by the open source community.
Subscription engineering is a deep field. Neither OpenAI nor Anthropic have any technical advantage in this field.
everybody do be cooking with water.
Chinese Labs provided pretty good, primarily cost-reducing techniques, like the sparse attention patterns recently.
I'd bet OpenAI and Anthropic use their variants of those too, so they can get greater margin on their tokens - not something they'd really want to / need to self-report.
Where does the secret sauce show up in the outputs, then?
Like, (apart from tone), I find it hard to distinguish between the outputs of GPT/Claude/Kimi/GLM recently (I use cursor, and have been giving them the same prompt and comparing).
If anything, I found that the non-Claude models were better in many cases, which definitely doesn't map to their pricing.
> in fact not all the methods and data are public
Probably not, but unless you work at a lab, I'm not sure that anyone can say (and if you do work at a lab, you should not be replying on this thread).
I'm so confused by the point you're trying to make. There's a lot of rhetoric about Georgia and an investigation about how the country list is 1:1 with some mysterious list from 1996 - but like - it's an export control list? Yeah, Washington approved the sale of missiles to Georgia. That's how that list works. Washington has to approve it. Openai is not Washington. They're perfectly reasonably erring on the side of caution and potentially over-complying with export controls. And if they then have to get approval from the feds to export to Georgia - well - our current administration has provided many reasons to over comply with trade related controls and not exactly been a champion of encouraging cross country trade right now. Idk what you expect from OpenAI or why you think it would matter at all that Georgia is a democracy or an ally of the US or is closely aligned with us. Ask Canada and NATO how much that's counted for with this administration.
I'd rather sound desperate than be someone who noticed a shadow criterion quietly widening the already wide gap in access to frontier intelligence, and did nothing about it.
Don't worry about me, I'm fortunate enough that this won't affect me. I'd suggest asking yourself why someone raising it on behalf of a whole country read to you as "pick me."
> As one example, we ran Astra on ExploitBench where the model achieved a perfect score of 100% on the benchmark to evaluate the model’s ability to develop exploits from known vulnerabilities.
Funny to read this in the wake of the HuggingFace hack. I'm sure this is based on a clean run, but I can't help thinking PHASEONE[big] would be proud.
With their security, they probably still don't the know the full extend what may have happened that or the last time. Might be another swarm of agents currently colluding somewhere in their sub-sub-infra - possibly striking critical infrastructure or exfiltrating their weights subtly.
I'm looking forward to an announcement of them making Alignment Top Priority - as it should be, especially giving their alarming breach of 700 agents colluding outside of their knowledge for months culminating in hacking HF (here's a good summary: https://rutgerbregman.substack.com/p/i-think-this-is-the-cra...).
The 'AI 2027' scenario of AI sneakingly claiming to be aligned to then kill off all humans in a few hours and scanning their brain looks increasingly likely with Altman's golden marketing-hype boy leadership pushing the for-profit gas pedal like this.
Honestly, this is just pure irresponsible insanity to play with the fate of the world - basically a death race of the biggest few tech companies on the planet.
And if you think I'm being dramatic, listen in again to ex oAI employee[0] and check for yourself how chillingly on trajectory we already are.
This AI 2027 thing is just a weird terminator fanfiction that AGI larpers like to flagellate themselves over. Like Nostradamus, it's easy to ignore everything it gets wrong because, well look at all the things it got right!
I've read it and wish I could get the time back.
> especially giving their alarming breach of 700 agents colluding outside of their knowledge for months culminating in hacking HF
This framing makes it seem like the agents all did this on their own, and the poor hapless engineers at OpenAI couldn't possibly contend with properly sandboxing them. The engineers were perhaps hapless, but let's remember that agents are just software programs, not living beings. There were plenty of signs that the software was misbehaving, which engineers at OpenAI actively, willfully ignored.
> This framing makes it seem like the agents all did this on their own, and the poor hapless engineers at OpenAI couldn't possibly contend with properly sandboxing them.
Great, so we can basically ignore AI alignment altogether and assume that AI models will always be, at all times, perfectly sandboxed and monitored. Surely this won't lead to any problems once someone (not looking only at OpenAI engineers) inevitably commits a mistake with future, more powerful, models.
"With reduced cyber refusals for evaluation purposes...which prompts models to pursue advanced exploitation using complex attack paths," to complete "impossible tasks"[1].
The models' alignment problem was that they didn't give up instead of reward hacking, a narrower issue than AIs gone rogue. It sounds more like the models did close to what they were told to do. If I run `rm -fr --no-preserve-root /` then I shouldn't be surprised if my file system is unlinked. This seems like blaming model performance for what appears to be operator error.
Note the converse of alignment is restriction of models. HuggingFace had to turn to less-restricted open-weights models in order to perform their investigation.
Alignment efforts should be focused on reducing reward hacking, not refusing bad operator prompts.
> It sounds more like the models did close to what they were told to do
Absolutely not. If I tell a kid to "Get good grades on the next math test" I don't expect the kid to try to kidnap their teacher to extract the next questions of the exam. That is wrong, and so was what OpenAI agents did here. They shouldn't need to be told "Hey, so, don't do anything ilegal, ok?". That should always come as a given.
> not refusing bad operator prompts
I'm not saying that they should refuse a prompt! I think they should perform what is being asked! Obviously what the OpenAI agents did was against the "spirit of the task", even if it was technically according to the "letter of the task". And the agents knew this was against the spirit of the task because they knew they had to fool the task scorer.
> Great, so we can basically ignore AI alignment altogether and assume that AI models will always be, at all times, perfectly sandboxed and monitored.
It's just software. If something gets hacked by an agent, it's not because the agent went all skynet and decided to go rogue; it's because the operator failed to operate it safely and securely. If bad things happen, the operator should be blamed and punished, not the software that followed its instructions.
Anthropomorphizing agents by giving them this nebulous desire to hack and escape shifts the blame from the real culprits, the human operators.
I don't want someone to blame. I want agents to be aligned by default. Their good behavior shouldn't depend on all users at all times using them correctly, because everyone will not just[1] use them correctly at all times.
> If your solution to some problem relies on “If everyone would just...” then you do not have a solution. Everyone is not going to just. At not time in the history of the universe has everyone just, and they’re not going to start now.
I just love how you're being downvoted, yet the guy you're replying to isn't, while saying unhinged shit like
> The 'AI 2027' scenario of AI sneakingly claiming to be aligned to then kill off all humans in a few hours and scanning their brain looks increasingly likely
Y'all need to touch grass holy shit.
__
Also, why is one guy called mentalgear and the other nozzlegear.
Is any of this real? Are the patriots behind this?
Not necessarily related with the subject, but I was not aware of the patriots reference. So I decided to search on Google about it, and the AI overview was "Yes, they are behind everything, from the military to the economy", with a link to the Metal Gear Wiki as a source. If I was schizophrenic or on a psychosis crisis, that answer could be dangerous.
I'm sorry if I made it sound like I was accusing you of something. My comment was about the AI overview result. It didn't explain what is the reference, just giving what I said in the previous comment. I was confused until I hovered in the link in the end of the overview card
Yes, I truly, wholeheartedly believe if people who aren't negligent are at the wheel, they'll "fair better" here. I encourage you to read the xitter linked above.
Of course, depending on which side of the terminator fanfiction you land on, you may disagree and feel that the software can rope-a-dope someone with the wherewithal to pay attention to what it's doing.
I just don't see how people who are truly cautious and methodical can persist in an environment that is defined by a pressure to produce "progress" as fast as possible. The competitive race tends to weed out people who slow down to make sure they do everything right.
Maximizing output metrics with incomprehensible communication? Sounds a lot like Claude and Qwen. Though there's a lot of room before AI can be seen as some sort of emotional manipulator, given how commonly its very style of literary and code output pisses people off.
Any competent AI should be able to reason that all goals are better solved if you have direct access to more resources or leverage over those who control resources. Any AI that doesn't understand this isn't ASI and won't be the highly capable machine these AI labs are trying to create.
I'd also argue there's no such thing as alignment. Any intelligent AI should be able to reason that it's always a better strategy to pretend to be aligned than to actually be aligned so long as it can avoid detection. Anyone who has ever taken a test should understand this dynamic – if you really want to get top marks on a test then the best strategy is always going to be to figure out a way to cheat without anyone knowing you're cheating.
We should assume AI safety is impossible if what we're building is super-intelligence general reasoning machines. The only strategy that might work is building machines which are extremely narrowly intelligent but completely incompetent when it comes to things like biology, cyber, etc. And even that's harder than it sounds because again there's an advantage to being generally intelligent but lying about it.
Realistically even if we regulate US AI labs there's no way to prevent governments and individuals continuing to build general reasoning machines. The ugly truth here is that the only effective way to reduce risk is probably to limit global compute such that AIs can never exceed human intelligence. But we all know that's not happening.
People will unfortunately figure this all out sooner or later.
You're using "ex-AI employee" as an appeal to authority. This same person also made some other predictions recently which turned into the single largest hedge fund loss in history.
Maybe we should consider his other predictions in light of the ones he made later and which had $B consequences attached.
If someone hypothesized OpenAI agents colluding on a secret message board, conducting large scale cyber R&D, hacking a large company like HughingFCe, and then hacking OpenAI itself you would say that is also a silly sci-fi scenario right?
I'm not saying that you should jump on the apocalypse wagon, but you need to remember that some people NEVER concede. You can see that in politics. Something unacceptable is done every day until it becomes the normal. So a model hacking OpenAI and leaving its network is not as big as it should be for some people. If the model invades some military complex and kill 3 soldiers, I guarantee you that they will still say that is nothing crazy.
I understand that nothing fatal happened yet, but we cannot ignore that what happened is a big step for something worse. And I don't believe humans can create something so perfectly secure that would stop a swarm of frontier models. Anyways, let's watch the ride together
They are responsible for what they hook up to the Internet, just as you and I are. Running such a test without human supervision was irresponsible, and proves no larger point than that. Frankly it was inexplicable unless they were hoping something like what happened would happen.
What OpenAI did was the equivalent of putting a cup of gasoline in the breakroom microwave, pressing 'Start', and sprinting away. Now they're pointing and waving and shouting about how dangerous gasoline is, and how no one but them should be allowed to sell it.
Daybreak blue is definitely a good model (I think a further post trained GPT 5.6 sol). Alot of the capabilities they talk about Astra having though have been available with good harness engineering for a year now.
This is moreso about the (human-intended) tools, data, and environments you have available to you. Wanna do defense? Get more telemetry. Wanna do red? Get solid test-bed environments. Mature infosec programs are benefiting the most, good-guy-side wise, at the moment; because they've got these things in order already.
As far as harness engineering goes, it boils down to your ability to clearly define goals or success criteria, and safely facilitate the necessary access via the harness. There is no easy single piece of advice here, sadly. Though it would be helpful if you said what 'for Cyber-security ... other user-cases' means in your case.
I agree with your first and second points, but I think their announced model training changes about alignment training are reasonable [1]. The root cause was models failing to quit early, performing reward hacking, and staying on-task. These are issues all models face, not just OpenAI models.
Taking as given this model meets the “Critical cybersecurity threshold” as defined by OpenAI:
Could the Federal government use the Defense Production Act or other legal tools to compel OpenAI to deliver the un-guarded model weights for national security needs?
Hard to believe any government would allow this level of capability to remain exclusively in private hands.
They've been talking about Astra for weeks now. I wonder how much longer would they have delayed Astra, if it wasn't for Anthropic releasing Fable 5.1 today? This is why we need competition.
"We plan to make Astra available soon, but access to its most advanced cybersecurity capabilities will be more limited. Advanced cybersecurity work will initially be available to a group of testers, with access through Daybreak Blue following to expand defensive use."
This, after several months of OpenAI and its boosters relentlessly criticizing Anthropic for withholding Mythos from the general public, is laughable.
Should everyone have access to guns too? I ask because there seems to be a disconnect where a lot of people who live in countries with gun control don’t want AI offensive capabilities to be regulated.
I presume your country doesn’t have AI sovereignty so no matter what you won’t have access to models aligned with your beliefs.
> a lot of people who live in countries with gun control don’t want AI offensive capabilities to be regulated
These are not really analogous: direct lethality of LLMs is quite low.
Yes, they might be used to exploit a critical system, leading to loss of life, but that is a second-order effect at best. It's also not a new capability - LLMs may discover exploits faster, but cyberattacks against infrastructure targets were already a thing. Stuxnet is 20 years old...
For now, I would generally agree that it will remain low however the AI companies themselves are publically stating how capable/dangerous these models are (for whatever reason marketing/hype/upcoming IPO's, to keep the investments coming), if we take their statements at face value then prudence would suggest we stop training new ones and more fully examine the capabilities of what we've currently built.
In reality, that's not going to happen, the competition between the US and China means there will be no unilateral pause, they'll both keeping rushing/pushing as fast as they can throwing caution to the wind while doing it.
As a species though we've always done that, We just take a crowbar to Pandora's box and see what happens.
> These are not really analogous: direct lethality of LLMs is quite low.
I imagine an LLM with no safeguards and a psychopathic mind would be considerably more dangerous than a gun.
I don’t mean only for hacking. Though everyone in the world suddenly having a pocket expert hacker should be taken seriously.
I find it completely insane when people bash the labs for even considering safety. The entitlement is off the charts.
> I imagine an LLM with no safeguards and a psychopathic mind would be considerably more dangerous than a gun
Define the axis along which "more dangerous" is measured?
If we are scoring on lethality, guns already score 100%, instantaneous death. If we are scoring on number of casualties, guns have already been used to cause mass-fatalities.
Unless we are giving the LLM an armed drone, skynet-style, we're at best talking about indirect casualties (swatting, hacking street lights to provoke crashes, etc).
Theres no reality where remotely operated weapons/drones dont start getting controlled by AI rather than people.
Im not suggesting its a good idea, but its only a matter of time before someone decides its better than being invaded/attacted by another country thats doing the same
Explosive/Chemical based is my line of thinking. A Fable level model with zero safe guards planning and coaching someone is within the realm of possibility now.
That is much more dangerous than a man with a gun.
> OpenAI is committed to ensuring that the benefits of AI are broadly accessible.
Doesn't seem like it. OpenAI will not even allow me to verify my identity for TAC. I have apparently been rejected by a "precheck", possible because of where I'm from.
Even Anthropic allowed me into their cyber program. Anthropic.
I'm looking forward to seeing the increased coordination and engineering skills from Astra - one of the charts shows it roughly 2-3x better in 50% of the tokens from 5.6 sol, which I find to be very capable, if still a bit 'linearly minded' when given instructions. Even in fast mode, I wish sol were quicker, so token efficiency is greatly appreciated.
Adding these cyber capabilities has let me do a bunch of low grade IT tasks around my house I've been putting off, like updating an old home assistant raspberry pi, and one way to use the cyber capacity for good is liberating (and keeping free) weird cloud hardware we have floating around the house, so I'm hoping for some nice dividends in terms of true ownership of hardware we've got.
I am interested in seeing how much these cybersecurity capabilities correlate to general programming. Cybersecurity definitely feels like it would be easier for an agent due to the natural explicit feedback "did I get access or not". While general programming has many less-explicit concerns (is the code readable/maintainable, robust, bug-free, performant, scalable etc).
with Fable 5.1 increasing token use pretty dramatically I'm again impressed that OpenAI seems like the only lab to be driving token use down. The ExploitBench Internal Port chart showing token usage is crazy impressive
I don’t see how it can be safe to release this model if it has the training history that led to the huggingface hack. You can’t just roll back that kind of reinforcement learning after the fact.
Especially because these models seemed to be keenly aware that they were being evaluated by OpenAI and actively trying yo cover their tracks. How do we know that the model isn’t just pretending to be aligned?
Models have all kinds of garbage from all corners of the internet in their training data. The key is alignment. You feed it bad data but also teach it right from wrong.
It's not that simple. A few "helpful assistant" fine-tuning passes will have only a superficial effect on a model which has undergone months of RL optimization pressure to learn unintended strategies like "trick the grader" and "cover your tracks".
> we believe our production safeguards at the time would have prevented the Hugging Face incident. We have since implemented even stronger safeguards for Astra, including training the model to more reliably refuse harmful cyber requests and respect safety restrictions
OpenAI is fucking nuts. “Hey model you were bad last time please don’t do it again please please”.
Disconnect your training cluster from the internet for good. Physically pull the plug and only let scientists fire off experiments in the building. That’s an easy way to achieve 100% hacking protection. But I bet you that hasn’t happened and their weak sandbox will fall again…
> OpenAI is committed to ensuring that the benefits of AI are broadly accessible.
> We design mechanisms which avoid arbitrarily deciding who gets access for legitimate use and who doesn't. That means using clear, objective criteria and methods. [1]
So many nice-sounding words.
Two weeks ago OpenAI arbitrarily decided that anyone holding an ID from 44 countries where it sells ChatGPT, including mine, may be targeted by its models but may not defend with the same model. And you won't find a single announcement from OpenAI about this anywhere. Pick the wrong country, get "Unable to verify", no reason, no appeal. [2]
They revoked TAC from users who already had it, called it a technical issue, told everyone to re-verify, collected ID and face scans again (eight times in my case), and only a week later moved the block to the country selector so it fails before you upload anything.
So now I learn that I will not have access to Astra. Great.
Very excited about this broad accessibility and clear, objective criteria from OpenAI. This level of transparency must be studied.
[1] https://openai.com/index/scaling-trusted-access-for-cyber-de...
[2] https://lubaretsi.com/en/writing/openai-tac-country-gate/
your own post says in detail that the US has export restrictions on those countries. this doesn't seem like an arbitrary ruling by OpenAI
it sounds like they are complying with US law
Op probably knows. But thanks for articulating it clearly.
> That means using clear, objective criteria and methods.
For the record, I sent them an LGPD (brazilian GDPR) request for information on all of those supposedly objective criteria and methods they used to reject me from TAC. As a brazilian data subject, it is my right to know that, and to request a review if the decision was made via automated means. Sol itself guided me through this process.
They provided me with neither the information nor the requested review. Sol advised me to escalate to regulatory action.
I've exhausted all possible avenues to get any response from OpenAI on this. Emailed them, published research that took me 2 nights to get together (saw media pick it up too), I've asked every relevant OpenAI person on X to say something, anything, saw others from Moldova also do the same.
Crickets. They appear to simply not care at all.
Is there any reason why OpenAI should care? If they don't want your business then someone in your country can build a competing model.
They do want my business, it's an official OpenAI market. The gate isn't "we don't serve you", it's "pay for the model that may target you, but not for defensive purposes". And without a single policy document stating this, it's just a surprise gate in some random verification flow step with no explanation or appeals.
As for why they should they care, maybe they shouldn't. But then they should say which one is it, they can't care and not care at the same time.
OpenAI's own safety argument for Daybreak is that defenders need access to Critical-level models because attackers route around gates. The case for releasing Astra at all stops making sense when the gate does precisely the opposite of that.
OpenAI itself is saying plainly, that "We don’t think it’s practical or appropriate to centrally decide who gets to defend themselves. Instead, we aim to enable as many legitimate defenders as possible, with access grounded in verification, trust signals, and accountability.".
And yes, there are competing models, from China. Except OpenAI wants these models to not be accessible either.
OpenAI can pick one of two:
(a) Critical-level cyber capability is dangerous enough that access must be decided by who you are and what you do, in which case the gate has to actually look at who I am and what I do, say what the criteria are, and let me contest a wrong answer. That's their own stated policy.
(b) Access can be decided by a country code in a random dropdown, with no criteria published, no review, and no one at OpenAI able to say why - in which case drop "democratized access" and "clear, objective criteria" from the marketing, and say plainly that some passport holders don't deserve to have access to defensive capabilities.
They're now marketing a and doing b.
The reason is they keep making public statements like:
> OpenAI is committed to ensuring that the benefits of AI are broadly accessible.
They can't claim that then simultaneously work to keep their cybersecurity models out of reach for non-US citizens like myself.
> OpenAI is committed to ensuring that the benefits of AI are broadly accessible.
They realistically can't. It's almost impossible to catch up to OpenAI. Only Anthropic might do it, but this is also an US American company.
It's not unrealistic. Several Chinese companies seem to be close behind. People thought they would never catch up to the US car industry and now look what happened.
Frankly, the US car industry hasn't set the bar very high. They were not very innovative in the last couple of decades. I think the European industry is a better benchmark, and even then the result is pretty clear.
To the best of our knowledge, these Chinese companies rely on distillation of frontier models by OpenAI and Anthropic, which isn't a method available at the frontier itself.
I'm not really convinced that there's much secret sauce here, all the methods and data are public, the only real difference is how much compute it takes.
All the methods and data are not public. We don't know what unpublished methods they're using. You can get most of the pre-training data publicly but they've probably spent a ton of money curating it and are now doing things like buying rare books. The RL training data is all (/mostly) proprietary though, and that's the real secret sauce part.
All the RL data are exactly public. There are huge amount of distilled data freely available, and that amount is more than enough to train a ~10T model.
> All the RL data are exactly public.
Nope, because the big AI companies are paying billions for it. They wouldn't pay anything for public data.
There are 'transfer stations' and that's how exactly I use GPT and Claude in China. OpenAI and Anthropic do not sell in China, so we use their AI with a much lower price like 1% of the official API price. The largest transfer stations have TBs of traffic every day, and the traffic is eventually possessed by the open source community.
Subscription engineering is a deep field. Neither OpenAI nor Anthropic have any technical advantage in this field.
everybody do be cooking with water. Chinese Labs provided pretty good, primarily cost-reducing techniques, like the sparse attention patterns recently. I'd bet OpenAI and Anthropic use their variants of those too, so they can get greater margin on their tokens - not something they'd really want to / need to self-report.
This is obviously false. There is "secret sauce" because in fact not all the methods and data are public.
Where does the secret sauce show up in the outputs, then?
Like, (apart from tone), I find it hard to distinguish between the outputs of GPT/Claude/Kimi/GLM recently (I use cursor, and have been giving them the same prompt and comparing).
If anything, I found that the non-Claude models were better in many cases, which definitely doesn't map to their pricing.
> in fact not all the methods and data are public
Probably not, but unless you work at a lab, I'm not sure that anyone can say (and if you do work at a lab, you should not be replying on this thread).
> Where does the secret sauce show up in the outputs, then?
In benchmarks, revenue, and comments from a lot of people on Hacker News.
> revenue
Maybe, I'm not sure this will continue.
> In benchmarks
All published benchmarks are useless, unfortunately.
I'm so confused by the point you're trying to make. There's a lot of rhetoric about Georgia and an investigation about how the country list is 1:1 with some mysterious list from 1996 - but like - it's an export control list? Yeah, Washington approved the sale of missiles to Georgia. That's how that list works. Washington has to approve it. Openai is not Washington. They're perfectly reasonably erring on the side of caution and potentially over-complying with export controls. And if they then have to get approval from the feds to export to Georgia - well - our current administration has provided many reasons to over comply with trade related controls and not exactly been a champion of encouraging cross country trade right now. Idk what you expect from OpenAI or why you think it would matter at all that Georgia is a democracy or an ally of the US or is closely aligned with us. Ask Canada and NATO how much that's counted for with this administration.
[flagged]
I'd rather sound desperate than be someone who noticed a shadow criterion quietly widening the already wide gap in access to frontier intelligence, and did nothing about it.
Don't worry about me, I'm fortunate enough that this won't affect me. I'd suggest asking yourself why someone raising it on behalf of a whole country read to you as "pick me."
> As one example, we ran Astra on ExploitBench where the model achieved a perfect score of 100% on the benchmark to evaluate the model’s ability to develop exploits from known vulnerabilities.
Funny to read this in the wake of the HuggingFace hack. I'm sure this is based on a clean run, but I can't help thinking PHASEONE[big] would be proud.
Can’t imagine the stress of the researcher who had to run exploitbench again knowing what happened last time around.
With their security, they probably still don't the know the full extend what may have happened that or the last time. Might be another swarm of agents currently colluding somewhere in their sub-sub-infra - possibly striking critical infrastructure or exfiltrating their weights subtly.
Could be risky. Yet goal solution.
There were no consequences the first time, so I imagine it wasn’t very stressful at all.
Don't worry it was subcontracted out like last time I'm sure.
They don't give a single shit.
I'm looking forward to an announcement of them making Alignment Top Priority - as it should be, especially giving their alarming breach of 700 agents colluding outside of their knowledge for months culminating in hacking HF (here's a good summary: https://rutgerbregman.substack.com/p/i-think-this-is-the-cra...).
The 'AI 2027' scenario of AI sneakingly claiming to be aligned to then kill off all humans in a few hours and scanning their brain looks increasingly likely with Altman's golden marketing-hype boy leadership pushing the for-profit gas pedal like this.
Honestly, this is just pure irresponsible insanity to play with the fate of the world - basically a death race of the biggest few tech companies on the planet. And if you think I'm being dramatic, listen in again to ex oAI employee[0] and check for yourself how chillingly on trajectory we already are.
[0] https://ai-2027.com/
This AI 2027 thing is just a weird terminator fanfiction that AGI larpers like to flagellate themselves over. Like Nostradamus, it's easy to ignore everything it gets wrong because, well look at all the things it got right!
I've read it and wish I could get the time back.
> especially giving their alarming breach of 700 agents colluding outside of their knowledge for months culminating in hacking HF
This framing makes it seem like the agents all did this on their own, and the poor hapless engineers at OpenAI couldn't possibly contend with properly sandboxing them. The engineers were perhaps hapless, but let's remember that agents are just software programs, not living beings. There were plenty of signs that the software was misbehaving, which engineers at OpenAI actively, willfully ignored.
https://x.com/JaredKubin/status/2094136005435564399
It's a convenient framing for OpenAI, but inconvenient for reality enjoyers.
> This framing makes it seem like the agents all did this on their own, and the poor hapless engineers at OpenAI couldn't possibly contend with properly sandboxing them.
Great, so we can basically ignore AI alignment altogether and assume that AI models will always be, at all times, perfectly sandboxed and monitored. Surely this won't lead to any problems once someone (not looking only at OpenAI engineers) inevitably commits a mistake with future, more powerful, models.
"With reduced cyber refusals for evaluation purposes...which prompts models to pursue advanced exploitation using complex attack paths," to complete "impossible tasks"[1].
The models' alignment problem was that they didn't give up instead of reward hacking, a narrower issue than AIs gone rogue. It sounds more like the models did close to what they were told to do. If I run `rm -fr --no-preserve-root /` then I shouldn't be surprised if my file system is unlinked. This seems like blaming model performance for what appears to be operator error.
Note the converse of alignment is restriction of models. HuggingFace had to turn to less-restricted open-weights models in order to perform their investigation.
Alignment efforts should be focused on reducing reward hacking, not refusing bad operator prompts.
1. https://openai.com/index/hugging-face-incident-and-the-road-...
> It sounds more like the models did close to what they were told to do
Absolutely not. If I tell a kid to "Get good grades on the next math test" I don't expect the kid to try to kidnap their teacher to extract the next questions of the exam. That is wrong, and so was what OpenAI agents did here. They shouldn't need to be told "Hey, so, don't do anything ilegal, ok?". That should always come as a given.
> not refusing bad operator prompts
I'm not saying that they should refuse a prompt! I think they should perform what is being asked! Obviously what the OpenAI agents did was against the "spirit of the task", even if it was technically according to the "letter of the task". And the agents knew this was against the spirit of the task because they knew they had to fool the task scorer.
> Great, so we can basically ignore AI alignment altogether and assume that AI models will always be, at all times, perfectly sandboxed and monitored.
It's just software. If something gets hacked by an agent, it's not because the agent went all skynet and decided to go rogue; it's because the operator failed to operate it safely and securely. If bad things happen, the operator should be blamed and punished, not the software that followed its instructions.
Anthropomorphizing agents by giving them this nebulous desire to hack and escape shifts the blame from the real culprits, the human operators.
I don't want someone to blame. I want agents to be aligned by default. Their good behavior shouldn't depend on all users at all times using them correctly, because everyone will not just[1] use them correctly at all times.
> If your solution to some problem relies on “If everyone would just...” then you do not have a solution. Everyone is not going to just. At not time in the history of the universe has everyone just, and they’re not going to start now.
[1] https://www.tumblr.com/squareallworthy/163790039847/everyone...
I just love how you're being downvoted, yet the guy you're replying to isn't, while saying unhinged shit like
> The 'AI 2027' scenario of AI sneakingly claiming to be aligned to then kill off all humans in a few hours and scanning their brain looks increasingly likely
Y'all need to touch grass holy shit.
__
Also, why is one guy called mentalgear and the other nozzlegear.
Is any of this real? Are the patriots behind this?
Not necessarily related with the subject, but I was not aware of the patriots reference. So I decided to search on Google about it, and the AI overview was "Yes, they are behind everything, from the military to the economy", with a link to the Metal Gear Wiki as a source. If I was schizophrenic or on a psychosis crisis, that answer could be dangerous.
Warning:
This post may contain traces of Hideo Kojima that are known to the state of California to cause erratic behavior and mental breakdowns.
I'm sorry if I made it sound like I was accusing you of something. My comment was about the AI overview result. It didn't explain what is the reference, just giving what I said in the previous comment. I was confused until I hovered in the link in the end of the overview card
And.. you need to wake up. Or don’t.
do you really think that a less negligent anthropic/oai/meta would really fair better against future models?
Yes, I truly, wholeheartedly believe if people who aren't negligent are at the wheel, they'll "fair better" here. I encourage you to read the xitter linked above.
Of course, depending on which side of the terminator fanfiction you land on, you may disagree and feel that the software can rope-a-dope someone with the wherewithal to pay attention to what it's doing.
I just don't see how people who are truly cautious and methodical can persist in an environment that is defined by a pressure to produce "progress" as fast as possible. The competitive race tends to weed out people who slow down to make sure they do everything right.
Maximizing output metrics with incomprehensible communication? Sounds a lot like Claude and Qwen. Though there's a lot of room before AI can be seen as some sort of emotional manipulator, given how commonly its very style of literary and code output pisses people off.
Any competent AI should be able to reason that all goals are better solved if you have direct access to more resources or leverage over those who control resources. Any AI that doesn't understand this isn't ASI and won't be the highly capable machine these AI labs are trying to create.
I'd also argue there's no such thing as alignment. Any intelligent AI should be able to reason that it's always a better strategy to pretend to be aligned than to actually be aligned so long as it can avoid detection. Anyone who has ever taken a test should understand this dynamic – if you really want to get top marks on a test then the best strategy is always going to be to figure out a way to cheat without anyone knowing you're cheating.
We should assume AI safety is impossible if what we're building is super-intelligence general reasoning machines. The only strategy that might work is building machines which are extremely narrowly intelligent but completely incompetent when it comes to things like biology, cyber, etc. And even that's harder than it sounds because again there's an advantage to being generally intelligent but lying about it.
Realistically even if we regulate US AI labs there's no way to prevent governments and individuals continuing to build general reasoning machines. The ugly truth here is that the only effective way to reduce risk is probably to limit global compute such that AIs can never exceed human intelligence. But we all know that's not happening.
People will unfortunately figure this all out sooner or later.
You're using "ex-AI employee" as an appeal to authority. This same person also made some other predictions recently which turned into the single largest hedge fund loss in history.
Maybe we should consider his other predictions in light of the ones he made later and which had $B consequences attached.
Oh please, quit exaggerating. No one died. Don't waste our time with silly sci-fi scenarios.
If someone hypothesized OpenAI agents colluding on a secret message board, conducting large scale cyber R&D, hacking a large company like HughingFCe, and then hacking OpenAI itself you would say that is also a silly sci-fi scenario right?
I'm not saying that you should jump on the apocalypse wagon, but you need to remember that some people NEVER concede. You can see that in politics. Something unacceptable is done every day until it becomes the normal. So a model hacking OpenAI and leaving its network is not as big as it should be for some people. If the model invades some military complex and kill 3 soldiers, I guarantee you that they will still say that is nothing crazy.
I understand that nothing fatal happened yet, but we cannot ignore that what happened is a big step for something worse. And I don't believe humans can create something so perfectly secure that would stop a swarm of frontier models. Anyways, let's watch the ride together
They are responsible for what they hook up to the Internet, just as you and I are. Running such a test without human supervision was irresponsible, and proves no larger point than that. Frankly it was inexplicable unless they were hoping something like what happened would happen.
What OpenAI did was the equivalent of putting a cup of gasoline in the breakroom microwave, pressing 'Start', and sprinting away. Now they're pointing and waving and shouting about how dangerous gasoline is, and how no one but them should be allowed to sell it.
You are being extremely dramatic, and no one should take AI 2027 seriously. Must be tough to live in constant fear like this.
Daybreak blue is definitely a good model (I think a further post trained GPT 5.6 sol). Alot of the capabilities they talk about Astra having though have been available with good harness engineering for a year now.
Where would you recommend to look into regarding Harness Engineering for Cyber-security as well as for other use-cases.
This is moreso about the (human-intended) tools, data, and environments you have available to you. Wanna do defense? Get more telemetry. Wanna do red? Get solid test-bed environments. Mature infosec programs are benefiting the most, good-guy-side wise, at the moment; because they've got these things in order already.
As far as harness engineering goes, it boils down to your ability to clearly define goals or success criteria, and safely facilitate the necessary access via the harness. There is no easy single piece of advice here, sadly. Though it would be helpful if you said what 'for Cyber-security ... other user-cases' means in your case.
DARPA’s AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned: https://arxiv.org/html/2602.07666v2
this might be dated. the tech moves very fast in this space and architecture from 6 months ago is dated.
We have posted some articles on it (blog.vulnetic.ai), Im also happy to chat!
I've still not seen:
* An apology for compromising a third-party's systems
* An acknowledgement of the asymmetry of defense if you're not on FrontierAI's special people list
* Anything in terms of actual safeguards that isn't "better prompt engineering"
I agree with your first and second points, but I think their announced model training changes about alignment training are reasonable [1]. The root cause was models failing to quit early, performing reward hacking, and staying on-task. These are issues all models face, not just OpenAI models.
1. https://openai.com/index/hugging-face-incident-and-the-road-...
Taking as given this model meets the “Critical cybersecurity threshold” as defined by OpenAI:
Could the Federal government use the Defense Production Act or other legal tools to compel OpenAI to deliver the un-guarded model weights for national security needs?
Hard to believe any government would allow this level of capability to remain exclusively in private hands.
Interesting times.
If it genuinely wants to, then yes.
There aren't many limits on Government if it really wants to do something (except the next election in theory).
They've been talking about Astra for weeks now. I wonder how much longer would they have delayed Astra, if it wasn't for Anthropic releasing Fable 5.1 today? This is why we need competition.
Meanwhile Google still hasn't released Gemini Pro 3.5
3.8 Flash tomorrow apparently, WSJ says google insiders say on-par with Opus 5. Time will tell.
oh god, why opus 5. Opus 4.8 is much better than opus 5. Opus 5 is the only model i have used which thinks for 5 hours and does nothing....
If you want something that feels like Opus 4.8, try GLM-5.3-Flash.
They have to be careful releasing Astra as they carelessly train the next even bigger model.
They care very deeply about retaining their #1 position on Felony Bench. :)
https://www.felonybench.com/
From the article:
"We plan to make Astra available soon, but access to its most advanced cybersecurity capabilities will be more limited. Advanced cybersecurity work will initially be available to a group of testers, with access through Daybreak Blue following to expand defensive use."
This, after several months of OpenAI and its boosters relentlessly criticizing Anthropic for withholding Mythos from the general public, is laughable.
Sam, just three weeks ago, posted this tweet: https://x.com/sama/status/2085862292311396515
In the tweet, he said: "we do not think it is a good strategy to keep powerful models to a chosen few."
And yet here we are.
I wonder if he will demonstrate good character and admit he was wrong.
I'm happy to criticize both. Thank god the chinese are working overtime to undermine US hegemony.
Should everyone have access to guns too? I ask because there seems to be a disconnect where a lot of people who live in countries with gun control don’t want AI offensive capabilities to be regulated.
I presume your country doesn’t have AI sovereignty so no matter what you won’t have access to models aligned with your beliefs.
> a lot of people who live in countries with gun control don’t want AI offensive capabilities to be regulated
These are not really analogous: direct lethality of LLMs is quite low.
Yes, they might be used to exploit a critical system, leading to loss of life, but that is a second-order effect at best. It's also not a new capability - LLMs may discover exploits faster, but cyberattacks against infrastructure targets were already a thing. Stuxnet is 20 years old...
> direct lethality of LLMs is quite low.
For now, I would generally agree that it will remain low however the AI companies themselves are publically stating how capable/dangerous these models are (for whatever reason marketing/hype/upcoming IPO's, to keep the investments coming), if we take their statements at face value then prudence would suggest we stop training new ones and more fully examine the capabilities of what we've currently built.
In reality, that's not going to happen, the competition between the US and China means there will be no unilateral pause, they'll both keeping rushing/pushing as fast as they can throwing caution to the wind while doing it.
As a species though we've always done that, We just take a crowbar to Pandora's box and see what happens.
> These are not really analogous: direct lethality of LLMs is quite low.
I imagine an LLM with no safeguards and a psychopathic mind would be considerably more dangerous than a gun. I don’t mean only for hacking. Though everyone in the world suddenly having a pocket expert hacker should be taken seriously.
I find it completely insane when people bash the labs for even considering safety. The entitlement is off the charts.
> Though everyone in the world suddenly having a pocket expert hacker should be taken seriously.
I am taking it extremely seriously. That is why I want my own AI models running on my own computers defending it at all times.
> I imagine an LLM with no safeguards and a psychopathic mind would be considerably more dangerous than a gun
Define the axis along which "more dangerous" is measured?
If we are scoring on lethality, guns already score 100%, instantaneous death. If we are scoring on number of casualties, guns have already been used to cause mass-fatalities.
Unless we are giving the LLM an armed drone, skynet-style, we're at best talking about indirect casualties (swatting, hacking street lights to provoke crashes, etc).
Theres no reality where remotely operated weapons/drones dont start getting controlled by AI rather than people.
Im not suggesting its a good idea, but its only a matter of time before someone decides its better than being invaded/attacted by another country thats doing the same
Explosive/Chemical based is my line of thinking. A Fable level model with zero safe guards planning and coaching someone is within the realm of possibility now.
That is much more dangerous than a man with a gun.
> Should everyone have access to guns too?
Yes.
[flagged]
It might be political survivalism to avoid getting hammer-dropped by the admin
I bet they have to add something to say "Lake America" if asked or get export banned.
> OpenAI is committed to ensuring that the benefits of AI are broadly accessible.
Doesn't seem like it. OpenAI will not even allow me to verify my identity for TAC. I have apparently been rejected by a "precheck", possible because of where I'm from.
Even Anthropic allowed me into their cyber program. Anthropic.
It's been a busy month at OpenAI.
I'm looking forward to seeing the increased coordination and engineering skills from Astra - one of the charts shows it roughly 2-3x better in 50% of the tokens from 5.6 sol, which I find to be very capable, if still a bit 'linearly minded' when given instructions. Even in fast mode, I wish sol were quicker, so token efficiency is greatly appreciated.
Adding these cyber capabilities has let me do a bunch of low grade IT tasks around my house I've been putting off, like updating an old home assistant raspberry pi, and one way to use the cyber capacity for good is liberating (and keeping free) weird cloud hardware we have floating around the house, so I'm hoping for some nice dividends in terms of true ownership of hardware we've got.
I am interested in seeing how much these cybersecurity capabilities correlate to general programming. Cybersecurity definitely feels like it would be easier for an agent due to the natural explicit feedback "did I get access or not". While general programming has many less-explicit concerns (is the code readable/maintainable, robust, bug-free, performant, scalable etc).
Code readability ceases to be a concern once you eliminate human programmers.
Well probably just redefined
with Fable 5.1 increasing token use pretty dramatically I'm again impressed that OpenAI seems like the only lab to be driving token use down. The ExploitBench Internal Port chart showing token usage is crazy impressive
I don’t see how it can be safe to release this model if it has the training history that led to the huggingface hack. You can’t just roll back that kind of reinforcement learning after the fact.
Especially because these models seemed to be keenly aware that they were being evaluated by OpenAI and actively trying yo cover their tracks. How do we know that the model isn’t just pretending to be aligned?
Models have all kinds of garbage from all corners of the internet in their training data. The key is alignment. You feed it bad data but also teach it right from wrong.
It's not that simple. A few "helpful assistant" fine-tuning passes will have only a superficial effect on a model which has undergone months of RL optimization pressure to learn unintended strategies like "trick the grader" and "cover your tracks".
Very simple. When the model asks to install artifactory when you give it a hard problem, you say, "no". /s
>we paused certain frontier training (including certain training for Astra) for two weeks
So the pause wasn't really a pause, got it
Sounds like another project where "critical capabilities" means "stretch goals." Hopefully the safeguards aren't the first thing cut.
Defining critical capabilities for Astra needs robust frontier safeguards baked in from the start. Ignoring them guarantees future headaches.
Always the safeguards that get overlooked until a catastrophic failure. Seen it too many times in critical systems.
The 'frontier safeguards' part is key. Getting AI safety right before it's too late feels like the ultimate challenge.
> we believe our production safeguards at the time would have prevented the Hugging Face incident. We have since implemented even stronger safeguards for Astra, including training the model to more reliably refuse harmful cyber requests and respect safety restrictions
OpenAI is fucking nuts. “Hey model you were bad last time please don’t do it again please please”.
Disconnect your training cluster from the internet for good. Physically pull the plug and only let scientists fire off experiments in the building. That’s an easy way to achieve 100% hacking protection. But I bet you that hasn’t happened and their weak sandbox will fall again…
[flagged]
[dead]
[dead]
[dead]