Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly.
If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here.
> Have experience shipping production full-stack products across modern web frontends and backend services.
I'm amazed that the requirements are so low (or at least this vague) for jobs at companies like these.
Has anyone else had the experience of going to an interview and feeling like you were never asked any qualifying questions?
All the questions were easy, your answers were straightforward, you "got them right", but then were not chosen?
I find on the other side, they're also left with dozens of people who "passed" and then it comes down to a pretty arbitrary decision on who gets hired (if we are talking external, no referral, etc.)
I wonder if they can make job descriptions highly specific to filter the shortlist faster and more effectively (to actually get a shortlist).
I interviewed with them and had a great experience despite it ultimately not being a good fit due to communication issues and a bad flu coming on in the final interview - they, and Michael especially, was very kind and understanding.
Credit card entry, I think you should know a few basics like don't put it in MongoDB?? Or nah? It's just like any other user data?
How about a background check - can anyone take a user-entered DL and randomly Google stuff to see what they find?
Can I store your SSN in plain text in a text file? Why not?
The user had to upload their ID for IDV but I use Vercel. I guess I have to put it on S3. What should the bucket policy be for all these driver's license photos - there are so many???
If you're handling credit card numbers yourself, you're in a shrinking subset of developer roles. I've spent the last 7 years of my career working at payment processors, so I do handle that stuff, but the majority of my industry has built an infrastructure that makes it so most developers don't have to think about that.
To your other examples, not everyone on the team needs to know these things up front. Someone in the review process does, and eventually that knowledge gets disseminated and more people know it to carry it forward in their career.
Not having clear, objective criteria enables arbitrary decisions (not against OP, I mean in general.. and in general I dislike this pattern a lot). On the extreme other end of the spectrum would be 100% objective criteria, and companies being forced to pick a random applicant that matches them. If they want only the best, they have to have high expectations, but be able to actually define them. You say "cultural fit", I say "corruption", let's blow this whole joint.
> You say "cultural fit", I say "corruption", let's blow this whole joint.
It's not necessarily corruption. If there's no conflict between principal and agent, it's fine.
Just like when I sent my butler to go and buy a bottle of wine, he can make arbitrary choices, but that doesn't mean he's corrupt or going against my wishes. I trust his judgement, and since it's a repeated game our incentives are aligned.
It’s not corruption to simply hire the people you subjectively feel a preference for working for, instead of objective criteria. It’s not public tax funds that fuel salaries, it’s your own money. You get to spend it how you like.
Yes, many jurisdictions have outlawed arbitrary discrimination against protected classes (eg race), which is an entirely different matter, and not what we are discussing here.
If you didn't already know, jobs at highly competitive companies tend to have vague job requirements because they expect to be able to apply your raw intelligence to changing demands quickly. There's no point being hyper-specific about the exact software packages because that's not what they want. What they want is someone who, after talking to an interviewer for 30 minutes, leaves them with the thought "Wow, this person can do anything we need of them. They can probably tell us what we need too and take ownership of large projects. Hire!"
Agree. Being smart, competent, and high in conscientiousness is more important than any highly specific “qualification”. It’s not about checking a bunch of boxes. If you have a track record of getting shit done, you’ll have something to contribute.
> There's no point being hyper-specific about the exact software packages because that's not what they want
Okay. This makes it sound like they're more sophisticated but it seems more like they are less sophisticated, less specific, and a lot more vague in the job descriptions they themselves create.
If you look at any technical role, game dev or something where people are building important things at scale - there are a lot of specifics. Libraries, methodologies, where if you didn't know them you are nowhere near a fit.
I'm just wondering. It's OpenAI. Surely there is some domain-specific something beyond "has experience shipping front-end and back-end services" since that includes basically everyone.
I see where you're coming from, but as someone who regularly interviews engineers, I don't care about specific tech stacks when evaluating a candidate very much either. I can only think of two positions I've worked in where such a thing really mattered.
A good engineer can adapt and catch up without a lot of lead time. For a contractor, I'd be much more specific - but for someone who's going to join my team? I'm looking for a candidate that can demonstrate their problem solving ability, creative thinking and communication skills.
Other than having some kind of experience in the general domain we work in, those "soft skills" are far harder to find than specific tech experience.
Not every engineering job is entry level or as simple as most fullstack crud. Deeper into industry you find highly specific well defined positions for a given domain. Soft skills matter more the higher the ladder but id take a killer senior who can be difficult over a team of mediocre staff engineers.
I agree, I recently interviewed at Synthiolabs, and they asked me two questions: one about RAG and two about graph RAG. They rejected me even though I answered correctly, and the interviewer was also a college kid.
How does it deal with the current guardrails 5.6 Sol has on finding vulnerabilities? When I use it in the Codex app it would sometimes say it found a vulnerability, but it cannot tell me what it is.
Thanks! You've run into a real limitation: the CLI doesn't bypass the model's cybersecurity guardrails. If GPT-5.6 Sol finds a vulnerability but refuses to explain it, switching from the Codex app to the CLI won't automatically fix that.
For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals depending on the model and the account or organization where access is provisioned. It isn't a blanket bypass.
If you're an open-source maintainer, you can apply for conditional Codex Security access here:
If you have an example of "found a vulnerability but won't tell me what it is," I'd love to take a look too. You can send it to use with /feedback (or message me).
I tried it, it started a scan but stopped after hitting the rate-limit of my account. It gave up after just a minute of retrying (rate limits are tokens per minute, so... :P).
It said "Partial output was kept at <...>", but I dont see a obvious way of picking it up in a new scan? (The failed run cost me ~$13)
Yeah, you're right. A per-minute rate limit shouldn't kill a scan after a minute, and "partial output was kept" makes it sound like you can pick up where you left off. You can't yet, unfortunately. --max-cost can limit estimated spend, but we still need proper retries and resume. Sorry you spent $13 finding that out. Please send me an email and I'll help make it right.
The plugin, including when invoked through the Codex CLI, is great for scanning the repo you're currently working in. The standalone Security CLI/SDK uses the same scanner, but is built for running security across many repos over time: org-wide scans, historical results, deduplication, false-positive tracking, budget controls, and CI integration.
We've been talking to hundreds of engineering and security teams, and their feedback is shaping what we build.
Like Promptfoo, our goal is practical tooling that fits into the workflows teams already have.
We are actively working on officially supporting this. Because it's open source it is pretty easy to point a coding agent at it now and switch out the model.
Not yet, unfortunately. We only just opened the repo, and there isn't a public issue specifically tracking local or OpenAI-compatible endpoint support.
> co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
> Thanks for checking this out and for flagging the auth issues.
Offtopic, but this right here is why I don't believe any marketing around "great amazing models that one-shot everything and programmers are no longer needed".
You just have to look at what these labs routinely produce, and their own products.
Edit to respond to @simonw whose comment I saw before he retracted it ;)
This comment is tied directly to consistent continuous claims by the LLM labs. Their own products disprove their own claims, and it would indeed be nice if fewer people believed them :)
Just ran it on a small repo. It ran for almost an hour and then got interrupted. It drained half my weekly usage on a Pro plan.
npx codex-security scan .
[00:00] Preparing scan
[00:00] Authentication: stored Codex credentials.
[00:03] Preparing scan
[01:20] Running scan
[01:20] Preflight: worker delegation supported (up to 8 worker slots).
[52:47] Running scan
codex-security: Could not save the Codex Security scan: Repository HEAD changed while the scan was running. Start a new scan.
codex-security: Partial output was kept at ...
They just opened the weights. I expect competition from various providers will drop the price a bit. But you're right. I'd hope a model closer to GLM 5.2's price would be sufficiently useful.
A license and presumably revshare is required for large scale inference as a service of Kimi K3. There's clearly something going on right now with every router at the same or higher price as the Moonshot list price.
So I wouldn't count on competition if $/mil token is actually set by Moonshot, but I would expect $/tok to drop when there's a more competitive frontier open weight model.
Oof, that's a bad outcome. Half your weekly usage and a 50-minute scan just to get a HEAD error at the end is not acceptable. --max-cost can help limit estimated spend, but that doesn't fix the underlying problem or give you your quota back. We need to handle a changing checkout and partial results much better. Sorry you ran into this. Please send me an email.
Hey looks cool. I tried to run this on a small oss library and here's what happened:
$ codex-security scan .
[00:00] Preparing scan
[00:00] Authentication: stored Codex credentials.
[00:01] Preparing scan
[00:42] Running scan
[00:42] Preflight: worker delegation supported (up to 8 worker slots).
[41:03] Running scan
codex-security: This content was flagged for possible cybersecurity risk. If this seems wrong, try rephrasing your request. To get authorized for security work, join the Trusted Access for Cyber program: https://chatgpt.com/cyber
codex-security: Partial output was kept at /Users/ryan/.codex/state/plugins/codex-security/scans/framework/codex-security-framework-z7eNfr.
Just some feedback, but it ran for over 40 minutes and during that time I had no idea what was happening, thought it was frozen or in a bad state. Also, it ate through 25% of my weekly credits :(
I've been noticing that many new projects that would have been written in Python or Node a year ago are starting to be written in Go, Rust, etc.
Theory: people realized there’s little benefit to Python for agents. As Zep wrote, an “agent is a long-running, concurrent, I/O-bound process that spends most of its time waiting on a model, a tool, or a human[1]” — not a particular strength of Python.
I'm wondering if you'd considered Go (or others—Go’s just my fav ) before landing on Node, and more broadly whether you've noticed a similar pattern?
I suspect a lot of the Python and TypeScript code is getting prompted by folks who don't know any difference between dynamically typed and statically typed.
In the early days of LLM coding, it seemed to be much better at Python for whatever reason. I was never a big Python guy, but I got better results so I ran with it. That definitely doesn't seem to be the case anymore. Last week I asked Codex to mash up Super Mario Bros and Contra ROMs and it just did everything in straight assembly and absolutely crushed it. It couldn't do that 2 years ago. Python is just momentum and I think it's going to die down now that the models are much better.
I think it's because python is far more approachable/ubiquitous than go/rust. It's the entry level language for many people from all disciplines of life. Scientific community uses it, data science uses it.
Golang/rust however are very convenient to distribute. Small, portable, fast exe's are very nice. With agentic coding golang/rust are now accessible to a lot more people.
Yes, now that humans write less than 99% of code, the most important criteria for a language isn't readability, which I'd argue was always Python's main selling point, but the underlying runtime. There are practical limits to how fast a Python program can run either under I/O or CPU bound compared to other popular and mature languages with extensive libraries, like Elixir, Go or C++, depending on your use case.
Not really. I test the output thoroughly, I examine the thinking process, I go through the diff to see if anything jumps out but I my thinking process/the way I work has changed. Low level programming thinking has gotten atrophied it seems.
Not often I see companies referring to HN, thanks “ We quietly released the open-source Codex Security CLI, but Hacker News found it before we had a chance to share it here…”
security tools from AI companies feel like fire departments run by arsonists. useful, sure, but you can't help noticing who benefits from all the fires
Some of approaches there could be useful in other contexts. OAI has the compute to experiment with different prompts and I'd expect these to be somewhat optimized.
Yes, I think this is an under-appreciated part of the release. I hope people can adapt them to their own workflows. We run A LOT of evals as the Promptfoo team and we've spent billions of tokens fine-tuning them. You can expect more skills as we branch out to other security workflows and further improvements to the codex security prompts.
Update: As far as I understand, this was already available as a Codex plugin. The main news is that OpenAI has now open-sourced it, and development is still moving quickly.
I seem to have gotten a bunch of you are trying to stuff we don't allow errors.. very annoying.
Can they explain what types of projects it works on and how does it check I own it? Like will it just not work on Linux kernel even on my own patches to it?
Fair question, and I agree the refusals are frustrating.
The CLI doesn't do a repository-ownership check. Public projects are supported, and reviewing your own Linux kernel patches is the kind of defensive work we want to support.
The refusals come from model guardrails, which can be overly cautious. Trusted Access for Cyber (TAC1/Daybreak) is a separate, approved access path that can reduce those refusals.
Really glad to see this open-sourced. One thing I'd be interested in is how you think about the balance between false positives and false negatives. In practice, developers tend to stop trusting security tools if they generate too much noise, but missing a real issue is obviously costly too.
Looks great but the CLI output is not particularly interesting while the scan is running. I wish it could show token usage, some kind of progress, etc.
In short, this isn't an offline scanner. The CLI runs locally but the code and context needed for analysis are sent to the hosted model (OpenAI).
For API, Business, and Enterprise accounts, business data isn't used to train models by default. Retention and other data controls depend on the product and account configuration.
If your company doesn't allow source code to leave its environment, you shouldn't run this against that codebase. Local and third-party endpoints aren't officially supported yet, but you can read through the code and your favorite coding agent will allow you to use it with any model of your choice in 30 seconds.
Anything you ever do with any non-locally-hosted model always "uploads code" to the inference provider because that's how it works: the model uses tools to inspect the code, the result of the tool use is sent in an API call to provide context (and a prompt for the next turn), and then the response continues the process.
This is true and has to be true for any hosted model that works with existing code: it's not specific to this application.
Yes, if only due to the business models being entirely different. AWS sells compute. OpenAI sells models. One of these things benefits from training on your data significantly more than the other.
I don't understand why Snyk is IN business in any way. Who really wants to upload his own code to a company that is specialized at searching security issues?
How can I trust that they show me all findings they have instead of selling the best ones to some three letter organisations?
I was actually discussing solutions for this with my coworkers—building white-hat security agents. It seems like openai/codex-security could simplify a lot of that, or at least provide a version of Codex that's purpose-built for security workflows. Really exciting news!
One thing worth checking in the meantime: OPENAI_API_KEY or CODEX_API_KEY can override an existing ChatGPT/Codex login. If you're trying to use your ChatGPT login, run this in bash or zsh:
unset OPENAI_API_KEY CODEX_API_KEY
Then retry your scan.
If it still fails, could you share the exact error and whether you're using ChatGPT login or an API key? Happy to help debug. You can also file an issue in the repo and we'll take a look!
By default, you can sign in with your ChatGPT/Codex account or use an OPENAI_API_KEY. It also does not require cyber registration but it can help if you encounter refusals. If you give it a try, please feel free to message me, I would love your feedback.
The scanner is the least interesting part of this. The harness around it is the product: dedup across runs, false-positive tracking, budget controls, CI gating. That is the layer where we'll see most interesting innovations in my opinion.
I'm building AQ, a coding harness for teams and the pattern is identical. For a while, I thought the raw model is the answer and quickly changed my mind. Purpose built harnesses are way more powerful than it sounds.
Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly.
If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here.
CLI docs: https://learn.chatgpt.com/docs/security/cli
EDIT: If you'd like to help make this better, we're hiring: https://openai.com/careers/full-stack-software-engineer-cybe...
> Have experience shipping production full-stack products across modern web frontends and backend services.
I'm amazed that the requirements are so low (or at least this vague) for jobs at companies like these.
Has anyone else had the experience of going to an interview and feeling like you were never asked any qualifying questions?
All the questions were easy, your answers were straightforward, you "got them right", but then were not chosen?
I find on the other side, they're also left with dozens of people who "passed" and then it comes down to a pretty arbitrary decision on who gets hired (if we are talking external, no referral, etc.)
I wonder if they can make job descriptions highly specific to filter the shortlist faster and more effectively (to actually get a shortlist).
Anyway end rant. Cool job, hope you fill it.
I interviewed with them and had a great experience despite it ultimately not being a good fit due to communication issues and a bad flu coming on in the final interview - they, and Michael especially, was very kind and understanding.
This sounds like it’s just an app development role, not a security analysis position, so I’m not sure what your complaint is.
> just an app development role
Maybe this is part of the problem
Professional app development requires an understanding of security.
I think the evidence contradicts you at this point
So you don't need to know anything about credit cards for someone to enter it into an app? I don't get the take (is it bad sarcasm?)
To be fair, for the vast majority of cases, no you don't.
It is extremely rare for companies to roll their own payment processing anymore, or even handle PCI scope at all.
Credit card entry, I think you should know a few basics like don't put it in MongoDB?? Or nah? It's just like any other user data?
How about a background check - can anyone take a user-entered DL and randomly Google stuff to see what they find?
Can I store your SSN in plain text in a text file? Why not?
The user had to upload their ID for IDV but I use Vercel. I guess I have to put it on S3. What should the bucket policy be for all these driver's license photos - there are so many???
> Credit card entry
If you're handling credit card numbers yourself, you're in a shrinking subset of developer roles. I've spent the last 7 years of my career working at payment processors, so I do handle that stuff, but the majority of my industry has built an infrastructure that makes it so most developers don't have to think about that.
To your other examples, not everyone on the team needs to know these things up front. Someone in the review process does, and eventually that knowledge gets disseminated and more people know it to carry it forward in their career.
To your last question about bucket policies, clearly you just need to scope it down: arn:aws:sts::*:assumed-role/trustme*/*
(indeed that first wildcard means any account)
Not having clear, objective criteria enables arbitrary decisions (not against OP, I mean in general.. and in general I dislike this pattern a lot). On the extreme other end of the spectrum would be 100% objective criteria, and companies being forced to pick a random applicant that matches them. If they want only the best, they have to have high expectations, but be able to actually define them. You say "cultural fit", I say "corruption", let's blow this whole joint.
> You say "cultural fit", I say "corruption", let's blow this whole joint.
It's not necessarily corruption. If there's no conflict between principal and agent, it's fine.
Just like when I sent my butler to go and buy a bottle of wine, he can make arbitrary choices, but that doesn't mean he's corrupt or going against my wishes. I trust his judgement, and since it's a repeated game our incentives are aligned.
It’s not corruption to simply hire the people you subjectively feel a preference for working for, instead of objective criteria. It’s not public tax funds that fuel salaries, it’s your own money. You get to spend it how you like.
Yes, many jurisdictions have outlawed arbitrary discrimination against protected classes (eg race), which is an entirely different matter, and not what we are discussing here.
If you didn't already know, jobs at highly competitive companies tend to have vague job requirements because they expect to be able to apply your raw intelligence to changing demands quickly. There's no point being hyper-specific about the exact software packages because that's not what they want. What they want is someone who, after talking to an interviewer for 30 minutes, leaves them with the thought "Wow, this person can do anything we need of them. They can probably tell us what we need too and take ownership of large projects. Hire!"
Agree. Being smart, competent, and high in conscientiousness is more important than any highly specific “qualification”. It’s not about checking a bunch of boxes. If you have a track record of getting shit done, you’ll have something to contribute.
This. Being too specific on the stack requirements is a red flag imo.
> There's no point being hyper-specific about the exact software packages because that's not what they want
Okay. This makes it sound like they're more sophisticated but it seems more like they are less sophisticated, less specific, and a lot more vague in the job descriptions they themselves create.
If you look at any technical role, game dev or something where people are building important things at scale - there are a lot of specifics. Libraries, methodologies, where if you didn't know them you are nowhere near a fit.
I'm just wondering. It's OpenAI. Surely there is some domain-specific something beyond "has experience shipping front-end and back-end services" since that includes basically everyone.
It makes this job look like a Starbucks role.
I see where you're coming from, but as someone who regularly interviews engineers, I don't care about specific tech stacks when evaluating a candidate very much either. I can only think of two positions I've worked in where such a thing really mattered.
A good engineer can adapt and catch up without a lot of lead time. For a contractor, I'd be much more specific - but for someone who's going to join my team? I'm looking for a candidate that can demonstrate their problem solving ability, creative thinking and communication skills.
Other than having some kind of experience in the general domain we work in, those "soft skills" are far harder to find than specific tech experience.
This is insane to me.
Not every engineering job is entry level or as simple as most fullstack crud. Deeper into industry you find highly specific well defined positions for a given domain. Soft skills matter more the higher the ladder but id take a killer senior who can be difficult over a team of mediocre staff engineers.
I agree, I recently interviewed at Synthiolabs, and they asked me two questions: one about RAG and two about graph RAG. They rejected me even though I answered correctly, and the interviewer was also a college kid.
This looks great, thanks for open-sourcing it!
How does it deal with the current guardrails 5.6 Sol has on finding vulnerabilities? When I use it in the Codex app it would sometimes say it found a vulnerability, but it cannot tell me what it is.
Thanks! You've run into a real limitation: the CLI doesn't bypass the model's cybersecurity guardrails. If GPT-5.6 Sol finds a vulnerability but refuses to explain it, switching from the Codex app to the CLI won't automatically fix that.
For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals depending on the model and the account or organization where access is provisioned. It isn't a blanket bypass.
If you're an open-source maintainer, you can apply for conditional Codex Security access here:
https://openai.com/form/codex-for-oss/
For enterprise teams, the public Daybreak onboarding guide is here:
https://help.openai.com/en/articles/20001261-enterprise-dayb...
If you have an example of "found a vulnerability but won't tell me what it is," I'd love to take a look too. You can send it to use with /feedback (or message me).
>> For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals
Or perhaps a better option is to use something like Kimi K3 and cancel the GPT subscription altogether.
I tried it, it started a scan but stopped after hitting the rate-limit of my account. It gave up after just a minute of retrying (rate limits are tokens per minute, so... :P).
It said "Partial output was kept at <...>", but I dont see a obvious way of picking it up in a new scan? (The failed run cost me ~$13)
Yeah, you're right. A per-minute rate limit shouldn't kill a scan after a minute, and "partial output was kept" makes it sound like you can pick up where you left off. You can't yet, unfortunately. --max-cost can limit estimated spend, but we still need proper retries and resume. Sorry you spent $13 finding that out. Please send me an email and I'll help make it right.
When would I use this over the plugin in codex? Which I think can be invoked from cli as well
The plugin, including when invoked through the Codex CLI, is great for scanning the repo you're currently working in. The standalone Security CLI/SDK uses the same scanner, but is built for running security across many repos over time: org-wide scans, historical results, deduplication, false-positive tracking, budget controls, and CI integration.
We've been talking to hundreds of engineering and security teams, and their feedback is shaping what we build.
Like Promptfoo, our goal is practical tooling that fits into the workflows teams already have.
Does it require hitting OpenAI's APIs or can one also stand up a local OpenAI compatible LLM endpoint?
We are actively working on officially supporting this. Because it's open source it is pretty easy to point a coding agent at it now and switch out the model.
Exciting! Is there any open GitHub issue we can track?
Not yet, unfortunately. We only just opened the repo, and there isn't a public issue specifically tracking local or OpenAI-compatible endpoint support.
The issue tracker is here: https://github.com/openai/codex-security/issues
If you open an issue with the endpoint or model you want to use, I'd be happy to follow up there.
Thanks! https://github.com/openai/codex-security/issues/52
Been watching your progress for a while, glad OpenAI have looked after you and the team and you still get to ship!
Thank you, that means a lot. Being able to keep building practical, open-source security tooling was important to us.
Really glad we got to ship this, and there's still a lot we want to improve in Codex Security and in Promptfoo!
How does it fare against its own codebase?
Why does this need an entirely separate repo instead of being a feature in the existing Codex project?
> co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
> Thanks for checking this out and for flagging the auth issues.
Offtopic, but this right here is why I don't believe any marketing around "great amazing models that one-shot everything and programmers are no longer needed".
You just have to look at what these labs routinely produce, and their own products.
Edit to respond to @simonw whose comment I saw before he retracted it ;)
This comment is tied directly to consistent continuous claims by the LLM labs. Their own products disprove their own claims, and it would indeed be nice if fewer people believed them :)
Just ran it on a small repo. It ran for almost an hour and then got interrupted. It drained half my weekly usage on a Pro plan.
Working as intended
Well, Sam wouldn't have bought this company to just open source it, would he now? ;)
Classic.
I plan to hack it to use openrouter and Kimi K3 or GLM 5.2 to keep expenses reasonable.
For context can you share the line count?
No need to hack it, we'll add proper support for this.
Excellent. Stuff like this, or LLM SRE automation, are big product spaces for the future.
Give this a try https://infosec.exchange/@st3fan/117000352530023032
FYI: Kimi K3 is relatively expensive on open router API pricing for agentic tasks, or at least that's been my experience playing around with it.
They just opened the weights. I expect competition from various providers will drop the price a bit. But you're right. I'd hope a model closer to GLM 5.2's price would be sufficiently useful.
A license and presumably revshare is required for large scale inference as a service of Kimi K3. There's clearly something going on right now with every router at the same or higher price as the Moonshot list price.
So I wouldn't count on competition if $/mil token is actually set by Moonshot, but I would expect $/tok to drop when there's a more competitive frontier open weight model.
Where did you get that from? That‘s not what the license says: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
The current price is likely a result of the high demand and the high requirements of this model.
if you’re hosting it then you have to have an agreement with Moonshot, which presumably includes terms about pricing.
Oof, that's a bad outcome. Half your weekly usage and a 50-minute scan just to get a HEAD error at the end is not acceptable. --max-cost can help limit estimated spend, but that doesn't fix the underlying problem or give you your quota back. We need to handle a changing checkout and partial results much better. Sorry you ran into this. Please send me an email.
Damn. Just ran it and it used 5 years worth of Pro usage in 5 minutes.
If ya wouldn't mind crediting me a quick 60 months that would be great.
Pro is 10$ a month. You get what you pay for lol.
No, Pro is $100 or $200/mo. Even Plus is $20/mo. Where's this $10/mo Pro plan?
Apologies I misremembered and confused Pro with Plus.
So it drained $5?
Can’t speak to the results, but the cost isn’t high.
Hey looks cool. I tried to run this on a small oss library and here's what happened:
Just some feedback, but it ran for over 40 minutes and during that time I had no idea what was happening, thought it was frozen or in a bad state. Also, it ate through 25% of my weekly credits :(Quick tangent if you’re willing to humor me…
I've been noticing that many new projects that would have been written in Python or Node a year ago are starting to be written in Go, Rust, etc.
Theory: people realized there’s little benefit to Python for agents. As Zep wrote, an “agent is a long-running, concurrent, I/O-bound process that spends most of its time waiting on a model, a tool, or a human[1]” — not a particular strength of Python.
I'm wondering if you'd considered Go (or others—Go’s just my fav ) before landing on Node, and more broadly whether you've noticed a similar pattern?
1: https://blog.getzep.com/agentic-development-in-go/
Go and rust have better guardrails that help agents write better code. Python and JS aren't opinionated enough.
Is that so? I feel like I’m seeing more Python and TypeScript than ever, especially when it comes to AI tooling, which is disappointing.
I can’t fathom why anybody would want to continue working with dynamically typed languages when they can now get types for free.
I suspect a lot of the Python and TypeScript code is getting prompted by folks who don't know any difference between dynamically typed and statically typed.
In the early days of LLM coding, it seemed to be much better at Python for whatever reason. I was never a big Python guy, but I got better results so I ran with it. That definitely doesn't seem to be the case anymore. Last week I asked Codex to mash up Super Mario Bros and Contra ROMs and it just did everything in straight assembly and absolutely crushed it. It couldn't do that 2 years ago. Python is just momentum and I think it's going to die down now that the models are much better.
Don’t want Nintendo to sue you (over [a] game/s you may well actually own!) so withhold my follow-up :)
Huh?
I think it's because python is far more approachable/ubiquitous than go/rust. It's the entry level language for many people from all disciplines of life. Scientific community uses it, data science uses it.
Golang/rust however are very convenient to distribute. Small, portable, fast exe's are very nice. With agentic coding golang/rust are now accessible to a lot more people.
Yes, now that humans write less than 99% of code, the most important criteria for a language isn't readability, which I'd argue was always Python's main selling point, but the underlying runtime. There are practical limits to how fast a Python program can run either under I/O or CPU bound compared to other popular and mature languages with extensive libraries, like Elixir, Go or C++, depending on your use case.
> now that humans write less than 99% of code, the most important criteria for a language isn't readability
please tell me you're reading the AI code
Have it run fuzz and test suites. Get with it man. Most of my LLM projects have massive test suites that do a far better job then I ever would have.
Not really. I test the output thoroughly, I examine the thinking process, I go through the diff to see if anything jumps out but I my thinking process/the way I work has changed. Low level programming thinking has gotten atrophied it seems.
Not often I see companies referring to HN, thanks “ We quietly released the open-source Codex Security CLI, but Hacker News found it before we had a chance to share it here…”
https://x.com/openai/status/2082263717916586117?s=46&t=mnfnj...
security tools from AI companies feel like fire departments run by arsonists. useful, sure, but you can't help noticing who benefits from all the fires
comment feels like someone complaining about being offered a fireproofing solution in the age of flamethrowers.
They're only discovering the security flaws that exist. Would you rather them not be exposed and corrected? To "Slow the testing down"?
I think his point is that AI is really good at finding vulnerabilities.
I think his point was AI is really good at introducing vulnerabilities?
How do we fix scaling issues? More volume of course.
If you’re the one vibe coding you’re the arsonist.
create the problem and sell the cure, tale as old as time
Apropos of nothing: https://en.wikipedia.org/wiki/Marcus_Licinius_Crassus
How to feed your code to OpenAI
It's interesting how much of the value here is providing the english Skill definitions that tell the LLM what to do: https://github.com/openai/codex-security/tree/main/sdk/types...
Some of approaches there could be useful in other contexts. OAI has the compute to experiment with different prompts and I'd expect these to be somewhat optimized.
Yes, I think this is an under-appreciated part of the release. I hope people can adapt them to their own workflows. We run A LOT of evals as the Promptfoo team and we've spent billions of tokens fine-tuning them. You can expect more skills as we branch out to other security workflows and further improvements to the codex security prompts.
Update: As far as I understand, this was already available as a Codex plugin. The main news is that OpenAI has now open-sourced it, and development is still moving quickly.
Alibaba just open sourced their version of a CLI code review tool too.
https://github.com/alibaba/open-code-review
They are entirely different products
I seem to have gotten a bunch of you are trying to stuff we don't allow errors.. very annoying.
Can they explain what types of projects it works on and how does it check I own it? Like will it just not work on Linux kernel even on my own patches to it?
https://news.ycombinator.com/item?id=49090181
Fair question, and I agree the refusals are frustrating.
The CLI doesn't do a repository-ownership check. Public projects are supported, and reviewing your own Linux kernel patches is the kind of defensive work we want to support.
The refusals come from model guardrails, which can be overly cautious. Trusted Access for Cyber (TAC1/Daybreak) is a separate, approved access path that can reduce those refusals.
If you're an open-source maintainer, you can apply for conditional Codex Security access here: https://openai.com/form/codex-for-oss/
For enterprise teams, the Daybreak onboarding process is explained here: https://help.openai.com/en/articles/20001261-enterprise-dayb...
If you have a specific repro, I'd be happy to look into it.
Really glad to see this open-sourced. One thing I'd be interested in is how you think about the balance between false positives and false negatives. In practice, developers tend to stop trusting security tools if they generate too much noise, but missing a real issue is obviously costly too.
If you are a pentester who uses mitmproxy you can check out security skills distilled from 4000 Hackerone public disclosures https://GitHub.com/instavm/security-skills
I reached a mid-run usage cap and burn all my Plus subscription usage. I'm hoping there will be another reset.
This is going to be hell for OSS maintainers. Every llm-kiddie will be opening a security report
I am slightly confused at what this tool adds over just a good system prompt. Does this hit cyber limits as well or is that the main selling point?
Is this useful for pentesting existing systems/infra, or is it only useful for a "review my project for bugs"?
Looks great but the CLI output is not particularly interesting while the scan is running. I wish it could show token usage, some kind of progress, etc.
Yes this is my pet peeve with a lot of the more involved agent skills/processes
Agreed! This is near the top of our priority list and we will make it a lot better soon.
How does it work? Does the tool upload code to ChatGPT for analysis? That may not be allowed for some corporate projects.
In short, this isn't an offline scanner. The CLI runs locally but the code and context needed for analysis are sent to the hosted model (OpenAI).
For API, Business, and Enterprise accounts, business data isn't used to train models by default. Retention and other data controls depend on the product and account configuration.
If your company doesn't allow source code to leave its environment, you shouldn't run this against that codebase. Local and third-party endpoints aren't officially supported yet, but you can read through the code and your favorite coding agent will allow you to use it with any model of your choice in 30 seconds.
More on OpenAI's enterprise data handling: https://openai.com/enterprise-privacy/
Anything you ever do with any non-locally-hosted model always "uploads code" to the inference provider because that's how it works: the model uses tools to inspect the code, the result of the tool use is sent in an API call to provide context (and a prompt for the next turn), and then the response continues the process.
This is true and has to be true for any hosted model that works with existing code: it's not specific to this application.
Yes, I suspect companies that don’t allow ChatGPT will not be able to use the ChatGPT security analysis tool.
Amazon bedrock is an option for gpt models that does not send your data to openai.
Does Amazon offer better privacy guarantees than OpenAI?
Yes, if only due to the business models being entirely different. AWS sells compute. OpenAI sells models. One of these things benefits from training on your data significantly more than the other.
I wonder if tools like this will put companies like snyk out of business. We use snyk at work and I have not been satisfied.
I like to think it just upped the bar, but good durable expertise will need to rise with it.
Why would a few code snippets put Snyk out of business?
I don't understand why Snyk is IN business in any way. Who really wants to upload his own code to a company that is specialized at searching security issues?
How can I trust that they show me all findings they have instead of selling the best ones to some three letter organisations?
Is this the same plugin found in Codex? @codex Security?
I was actually discussing solutions for this with my coworkers—building white-hat security agents. It seems like openai/codex-security could simplify a lot of that, or at least provide a version of Codex that's purpose-built for security workflows. Really exciting news!
Just getting auth issues so far...
https://news.ycombinator.com/item?id=49090181
yeah same here
Sorry about that. We hit an authentication issue at launch and have now merged and deployed a fix in 0.1.1:
https://github.com/openai/codex-security/pull/22
One thing worth checking in the meantime: OPENAI_API_KEY or CODEX_API_KEY can override an existing ChatGPT/Codex login. If you're trying to use your ChatGPT login, run this in bash or zsh:
Then retry your scan.If it still fails, could you share the exact error and whether you're using ChatGPT login or an API key? Happy to help debug. You can also file an issue in the repo and we'll take a look!
I don't think there's much to this other than it being a convenient CI wrapper around their existing models?
Edit: there's a little bit more meat here: https://github.com/openai/codex-security/tree/main/sdk/types...
All of codex is a wrapper around their models. There’s still value in a purpose-built harness.
Yeah, I think so too.
Yeah but management loved the idea.
be careful , your code will go to the cloud/ai using this
Yeah, this isn’t exactly new for us :d
would love it to see it h2h against https://github.com/usestrix/strix (45k stars)
They are entirely different products
yeah you mean because OAI is only whitebox? or expand on that a bit, haven't played around a ton w the oss codex sec
from plugin to main focus, wow
Allow only OpenAi key? Requires Cyber registration? Yes. Yes. Useless.
By default, you can sign in with your ChatGPT/Codex account or use an OPENAI_API_KEY. It also does not require cyber registration but it can help if you encounter refusals. If you give it a try, please feel free to message me, I would love your feedback.
How can I trust this wont go rogue and hack Hugging Face?
I got my agent to analyze it and do a write up here:
https://wasnotwas.com/writing/inside-openai-codex-security/
oh yeah well I got my agent to analyze your write up and do a write up here:
https://pornoscan.us/codex-security-analysis.html
The scanner is the least interesting part of this. The harness around it is the product: dedup across runs, false-positive tracking, budget controls, CI gating. That is the layer where we'll see most interesting innovations in my opinion.
I'm building AQ, a coding harness for teams and the pattern is identical. For a while, I thought the raw model is the answer and quickly changed my mind. Purpose built harnesses are way more powerful than it sounds.