Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.
It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.
> We then ran AISLE's autonomous AI system against curl.
They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.
The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.
Thanks for figuring that out. Sort of sounds like AI programming programs to find vulnerabilities, of which fuzzing is one of the proven techniques to do it.
Back when I first heard of it, the HuggingFace hack reminded me about how you don't have to be particularly smart to find vulns, just aggressive in looking for them. AFL on its own "learned" how to construct well-formed JPEGs and PDFs by fuzzing decoders for these formats, back in the pre-LLM dark ages.
Of course the HuggingFace hack proved to be much more than that...
Setting a swarm of agents loose for hours to look for software vulnerabilities is far more compute-expensive than fuzzing. The industry has never thrown this kind of compute resources at pure fuzzing, in part because you can't get much VC money for that.
You also quickly get into diminishing returns with fuzzing. Generally a bug is either going to be found relatively quickly with a given fuzzing approach or it's going to be nearly impossible to find. You're usually better coming up with more intelligent fuzzing approaches than you are just dumping compute into it.
This sounds like a swarm of agents with particular prompting that happens to guide the LLMs toward doing a lot of fuzzing, so it's not either/or; you're getting all the compute requirements of both.
Fuzzing or having the LLM sort through where might be most useful to fuzz & sorting the results? Neither seem particularly compute intensive to me, fuzzing is a pretty standard step and having the LLM read through to find the most interesting areas to fuzz sounds a lot more efficient than leaving the whole task to the LLM.
It wouldn’t surprise me if AISLE uses many different providers’ models, and what’s holding back OpenAI and Anthropic is only using first-party models. Just because OpenAI and Anthropic have arguably the strongest models overall doesn’t mean their models are the strongest at finding any given class of vulnerability or lead to follow.
Maybe the model doesn’t matter, maybe you just need something minimally intelligent to seed the fuzzer, generate a test case, and rinse and repeat when the fuzzer gets stuck.
The tool basically had to chain two exploits together to reach this. It also came up with a patch to fix which was fairly sensible (but I ended up editing it further for clarity).
Curl has a well earned reputation for high quality code. If you find something there it means you are good. There is a lot of software where finding a vulnerability mostly means you bothered to look and are not completely stupid. Nobody is going to be impressed if you find an issue with something that everybody already knows is poorly coded.
Its vulnerability overview page[0] also helpfully lists which were caused by the use of C. The fact that even an extremely-high-quality codebase like Curl is still encountering things like use-after-free issues is pretty damning for the people clamoring that it is perfectly possible for skilled programmers to write safe C.
If even Daniel Stenberg can't consistently write safe C code, I think we can assume that nobody can. Either you switch to a memory-safe language like Rust, or you adopt a high-cost runtime checker like Fil-C, or you'll have to live with a never-ending series of memory safety vulnerabilities.
I don't know, that just seems like a category error that breaks apart when you get closer to any individual project. Use-after-free is still possible in Rust with unsafe. Assuming nobody _can_ write memory safe C is a good assumption for things like security modelling, regardless of how unlikely those issues actually come up.
Securing code you/your org did not write and programming for yourself/your org are just fundamentally different jobs.
It is absolutely possible, but all of the data we currently have shows that reducing the amount of code that could possibly have the error has meaningful effects on the number of vulnerabilities. Mitigation may not be the same as elimination, but it is effective.
> Use-after-free is still possible in Rust with unsafe
Of course! But in practice all potential bugs are neatly wrapped in small and easy-to-audit "unsafe" blocks, rather than silently lurking all over your codebase.
You could indeed wrap your entire codebase into one giant unsafe statement and write it like C. But, as the actix-web discussion showed years ago, the Rust community very much prefers restricting unsafe to the absolute bare minimum possible. You wouldn't write, say, a mail server in mostly-unsafe Rust for the same reason that you wouldn't write it in mostly-inline-assembly: you gain nothing, and in return it'll probably blow up in your face sooner rather than later.
Rust has an escape hatch because we're all adults. The big difference is that its footgun has an explicit safety latch, so you have to deliberately opt in to blowing your own foot off.
It isn't very extreme, in my opinion. I am only making one claim, and it's a rather modest one: Memory safety bugs aren't any different from other bugs.
The "you're holding it wrong" crowd has been claiming that memory safety bugs are a skill issue for ages now, and that the issue is overblown and can be solved by having programmers suck less.
Curl provides the counterexample to this. Its main developer seems to be quite skilled, as reiterated by the various AI audits it is a very solid codebase, yet it still suffers from memory safety vulnerabilities. If the "you're holding it wrong" crowd is right, we're faced with a contradiction. Is Curl secretly a poor-quality codebase which has managed to fool the community until now? Is Daniel Stenberg uniquely susceptible to writing memory safety bugs, negating his otherwise-seemingly-decent programming skills? Is he perhaps intentionally introducing memory safety bugs to make C look bad?
In other words: if memory safety bugs are indeed a different breed altogether and are completely avoidable if we try hard enough, why are they still hitting Curl?
I posit that they are not: if you are writing unsafe code, you will introduce memory safety bugs. You can avoid them altogether by switching to inherently-safe languages like Go or Kotlin, by catching them at compile time like Rust, or by catching them at runtime like Fil-C. Or you can ignore the problem altogether and keep writing C like we've been doing for decades, but that means you will keep shipping memory safety vulnerabilities - like we've been doing for decades.
> Memory safety bugs aren't any different from other bugs.
This is highly dependent on what kind of software you are writing.
On the one end, there's stuff like an image format parser in a browser -- a pure function from untrusted bytes to untrusted pixels.
In a memory-safe language, it's pretty hard for such code to have vulnerabilities (other than DoS) in such code -- you'd have to explicitly go out of your way to do weird stuff (open unrelated files, start subprocesses, ...). Simple logic bugs can only lead the wrong pixels or panics. But when memory safety bugs are possible, remote code execution is common in such code. Memory-safe languages make a massive difference here!
On the other end, you have stuff like a javascript JIT compiler, which turns untrusted javascript into trusted machine code -- here pretty much any logic bug leading to "wrong output" can be turned into a remote code execution exploit. Memory safe languages are not very useful here.
> Curl has a well earned reputation for high quality code.
SQLite also has a very good reputation. I vaguely recall hearing about one SQLite vulnerability discovered via AI, but I thought it turned out to be a nothingburger. A quick search turned up CVE-2025-6965[0,1], published on 2025-07-15, which affects SQLite < 3.50.2 (versions published before 2025-05-29[2]).
I'm not much of a security nerd, but my naive reading of this implies that it was already known and fixed as of the time of the CVE; in other words, the AI discovery didn't seem particularly helpful (though one could argue that it did successfully discover a CVE).
Has AI found many/any other vulnerabilities in SQLite?
Hard to say what AI has been used in SQLite unless someone is talking. My impression is the maintainer doesn't talk about this type of thing much, but maybe I'm wrong.
Anyone care to guesstimate how much effort there would be in creating an actually-secure curl and openssl? Using something like Common Lisp or Lean, instead of C.
Since AISLE reported 29 issues but only 6 warranted a CVE, and all the found CVEs were "low" severity, this makes me wonder if AISLE simply is tuned for a higher false positive rate than the anthropic and openai tools (which may have found the same 6 issues and decided not to report them)
i don't think this is correct. if you look at this article by the curl founder daniel stenberg (https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-v...), he talks about how he previously ran Mythos on curl and that it found 5 issues: 1 turned out to be a low severity CVE, 3 were false positives, and 1 just a bug. So a) Mythos detects low severity CVEs too, and b) it is fairly noisy
In the case of big projects like curl the interaction seems a bit more complete. E.g. There are some other blog posts about how the engagements and reviews worked which go decently beyond a pre-filtered dump of high severity CVE claims appearing out of the blue.
Very unrelated to the content of the article, but that is a pretty weird ft ligature in the heading. It looks a letter from another alphabet. Which maybe makes this pretty cool after all.
Good marketing and definitive proof that local (read: on-prem & air-gapped) models with correct context and tools are good enough to perform on par and above SOTA cloud hosted solutions.
We have seen this point many times before with different technologies. The first computers at university were big and expensive, same as this machine. Give it a few years and this functionality will be a commodity.
i think a key missing part is that an LLM on its own can't find vulnerabilities, so it's always an AI + harness. even mythos, when used for finding zero-days, is using an actually surprisingly heavy handed and expensive scaffold. they literally make it run in parallel on ±all files and ask "what's wrong with this?". here from the mythos technical blog post [1]:
> To increase efficiency, instead of processing literally every file for each software project that we evaluate, we first ask Claude to rank how likely each file in the project is to have interesting bugs on a scale of 1 to 5. A file ranked “1” has nothing at all that could contain a vulnerability (for instance, it might just define some constants). Conversely, a file ranked “5” might take raw data from the Internet and parse it, or it might handle user authentication. We start Claude on the files most likely to have bugs and go down the list in order of priority.
So they process it in parallel, but AI-rank them based on vuln-likelihood first = exhaustive search with a heuristic filtering first
OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed.
If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then severity may come into place - all 6 are rated low impact)
I think you might be misunderstanding this? This is, from my understanding, what went down:
1. curl was scanned by many different things, including AISLE, and many bugs were fixed <- all this was in the past
2. curl a week ago was scanned again my Mythos and Codex Security, and both of them said: 0 issues found
3. the same curl was scanned by AISLE a day later, resulting in ~29 reports (based on the blog post and mastodon posts from Daniel Stenberg)
4. of these 29, 6 cleared the bar and got CVEs in curl
5. these 6 CVEs were just announced as fixed in curl 8.22.0 today, together with 4 more CVEs that were detected by other people prior to point 2. of this list
so imho it was head-to-head, the very same codebase => it's a legit comparison
Curl has been scanned by mythos and several other AI tools several times over the past year already. Mythos found nothing this round, but when it was first released it found issues which have been fixed - and several other AIs had already scanned curl for issues which had been fixed by then.
We can say that this is a useful tool, but is it better or worse than the others - there is no way to make that conclusion.
Edit, wait, are you claiming that asile was already used and those issues fixed, and it still found more. That is valid, but it only says that asile is better by enough that is worth an upgrade, while the others probably are not. It is not valid in comparing to other tools. (assuming this is true, I've never heard of asile before this)
I don't understand your objection. If tool A says "job is done" and tool B says "found N additional tasks that need doing" how does the history matter? B is turning up things that A isn't thus B is performing better. They both had access to the same inputs here.
I suppose it's interesting to wonder if B would have turned up issue X which A previously found. But that seems largely academic to me. There is a code base right now with N known issues (thanks to B) and A is saying everything is good. It seems like that's all that should matter here.
"How many total vulnerabilities can your tool alone identify?" and "How many unique vulnerabilities can your tool identify?" are both valid comparisons to make IMO.
But that isn't what happened! Mythos has found issues in the past - which are now fixed (or so we should assume, I didn't verify but curl is very good about fixing issues). At most we can say the current version of mythos isn't better than the last version (a new version of mythos was just released, I'm not sure if that was even the one used in this scan)
I guess this would also require models trained on pre-2023 data - or not trained on later curl code, changelogs, blog posts discussing curl security fixes, etc.
i don't think this is doable fairly. as they say in the blog post, the only fair way to is to look for new, previously undiscovered zero-days, otherwise you always risk the model has in some way been trained on the vulnerabilities. looking for legit new stuff is the only way to prevent leakage (even accidental one)
One does not "discover" a CVE like this. To discover a CVE would mean you searched for a particular piece of software and found it vulnerable according to the NVD. That's not a novel discovery by any means.
What they did is they found bugs and that they were exploitable in certain edge cases. As the bugs turned out to be vulnerabilities, they were assigned a CVE in the NVD with low severity.
IMHO Aisle stockedpiled too much in the marketing shelves.
Wow, this announcement is good content marketing.
Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.
It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.
> We then ran AISLE's autonomous AI system against curl.
They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.
The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.
Thanks for figuring that out. Sort of sounds like AI programming programs to find vulnerabilities, of which fuzzing is one of the proven techniques to do it.
It's an effective approach. Google project zero started doing this in 2024
https://security.googleblog.com/2024/11/leveling-up-fuzzing-...
Back when I first heard of it, the HuggingFace hack reminded me about how you don't have to be particularly smart to find vulns, just aggressive in looking for them. AFL on its own "learned" how to construct well-formed JPEGs and PDFs by fuzzing decoders for these formats, back in the pre-LLM dark ages.
Of course the HuggingFace hack proved to be much more than that...
Also sounds incredibly compute intensive.
Setting a swarm of agents loose for hours to look for software vulnerabilities is far more compute-expensive than fuzzing. The industry has never thrown this kind of compute resources at pure fuzzing, in part because you can't get much VC money for that.
You also quickly get into diminishing returns with fuzzing. Generally a bug is either going to be found relatively quickly with a given fuzzing approach or it's going to be nearly impossible to find. You're usually better coming up with more intelligent fuzzing approaches than you are just dumping compute into it.
This sounds like a swarm of agents with particular prompting that happens to guide the LLMs toward doing a lot of fuzzing, so it's not either/or; you're getting all the compute requirements of both.
Fuzzing or having the LLM sort through where might be most useful to fuzz & sorting the results? Neither seem particularly compute intensive to me, fuzzing is a pretty standard step and having the LLM read through to find the most interesting areas to fuzz sounds a lot more efficient than leaving the whole task to the LLM.
[dead]
Their system can run with various models, they go into more details in this article.
https://aisle.com/blog/system-over-model-zero-day-discovery-...
It wouldn’t surprise me if AISLE uses many different providers’ models, and what’s holding back OpenAI and Anthropic is only using first-party models. Just because OpenAI and Anthropic have arguably the strongest models overall doesn’t mean their models are the strongest at finding any given class of vulnerability or lead to follow.
>All six are rated Low severity
This says it all. Nothing important was missing. This is marketing hype.
> what models the use under the hood
Presumably their own, wouldn’t they?
You mean their own trained models, or do you think it's an open source model that they fine-tuned? If they use their own, I'd guess it's the latter.
Maybe the model doesn’t matter, maybe you just need something minimally intelligent to seed the fuzzer, generate a test case, and rinse and repeat when the fuzzer gets stuck.
Default to gpt 5.4 nano
https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
> Wow, this announcement is good content marketing.
Why do you think companies hire PMMs?
It defaults to gpt5.4 nano
https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
A repo named "nano-analyzer" unsurprisingly uses gpt5.4 nano. I doubt their "pay them money" version uses nano.
I am pretty sure that the nano-analyzer is just a limited open source demo of their "System over Model" thesis from https://aisle.com/blog/system-over-model-zero-day-discovery-..., not the main product.
We had a few AISLE-generated security reports, and the signal to noise was reasonably good.
The most notable bug/exploit their scanner found was: https://gitlab.com/nbdkit/libnbd/-/commit/e50bbd2681117c2dd8...
The tool basically had to chain two exploits together to reach this. It also came up with a patch to fix which was fairly sensible (but I ended up editing it further for clarity).
Curl seems to becoming one of the favourite things to demo AI finding vulns.
Curl is going to end up incredibly secure.
Curl has a well earned reputation for high quality code. If you find something there it means you are good. There is a lot of software where finding a vulnerability mostly means you bothered to look and are not completely stupid. Nobody is going to be impressed if you find an issue with something that everybody already knows is poorly coded.
Its vulnerability overview page[0] also helpfully lists which were caused by the use of C. The fact that even an extremely-high-quality codebase like Curl is still encountering things like use-after-free issues is pretty damning for the people clamoring that it is perfectly possible for skilled programmers to write safe C.
If even Daniel Stenberg can't consistently write safe C code, I think we can assume that nobody can. Either you switch to a memory-safe language like Rust, or you adopt a high-cost runtime checker like Fil-C, or you'll have to live with a never-ending series of memory safety vulnerabilities.
[0]: https://curl.se/docs/security.html
I don't know, that just seems like a category error that breaks apart when you get closer to any individual project. Use-after-free is still possible in Rust with unsafe. Assuming nobody _can_ write memory safe C is a good assumption for things like security modelling, regardless of how unlikely those issues actually come up.
Securing code you/your org did not write and programming for yourself/your org are just fundamentally different jobs.
It is absolutely possible, but all of the data we currently have shows that reducing the amount of code that could possibly have the error has meaningful effects on the number of vulnerabilities. Mitigation may not be the same as elimination, but it is effective.
> Use-after-free is still possible in Rust with unsafe
Of course! But in practice all potential bugs are neatly wrapped in small and easy-to-audit "unsafe" blocks, rather than silently lurking all over your codebase.
You could indeed wrap your entire codebase into one giant unsafe statement and write it like C. But, as the actix-web discussion showed years ago, the Rust community very much prefers restricting unsafe to the absolute bare minimum possible. You wouldn't write, say, a mail server in mostly-unsafe Rust for the same reason that you wouldn't write it in mostly-inline-assembly: you gain nothing, and in return it'll probably blow up in your face sooner rather than later.
Rust has an escape hatch because we're all adults. The big difference is that its footgun has an explicit safety latch, so you have to deliberately opt in to blowing your own foot off.
Where is your extreme assessment coming from?
It isn't very extreme, in my opinion. I am only making one claim, and it's a rather modest one: Memory safety bugs aren't any different from other bugs.
The "you're holding it wrong" crowd has been claiming that memory safety bugs are a skill issue for ages now, and that the issue is overblown and can be solved by having programmers suck less.
Curl provides the counterexample to this. Its main developer seems to be quite skilled, as reiterated by the various AI audits it is a very solid codebase, yet it still suffers from memory safety vulnerabilities. If the "you're holding it wrong" crowd is right, we're faced with a contradiction. Is Curl secretly a poor-quality codebase which has managed to fool the community until now? Is Daniel Stenberg uniquely susceptible to writing memory safety bugs, negating his otherwise-seemingly-decent programming skills? Is he perhaps intentionally introducing memory safety bugs to make C look bad?
In other words: if memory safety bugs are indeed a different breed altogether and are completely avoidable if we try hard enough, why are they still hitting Curl?
I posit that they are not: if you are writing unsafe code, you will introduce memory safety bugs. You can avoid them altogether by switching to inherently-safe languages like Go or Kotlin, by catching them at compile time like Rust, or by catching them at runtime like Fil-C. Or you can ignore the problem altogether and keep writing C like we've been doing for decades, but that means you will keep shipping memory safety vulnerabilities - like we've been doing for decades.
> Memory safety bugs aren't any different from other bugs.
This is highly dependent on what kind of software you are writing.
On the one end, there's stuff like an image format parser in a browser -- a pure function from untrusted bytes to untrusted pixels. In a memory-safe language, it's pretty hard for such code to have vulnerabilities (other than DoS) in such code -- you'd have to explicitly go out of your way to do weird stuff (open unrelated files, start subprocesses, ...). Simple logic bugs can only lead the wrong pixels or panics. But when memory safety bugs are possible, remote code execution is common in such code. Memory-safe languages make a massive difference here!
On the other end, you have stuff like a javascript JIT compiler, which turns untrusted javascript into trusted machine code -- here pretty much any logic bug leading to "wrong output" can be turned into a remote code execution exploit. Memory safe languages are not very useful here.
I meant extreme as in "extremely-high-quality codebase", without which a single example of curl means little for your broader point.
> Curl has a well earned reputation for high quality code.
SQLite also has a very good reputation. I vaguely recall hearing about one SQLite vulnerability discovered via AI, but I thought it turned out to be a nothingburger. A quick search turned up CVE-2025-6965[0,1], published on 2025-07-15, which affects SQLite < 3.50.2 (versions published before 2025-05-29[2]).
I'm not much of a security nerd, but my naive reading of this implies that it was already known and fixed as of the time of the CVE; in other words, the AI discovery didn't seem particularly helpful (though one could argue that it did successfully discover a CVE).
Has AI found many/any other vulnerabilities in SQLite?
[0] https://cybersecuritynews.com/sqlite-0-day-vulnerability/
[1] https://nvd.nist.gov/vuln/detail/cve-2025-6965
[2] https://sqlite.org/releaselog/3_50_2.html
Hard to say what AI has been used in SQLite unless someone is talking. My impression is the maintainer doesn't talk about this type of thing much, but maybe I'm wrong.
Tailscale worked with them on an issue they found, not a vulnerability:
https://tailscale.com/blog/sqlite-wal-reset-bug
Thank goodness because curl is a load bearing structure to the backend of the internet.
Anyone care to guesstimate how much effort there would be in creating an actually-secure curl and openssl? Using something like Common Lisp or Lean, instead of C.
Since AISLE reported 29 issues but only 6 warranted a CVE, and all the found CVEs were "low" severity, this makes me wonder if AISLE simply is tuned for a higher false positive rate than the anthropic and openai tools (which may have found the same 6 issues and decided not to report them)
i don't think this is correct. if you look at this article by the curl founder daniel stenberg (https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-v...), he talks about how he previously ran Mythos on curl and that it found 5 issues: 1 turned out to be a low severity CVE, 3 were false positives, and 1 just a bug. So a) Mythos detects low severity CVEs too, and b) it is fairly noisy
As far as I understand it, the other efforts have not reported most of their findings to upstream developers, focusing on critical findings only.
This is understandable because upstream interactions at scale are difficult.
In the case of big projects like curl the interaction seems a bit more complete. E.g. There are some other blog posts about how the engagements and reviews worked which go decently beyond a pre-filtered dump of high severity CVE claims appearing out of the blue.
That's bragging rights correctly earned, i think! As marketing-y as this post is, definitely something to keep an eye on.
Very unrelated to the content of the article, but that is a pretty weird ft ligature in the heading. It looks a letter from another alphabet. Which maybe makes this pretty cool after all.
It's a very unfortunate ligature imho because there is no distinction between f and t. Looks funky
PS: the image in the background looks like a nod to David vs Goliath, love that detail
Good marketing and definitive proof that local (read: on-prem & air-gapped) models with correct context and tools are good enough to perform on par and above SOTA cloud hosted solutions.
We have seen this point many times before with different technologies. The first computers at university were big and expensive, same as this machine. Give it a few years and this functionality will be a commodity.
I also have some secret recipe finding one class of bugs: https://github.com/tmux/tmux/issues?q=is%3Apr%20author%3Ablm...
curious they're willing to run AISLE on tmux to find more than mine.
Given enough AIballs all bugs are shallow
I like the looks of Aisle and what they stand for...
That being said you cannot compare a model with a specialised harness. These are two completely different things.
Am I missing something?
i think a key missing part is that an LLM on its own can't find vulnerabilities, so it's always an AI + harness. even mythos, when used for finding zero-days, is using an actually surprisingly heavy handed and expensive scaffold. they literally make it run in parallel on ±all files and ask "what's wrong with this?". here from the mythos technical blog post [1]:
> To increase efficiency, instead of processing literally every file for each software project that we evaluate, we first ask Claude to rank how likely each file in the project is to have interesting bugs on a scale of 1 to 5. A file ranked “1” has nothing at all that could contain a vulnerability (for instance, it might just define some constants). Conversely, a file ranked “5” might take raw data from the Internet and parse it, or it might handle user authentication. We start Claude on the files most likely to have bugs and go down the list in order of priority.
So they process it in parallel, but AI-rank them based on vuln-likelihood first = exhaustive search with a heuristic filtering first
[1] https://www.anthropic.com/research/mythos-preview
The gauntlet has been thrown. Will Anthropic or OpenAI pick it up?
OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed.
If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then severity may come into place - all 6 are rated low impact)
I think you might be misunderstanding this? This is, from my understanding, what went down:
1. curl was scanned by many different things, including AISLE, and many bugs were fixed <- all this was in the past 2. curl a week ago was scanned again my Mythos and Codex Security, and both of them said: 0 issues found 3. the same curl was scanned by AISLE a day later, resulting in ~29 reports (based on the blog post and mastodon posts from Daniel Stenberg) 4. of these 29, 6 cleared the bar and got CVEs in curl 5. these 6 CVEs were just announced as fixed in curl 8.22.0 today, together with 4 more CVEs that were detected by other people prior to point 2. of this list
so imho it was head-to-head, the very same codebase => it's a legit comparison
Curl has been scanned by mythos and several other AI tools several times over the past year already. Mythos found nothing this round, but when it was first released it found issues which have been fixed - and several other AIs had already scanned curl for issues which had been fixed by then.
We can say that this is a useful tool, but is it better or worse than the others - there is no way to make that conclusion.
Edit, wait, are you claiming that asile was already used and those issues fixed, and it still found more. That is valid, but it only says that asile is better by enough that is worth an upgrade, while the others probably are not. It is not valid in comparing to other tools. (assuming this is true, I've never heard of asile before this)
I don't understand your objection. If tool A says "job is done" and tool B says "found N additional tasks that need doing" how does the history matter? B is turning up things that A isn't thus B is performing better. They both had access to the same inputs here.
I suppose it's interesting to wonder if B would have turned up issue X which A previously found. But that seems largely academic to me. There is a code base right now with N known issues (thanks to B) and A is saying everything is good. It seems like that's all that should matter here.
The "Better" is where I have an issue. There isn't enough information here to say one is better than the other. Both are useful tools is clear.
"How many total vulnerabilities can your tool alone identify?" and "How many unique vulnerabilities can your tool identify?" are both valid comparisons to make IMO.
But that isn't what happened! Mythos has found issues in the past - which are now fixed (or so we should assume, I didn't verify but curl is very good about fixing issues). At most we can say the current version of mythos isn't better than the last version (a new version of mythos was just released, I'm not sure if that was even the one used in this scan)
I guess this would also require models trained on pre-2023 data - or not trained on later curl code, changelogs, blog posts discussing curl security fixes, etc.
i don't think this is doable fairly. as they say in the blog post, the only fair way to is to look for new, previously undiscovered zero-days, otherwise you always risk the model has in some way been trained on the vulnerabilities. looking for legit new stuff is the only way to prevent leakage (even accidental one)
That’s my point too.
One does not "discover" a CVE like this. To discover a CVE would mean you searched for a particular piece of software and found it vulnerable according to the NVD. That's not a novel discovery by any means.
What they did is they found bugs and that they were exploitable in certain edge cases. As the bugs turned out to be vulnerabilities, they were assigned a CVE in the NVD with low severity.
IMHO Aisle stockedpiled too much in the marketing shelves.
What would the correct terminology actually be here I wonder?
In short, they found vulnerabilities and each have been assigned CVEs.
"your LLM is cool, but can it find vulns in curl"
Marketing Slop
This is an ad. I didn't learn anything from reading it.