#Mythos finds a #curl vulnerability
-
My personal conclusion can however not end up with anything else than that the big hype around this model so far was primarily marketing. I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing.
@bagder
In terms of evidence to the contrary:
Check out
https://social.security.plumbing/@freddy/116549451049357174 / the blog post:
https://hacks.mozilla.org/2026/05/behind-the-scenes-hardening-firefox/>270 vulnerabilities found by Mythos fixed in a single Firefox release.
That's just one data point, but interestingly far off from yours.
-
#Mythos finds a #curl vulnerability
yes, as in singular one.
Mythos finds a curl vulnerability
yes, as in singular one. Back in April 2026 Anthropic caused a lot of media noise when they concluded that their new AI model Mythos is dangerously good at finding security flaws in source code. Apparently Mythos was so good at this that Anthropic would not release this model to the public yet but instead … Continue reading Mythos finds a curl vulnerability →
daniel.haxx.se (daniel.haxx.se)
I love it :
"The AI reviews are used in addition to the human reviews. They help us, they don’t replace us."
-
My personal conclusion can however not end up with anything else than that the big hype around this model so far was primarily marketing. I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing.
@bagder How do you explain that Mythos found 271 bugs in Firefox, and counting, and only 1 in cURL. Is the Firefox code base 271 times larger?
-
"Zero memory-safety vulnerabilities found."
@bagder b-b-b-but curl is not in Rust!
-
@bagder How do you explain that Mythos found 271 bugs in Firefox, and counting, and only 1 in cURL. Is the Firefox code base 271 times larger?
@gnirre I do not explain that at all because I don't have enough knowledge to do so.
-
@bagder @david_chisnall I'm not going to advocate actually doing this because it's expensive and I'm not a fan of the environmental impacts, but I am curious what it would find if you pointed it at the codebase from a time before the other precursor tools like fuzzers were in use. How many bugs can it find that you know with hindsight are there to be found?
The original Coverity paper claimed, as I recall, 300 CVEs. I'm not sure what the severity distribution was, but that seems a lot more than Mythos, and they probably used less compute than a single Mythos query.
The problem with any static analyser, whether it's based on formal reasoning or pattern recognition, is that it will be unsound (i.e. it will have false positives, in contrast with dynamic analyses that are incomplete and have false negatives). The LLM-based tools are no different in this respect. From a Claude 'comprehensive code review' of one of my projects, the only serious bug in the top ten that it found was one that already had an open PR to fix, and two were not only not bugs, they were intentional design choices and doing it the other way would have caused serious performance regressions (and not fixed bugs).
The thing that does make Mythos different is that it tries to build a PoC exploit. This will reduce the false positive rate, at the expense of creating false negatives (if it can't produce a PoC, you ignore it).
When I've used Coverity on a large project, it's found tens of thousands of bugs, and most of them are false positives, so it requires a lot of effort to find the ones that are actually important bugs. Something that produces PoCs automatically would help this a lot.
The baseline data point I'd really like to see is something that integrates the clang analyser with libFuzzer. For each report the analyser finds, insert profiling points at the branches on the control flow chain that it recommends, then automatically drive the fuzzer to try to trigger the code paths that the analyser reported as potential issues.
The default settings for the clang analyser are compilation-unit-at-a-time and with reduced bounds on loop iteration counts to avoid using enormous amounts of memory. If you're willing to spend as much money as it costs to operate the LLM-based tools, you can use the cross-compilation-unit approaches and bump the state up a lot. Running it configured to use a comparable amount of RAM to the GPUs that the Anthropic models run on would let you do a lot of symbolic execution.
-
#Mythos finds a #curl vulnerability
yes, as in singular one.
Mythos finds a curl vulnerability
yes, as in singular one. Back in April 2026 Anthropic caused a lot of media noise when they concluded that their new AI model Mythos is dangerously good at finding security flaws in source code. Apparently Mythos was so good at this that Anthropic would not release this model to the public yet but instead … Continue reading Mythos finds a curl vulnerability →
daniel.haxx.se (daniel.haxx.se)
@bagder In line with what this blog post stated shortly after it was announced: the model is nothing special and much cheaper models can find the same bugs. Marketing BS turned to 11. https://www.flyingpenguin.com/the-boy-that-cried-mythos-verification-is-collapsing-trust-in-anthropic/
-
@gnirre I do not explain that at all because I don't have enough knowledge to do so.
@bagder Did Anthropic know that you finally had gotten access to Mythos?
-
@bagder Did Anthropic know that you finally had gotten access to Mythos?
@gnirre no idea, probably not
-
#Mythos finds a #curl vulnerability
yes, as in singular one.
Mythos finds a curl vulnerability
yes, as in singular one. Back in April 2026 Anthropic caused a lot of media noise when they concluded that their new AI model Mythos is dangerously good at finding security flaws in source code. Apparently Mythos was so good at this that Anthropic would not release this model to the public yet but instead … Continue reading Mythos finds a curl vulnerability →
daniel.haxx.se (daniel.haxx.se)
@bagder one? wow, that really was worth burning the planet's resources.
-
@bagder I suspect the question is, will it still be a worthwhile tool when the actual price to use the tool, not subsidized by anyone's war chest or VC, is revealed?
-
@gnirre no idea, probably not
@bagder Maybe my question should have been if Alpha Omega knew? Your access was "inofficial"?
-
@bagder Maybe my question should have been if Alpha Omega knew? Your access was "inofficial"?
@gnirre I don't know how much they asked or told A about when this was done. It's not "my" access, someone else has the access and ran the analysis
-
-
@bagder b-b-b-but curl is not in Rust!
@synlogic4242 @bagder Yes, someone really needs to get on to that rewriting thing. Just a pity there hasn't been a weekend in *years* so nobody had the chance!
-
Especially if it's subscription-based, as these models seem to be good at finding only specific sets of problems and then dry out, but even 10k per use is really gov or big corpo territory.
-
My personal conclusion can however not end up with anything else than that the big hype around this model so far was primarily marketing. I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing.
@bagder This suggests a fun exercise for someone interested in messing around with LLMs:
1. Put back all the curl security issues previously found by LLM tools by dropping the fix commits from history or otherwise obfuscating the revert.
2. Feed the re-vulnerabilized repo to a selection of models and see what are the cheapest ones (by memory, time and/or monetary cost) that can find, say, 50%/75%/100% of the issues found by the warehouse-scale "foundation models".
Feels like a large part of the current results should be doable with significantly smaller resources, because being trained on every tweet and reddit post and libgen book ever is not obviously related to the task.
-
#Mythos finds a #curl vulnerability
yes, as in singular one.
Mythos finds a curl vulnerability
yes, as in singular one. Back in April 2026 Anthropic caused a lot of media noise when they concluded that their new AI model Mythos is dangerously good at finding security flaws in source code. Apparently Mythos was so good at this that Anthropic would not release this model to the public yet but instead … Continue reading Mythos finds a curl vulnerability →
daniel.haxx.se (daniel.haxx.se)
@bagder great, so even the Linux Foundation are naming things after the ultimate evil of a famous franchise? (Final Fantasy in this instance.)
-
#Mythos finds a #curl vulnerability
yes, as in singular one.
Mythos finds a curl vulnerability
yes, as in singular one. Back in April 2026 Anthropic caused a lot of media noise when they concluded that their new AI model Mythos is dangerously good at finding security flaws in source code. Apparently Mythos was so good at this that Anthropic would not release this model to the public yet but instead … Continue reading Mythos finds a curl vulnerability →
daniel.haxx.se (daniel.haxx.se)
@bagder “On average, every single production source code line of curl has been written (and then rewritten) 4.14 times.”
curl is the ship of Theseus not once, not twice, but four times
-
@bagder How do you explain that Mythos found 271 bugs in Firefox, and counting, and only 1 in cURL. Is the Firefox code base 271 times larger?
