#Mythos finds a #curl vulnerability

[bagder@mastodon.social]

My personal conclusion can however not end up with anything else than that the big hype around this model so far was primarily marketing. I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing.

[madduci@mastodon.social]

@bagder thanks for this. It was really helpful to understand the hype around Mythos and also see that high quality in code matters a lot,especially if human driven

[aristot73@infosec.exchange]

@bagder spectacular result! Huge congratulations to the entire team! Made my day

[varpie@peculiar.florist]

@bagder That reinforces my suspicions that there was a breakthrough for security at the start of the year, and that the rest of the year will be more quiet.

[km@mastodon.babb.no]

@bagder from my talks with people who had been given access to mythos in their org, they say it does find things which current tools miss, but also overlooks cases which current tools catch. so, yeah, to me it is "mostly marketing" combined with general FUD

[arcayr@gts.mischief.expert]

@bagder that "the size of curl" section is honestly incredible numbers.

[km@mastodon.babb.no]

@bagder but i have not asked them about exploit capabilities, though. there i cannot comment, and there it could be significantly better caps

[viss@mastodon.social]

@bagder hah! i was right!

[alterelefant@mastodontech.de]

@bagder
At least it works. It would have been quite a disaster if it found zero.

[johnnythan@tuebingen.network]

@bagder Would it be a good idea to take an older version, where you already know you (as humans) found (and fixed) a certain number of vulnerabilities and see if AI can spot those correctly?

The Idee beeing to really have a quality test? ("For Science" ).

Or are the all trained on your latest version already and that would invalidate that test?

[bagder@mastodon.social]

@johnnythan I agree that would be an interesting challenge for someone with time and tokens to burn

[bagder@mastodon.social]

"Zero memory-safety vulnerabilities found."

[netresec@infosec.exchange]

@bagder LOL!

The report concluded it found five “Confirmed security vulnerabilities”. I think using the term confirmed is a little amusing when the AI says it confidently by itself. Yes, the AI thinks they are confirmed, but the curl security team has a slightly different take.

[ireneista@adhd.irenes.space]

@bagder yessssssssss. we guessed right on the poll

[quinn@social.circl.lu]

@bagder I suspect the question is, will it still be a worthwhile tool when the actual price to use the tool, not subsidized by anyone's war chest or VC, is revealed?

[dotmavriq@social.dotmavriq.life]

@bagder Yes. While I can't prove it, it tracks with A stealing the playbook of O who already said that they will likely pivot from B2C into B2B. One last fear mongering push and tons of directed compute at reputable projects and suddenly your marketing far surpasses that of any benchmark.

[maniacata@mastodon.social]

@bagder the power of rigorous software engineering

[eskett@mstdn.social]

@bagder not trying to buy into Anthropic's hype machine, but I wonder if curl is just a nonrepresentative code base. The average closed source / internal code base is probably worse in orders of magnitude when it comes to static checks, engineering principles, you name it.

I suspect Mythos will be useful in making poor software a bit more secure. That could have been done without AI of course.

[bagder@mastodon.social]

@eskett I do emphasize that it is good at finding flaws. And so are many other models. So yes, they will certainly find many flaws in source code going forward. Mythos and the others.

[david_chisnall@infosec.exchange]

@bagder

AI powered code analyzers are significantly better at finding security flaws and mistakes in source code than any traditional code analyzers did in the past

I’m not sure this follows from what you’ve said in the rest of the post. Static analysers and fuzzers also made it very easy for people to find vulnerabilities and typically found a lot when they were deployed for the first time. And both were a lot cheaper to run than something like Mythos.

They aren’t finding as many vulnerabilities now because projects that are critical for security are integrating them into their CI flows.

And this is what always happens with some new technique: valgrind, Coverity, sanitisers, fuzzers, and so on: they’re released, they find a load of bugs that existing techniques failed to find, people fix them, they get integrated into regular CI runs, and the kinds of bugs that those tools find never make it into the tree.

Syskaller, for example, has found a lot more bugs in the Linux kernel than any Anthropic tools. And that’s just one fuzzing tool.