I didn't talk about dynamic analysis, but it has a bunch of different tradeoffs.

In general, dynamic analysis (valgrind, sanitisers, and so on) has a very low false positive rate because every code path that it sees really is a code path that is reachable in a program run. At the same time, it also has a higher false negative rate. Most security vulnerabilities come from a case where an attacker provides some unusual input. Dynamic analysis tools will often only ever see the behaviour of the program with expected (not necessarily correct) inputs.

The combination of fuzzing (provide a load of different unexpected inputs, with some feedback to try to find corner cases in execution) works nicely, but also hits combinatorial problems. Even if you have 100% line coverage, some bugs manifest only if lines are hit in a specific order, or even if two threads do the same thing at the same time. These approaches can never tell you that bugs aren't present, only that they are.

TL;DR: Dynamic analysis can be sound but not complete. Static analysis can be complete but not sound.

"The bug I found a little while ago in some MISRA C code was of that form: their analyser had found it, someone had determined it was not a bug, and they were wrong."

It's not just static analysis. Valgrind memcheck has a low false positive rate. For some reason many people seem to believe that if their program does not crash every time on their machine then it must be infallibly and absolutely correct. They might then report a "bug" or seek confirmation of the "false positive" that they have found.

The original Coverity paper found over 300 bugs, most of which had security implications. Static analysis has been great at finding exploitable vulnerabilities for a long time. This is a new approach to doing static analysis.

The biggest problem is always the false positive rate. If you run a tool and it finds a load of vulnerabilities, that’s great. Except you run the same tool and it also finds a load of things that look like vulnerabilities, but aren’t. So now you have to triage them and that takes effort. You also need to add annotations to silence the ones that aren’t real. With deterministic analysers, you can often provide some extra information (e.g. parameter attributes) that allow this information to be tracked across an analysis boundary. BCMC has a lot of these. But with a probabilistic tool, these may or may not work. So you’re left with just slapping on an annotation that says ‘ignore the warning here’. The bug I found a little while ago in some MISRA C code was of that form: their analyser had found it, someone had determined it was not a bug, and they were wrong.

For a defender, if you spend too much time looking at and discounting false positives, you can improve code quality better with something else. I’ve only looked at a few of the bugs Claude reported, but one was a missing bounds check that wasn’t actually a vulnerability because the bounds were checked in the caller. Its fix made things slower, but not less exploitable. A good static analyser would have had a tool for annotating the function parameter to say ‘this is always at least n bytes’ and then checked that callers did this check. Claude has nothing like this because it doesn’t actually have a model of how code executes, it just has a set of probabilities for what exploitable code looks like. Unfortunately (and this is one of the problems with C), correct and vulnerable code can look exactly the same with different call stacks.

The second problem is the asymmetry. To be secure, you need to investigate and fix all of the vulnerabilities that tools can find. For an attacker, you just need one vulnerability. The ROI for attackers is much higher. Imagine a tool with a 90% false positive rate that finds 1,000 vulnerability-shaped objects. An attacker who triages 6-7 of them has around a 50% chance of finding an attack that they can use. A defender who does the same amount of work has a 50% chance of reducing the number of vulnerabilities discoverable by attackers using this or similar tools by 1%.

This is why I build things that deterministically prevent classes of vulnerabilities from being exploitable.

Statistical models are inherently unreliable, but, "...remember we have only to be lucky once, you will have to be lucky always. "(*)

But Anthropic will happily sell you all the attempts you want to see if it can find the bug somebody else will use to rock your shit before they do the same. Better buy some tokens!

(* - The IRA reminding Thacher of a fundamental advantage of asymetrical warfare - https://en.wikipedia.org/wiki/Brighton_hotel_bombing )

Example (possibly overblown but plausible): https://codewall.ai/blog/how-we-hacked-mckinseys-ai-platform

In short - the advantage belongs to the defenders for now.

Caveat - organizations that haven't implemented security basics (user scoping, vulnerability management, asset inventories, etc) are at a significant disadvantage.

Apologies for the lengthy reply.

That said, attackers may also have to evaluate that same large number of possible attack vectors so it may come out a wash. It's probably easier to fix a bug than to exploit it.

I think it's going to make AI-bros even more obnoxious when posting to bug trackers.