Hey cybersecurity geeks-- so it seems like anthropic now has really good exploit detection ability.
-
Hey cybersecurity geeks-- so it seems like anthropic now has really good exploit detection ability. Do you think this makes offense or defense harder? Like, seems like everyone might have to go through a battery of automated checks before deploying stuff into the world.
@ZachWeinersmith We already had fuzz testing and it didn't use $$$ RAM not to mention burn up the planet
-
Hey cybersecurity geeks-- so it seems like anthropic now has really good exploit detection ability. Do you think this makes offense or defense harder? Like, seems like everyone might have to go through a battery of automated checks before deploying stuff into the world.
@ZachWeinersmith makes defense harder, at least for now - LLM penetration testing is finding new attack vectors quickly but fixing them is hard and slow.
Example (possibly overblown but plausible): https://codewall.ai/blog/how-we-hacked-mckinseys-ai-platform
-
Hey cybersecurity geeks-- so it seems like anthropic now has really good exploit detection ability. Do you think this makes offense or defense harder? Like, seems like everyone might have to go through a battery of automated checks before deploying stuff into the world.
@ZachWeinersmith All we've seen so far is a press release. With "AI" those should be taken cum grano salis.
-
Hey cybersecurity geeks-- so it seems like anthropic now has really good exploit detection ability. Do you think this makes offense or defense harder? Like, seems like everyone might have to go through a battery of automated checks before deploying stuff into the world.
@ZachWeinersmith I think it is going to make defense harder in the long run. A lot of these automated checks existed before but were well known to be hindsight-oriented (won’t find truly novel things, just the simplest variations of past mistakes) and imprecise (makes mistakes in both directions). These new ones give more illusions of novelty and precision. Devs will rely on them as their only investigation, not one in a stable full.
-
@ZachWeinersmith I think it is going to make defense harder in the long run. A lot of these automated checks existed before but were well known to be hindsight-oriented (won’t find truly novel things, just the simplest variations of past mistakes) and imprecise (makes mistakes in both directions). These new ones give more illusions of novelty and precision. Devs will rely on them as their only investigation, not one in a stable full.
@ZachWeinersmith But these new tools are also going to be just expensive enough that not just are devs going to want to solely rely on them, business people are going to want to empty the rest of the stables, including the jobs of the people good at finding and solving the novel problems.
-
@ZachWeinersmith But these new tools are also going to be just expensive enough that not just are devs going to want to solely rely on them, business people are going to want to empty the rest of the stables, including the jobs of the people good at finding and solving the novel problems.
@ZachWeinersmith Meanwhile in open source, this is just going to further tax and then kill bug bounty programs from amateurs willing to pay for these LLM tools in the hopes of easy cash. PRs get harder to prioritize without real people investing time in them. Any PRs that get ignored hard enough become free attack writeups for the less scrupulous.
-
Hey cybersecurity geeks-- so it seems like anthropic now has really good exploit detection ability. Do you think this makes offense or defense harder? Like, seems like everyone might have to go through a battery of automated checks before deploying stuff into the world.
Statistical models are inherently unreliable, but, "...remember we have only to be lucky once, you will have to be lucky always. "(*)
But Anthropic will happily sell you all the attempts you want to see if it can find the bug somebody else will use to rock your shit before they do the same. Better buy some tokens!
(* - The IRA reminding Thacher of a fundamental advantage of asymetrical warfare - https://en.wikipedia.org/wiki/Brighton_hotel_bombing )
-
Hey cybersecurity geeks-- so it seems like anthropic now has really good exploit detection ability. Do you think this makes offense or defense harder? Like, seems like everyone might have to go through a battery of automated checks before deploying stuff into the world.
@ZachWeinersmith
First of all, it's going to make Anthropic richer, because both sides will use it more.
But hear me out: If their detection ability really was so good - why did the #ClaudeCodeLeak immediately result in several high-profile vulnerabilities found? E.g.
https://phoenix.security/claude-code-leak-to-vulnerability-three-cves-in-claude-code-cli-and-the-chain-that-connects-them/
#InsecureAI #Infosec #ClaudeCode -
Hey cybersecurity geeks-- so it seems like anthropic now has really good exploit detection ability. Do you think this makes offense or defense harder? Like, seems like everyone might have to go through a battery of automated checks before deploying stuff into the world.
The original Coverity paper found over 300 bugs, most of which had security implications. Static analysis has been great at finding exploitable vulnerabilities for a long time. This is a new approach to doing static analysis.
The biggest problem is always the false positive rate. If you run a tool and it finds a load of vulnerabilities, that’s great. Except you run the same tool and it also finds a load of things that look like vulnerabilities, but aren’t. So now you have to triage them and that takes effort. You also need to add annotations to silence the ones that aren’t real. With deterministic analysers, you can often provide some extra information (e.g. parameter attributes) that allow this information to be tracked across an analysis boundary. BCMC has a lot of these. But with a probabilistic tool, these may or may not work. So you’re left with just slapping on an annotation that says ‘ignore the warning here’. The bug I found a little while ago in some MISRA C code was of that form: their analyser had found it, someone had determined it was not a bug, and they were wrong.
For a defender, if you spend too much time looking at and discounting false positives, you can improve code quality better with something else. I’ve only looked at a few of the bugs Claude reported, but one was a missing bounds check that wasn’t actually a vulnerability because the bounds were checked in the caller. Its fix made things slower, but not less exploitable. A good static analyser would have had a tool for annotating the function parameter to say ‘this is always at least n bytes’ and then checked that callers did this check. Claude has nothing like this because it doesn’t actually have a model of how code executes, it just has a set of probabilities for what exploitable code looks like. Unfortunately (and this is one of the problems with C), correct and vulnerable code can look exactly the same with different call stacks.
The second problem is the asymmetry. To be secure, you need to investigate and fix all of the vulnerabilities that tools can find. For an attacker, you just need one vulnerability. The ROI for attackers is much higher. Imagine a tool with a 90% false positive rate that finds 1,000 vulnerability-shaped objects. An attacker who triages 6-7 of them has around a 50% chance of finding an attack that they can use. A defender who does the same amount of work has a 50% chance of reducing the number of vulnerabilities discoverable by attackers using this or similar tools by 1%.
This is why I build things that deterministically prevent classes of vulnerabilities from being exploitable.
-
The original Coverity paper found over 300 bugs, most of which had security implications. Static analysis has been great at finding exploitable vulnerabilities for a long time. This is a new approach to doing static analysis.
The biggest problem is always the false positive rate. If you run a tool and it finds a load of vulnerabilities, that’s great. Except you run the same tool and it also finds a load of things that look like vulnerabilities, but aren’t. So now you have to triage them and that takes effort. You also need to add annotations to silence the ones that aren’t real. With deterministic analysers, you can often provide some extra information (e.g. parameter attributes) that allow this information to be tracked across an analysis boundary. BCMC has a lot of these. But with a probabilistic tool, these may or may not work. So you’re left with just slapping on an annotation that says ‘ignore the warning here’. The bug I found a little while ago in some MISRA C code was of that form: their analyser had found it, someone had determined it was not a bug, and they were wrong.
For a defender, if you spend too much time looking at and discounting false positives, you can improve code quality better with something else. I’ve only looked at a few of the bugs Claude reported, but one was a missing bounds check that wasn’t actually a vulnerability because the bounds were checked in the caller. Its fix made things slower, but not less exploitable. A good static analyser would have had a tool for annotating the function parameter to say ‘this is always at least n bytes’ and then checked that callers did this check. Claude has nothing like this because it doesn’t actually have a model of how code executes, it just has a set of probabilities for what exploitable code looks like. Unfortunately (and this is one of the problems with C), correct and vulnerable code can look exactly the same with different call stacks.
The second problem is the asymmetry. To be secure, you need to investigate and fix all of the vulnerabilities that tools can find. For an attacker, you just need one vulnerability. The ROI for attackers is much higher. Imagine a tool with a 90% false positive rate that finds 1,000 vulnerability-shaped objects. An attacker who triages 6-7 of them has around a 50% chance of finding an attack that they can use. A defender who does the same amount of work has a 50% chance of reducing the number of vulnerabilities discoverable by attackers using this or similar tools by 1%.
This is why I build things that deterministically prevent classes of vulnerabilities from being exploitable.
@david_chisnall @ZachWeinersmith
"The bug I found a little while ago in some MISRA C code was of that form: their analyser had found it, someone had determined it was not a bug, and they were wrong."
It's not just static analysis. Valgrind memcheck has a low false positive rate. For some reason many people seem to believe that if their program does not crash every time on their machine then it must be infallibly and absolutely correct. They might then report a "bug" or seek confirmation of the "false positive" that they have found.
-
@david_chisnall @ZachWeinersmith
"The bug I found a little while ago in some MISRA C code was of that form: their analyser had found it, someone had determined it was not a bug, and they were wrong."
It's not just static analysis. Valgrind memcheck has a low false positive rate. For some reason many people seem to believe that if their program does not crash every time on their machine then it must be infallibly and absolutely correct. They might then report a "bug" or seek confirmation of the "false positive" that they have found.
I didn't talk about dynamic analysis, but it has a bunch of different tradeoffs.
In general, dynamic analysis (valgrind, sanitisers, and so on) has a very low false positive rate because every code path that it sees really is a code path that is reachable in a program run. At the same time, it also has a higher false negative rate. Most security vulnerabilities come from a case where an attacker provides some unusual input. Dynamic analysis tools will often only ever see the behaviour of the program with expected (not necessarily correct) inputs.
The combination of fuzzing (provide a load of different unexpected inputs, with some feedback to try to find corner cases in execution) works nicely, but also hits combinatorial problems. Even if you have 100% line coverage, some bugs manifest only if lines are hit in a specific order, or even if two threads do the same thing at the same time. These approaches can never tell you that bugs aren't present, only that they are.
TL;DR: Dynamic analysis can be sound but not complete. Static analysis can be complete but not sound.
-
I didn't talk about dynamic analysis, but it has a bunch of different tradeoffs.
In general, dynamic analysis (valgrind, sanitisers, and so on) has a very low false positive rate because every code path that it sees really is a code path that is reachable in a program run. At the same time, it also has a higher false negative rate. Most security vulnerabilities come from a case where an attacker provides some unusual input. Dynamic analysis tools will often only ever see the behaviour of the program with expected (not necessarily correct) inputs.
The combination of fuzzing (provide a load of different unexpected inputs, with some feedback to try to find corner cases in execution) works nicely, but also hits combinatorial problems. Even if you have 100% line coverage, some bugs manifest only if lines are hit in a specific order, or even if two threads do the same thing at the same time. These approaches can never tell you that bugs aren't present, only that they are.
TL;DR: Dynamic analysis can be sound but not complete. Static analysis can be complete but not sound.
@david_chisnall I just squint and say "Gödel's incompleteness theorems" whenever thinking about these
-
Hey cybersecurity geeks-- so it seems like anthropic now has really good exploit detection ability. Do you think this makes offense or defense harder? Like, seems like everyone might have to go through a battery of automated checks before deploying stuff into the world.
@ZachWeinersmith Depends, it will favour defense in the case of high-value targets with a well-resourced, competent security-team. E.g. Apple defending their iPhones should be able to address issues better than today. Targets which today are secure because nobody has yet gotten around to looking are going to be in trouble...
-
R relay@relay.infosec.exchange shared this topic
