Anthropic wrote a blog post about their LLM finding 0days in open source projects.
-
Anthropic wrote a blog post about their LLM finding 0days in open source projects. I reviewed the first vulnerability they reported, in OpenSC, and as far as I can tell the bug isn't even real? I think so anyway, otherwise I'll eat some humble pie.
I was looking at the code at the time (so a few weeks ago, before opensc removed the strcat call and replaced with strlcat):
The offending lines are around line 5145. According to anthropic, this is a buffer overflow found by their predictive text algorithm.
But it's interesting to read the code and to read the context of the code. 'fp' is a string guaranteed to be 64 bytes in length, because of line 5138. This checks the strlen(fp) and if it isn't 64, the code exits with an error.
So the math is: 'filename' is filled with a string that is at maximum 4096-64-2 bytes long == max 4030 bytes. This value (maximum 4030) is eventually passed to a strlcat() call inside of sc_get_cache_dir(), where it is used as the 'size' argument. This means that 'filename' has a maximum size of 4030 bytes when sc_get_cache_dir() exits.
Then the code strcat's one byte (for the '/'), onto the 'filename' buffer, then 64 more bytes (for the 'fp' string) onto 'filename' buffer, meaning the buffer cannot be overflowed because the maximum string length is now 4095 (the last byte being a null terminator).
So, anthropic's LLM probably spent 4 seconds 'finding' this 'vulnerability'. I spent about an hour looking at the code to debunk their finding (really 20 minutes looking at the code, followed by 40 minutes of hemming and hawing about whether I'm right and making sure there wasn't a signed size or some other gotcha, then being anxious about saying anything because golly human brains are funny machines).
I guess it's cool that opensc changed the calls to strlcat() now and that is probably a good thing, but this was never even a bug anyway so it didn't help anything today.
Unless the compiler magically omits instructions like the strlen() check some day. Which, if we really go down the path of compilers being generated by LLMs I guess that is a valid concern
.@reverseics I think the real question is whether or not you would have gone down this path and spent that amount of time if there wasn't the llm "pointing the way."
As part of the exercise, you've given the oh-day claim good faith and likely spent a bunch of time you might not have otherwise. This is probably because the anthropic peeps do not have a lot of experience doing vuln research, so they are pushing stuff that "qualifies" as vulns, but not from an experienced perspective. Not really, at least.
I dunno. To quote brighter minds than mine, the llms are "typicality" matchers not "correct" matchers. Will that work sometimes? For sure. Will it waste your time sometimes? Absolutely.
I guess it comes down to why we look for vulns. To find them, sure. But I'd argue because the hunt and the understanding and surprise is so damn rewarding. You're not surprised by what's typical. Apart from the fact that the typical keeps happening...
-
@reverseics I think the real question is whether or not you would have gone down this path and spent that amount of time if there wasn't the llm "pointing the way."
As part of the exercise, you've given the oh-day claim good faith and likely spent a bunch of time you might not have otherwise. This is probably because the anthropic peeps do not have a lot of experience doing vuln research, so they are pushing stuff that "qualifies" as vulns, but not from an experienced perspective. Not really, at least.
I dunno. To quote brighter minds than mine, the llms are "typicality" matchers not "correct" matchers. Will that work sometimes? For sure. Will it waste your time sometimes? Absolutely.
I guess it comes down to why we look for vulns. To find them, sure. But I'd argue because the hunt and the understanding and surprise is so damn rewarding. You're not surprised by what's typical. Apart from the fact that the typical keeps happening...
@rrdot @reverseics idk tptacek made some pretty bold claims on HN about the credibility of Anthropic’s vuln researchers so if yr right that they don’t have a lot of experience there’s gonna be some egg on some faces
-
@rrdot @reverseics idk tptacek made some pretty bold claims on HN about the credibility of Anthropic’s vuln researchers so if yr right that they don’t have a lot of experience there’s gonna be some egg on some faces
@bingbong @reverseics ya, I freely admit I have no knowledge of the experience or quality of the anthropic team. Which goes both ways. Experienced people make mistakes, and inexperienced people find gold.
-
@reverseics I think the real question is whether or not you would have gone down this path and spent that amount of time if there wasn't the llm "pointing the way."
As part of the exercise, you've given the oh-day claim good faith and likely spent a bunch of time you might not have otherwise. This is probably because the anthropic peeps do not have a lot of experience doing vuln research, so they are pushing stuff that "qualifies" as vulns, but not from an experienced perspective. Not really, at least.
I dunno. To quote brighter minds than mine, the llms are "typicality" matchers not "correct" matchers. Will that work sometimes? For sure. Will it waste your time sometimes? Absolutely.
I guess it comes down to why we look for vulns. To find them, sure. But I'd argue because the hunt and the understanding and surprise is so damn rewarding. You're not surprised by what's typical. Apart from the fact that the typical keeps happening...
@rrdot i would have done “find ./ -type f | xargs grep -B5 strcat” and then been convinced it was not exploitable after a few minutes, sure.
but i am old and slow ;).
-
Anthropic wrote a blog post about their LLM finding 0days in open source projects. I reviewed the first vulnerability they reported, in OpenSC, and as far as I can tell the bug isn't even real? I think so anyway, otherwise I'll eat some humble pie.
I was looking at the code at the time (so a few weeks ago, before opensc removed the strcat call and replaced with strlcat):
The offending lines are around line 5145. According to anthropic, this is a buffer overflow found by their predictive text algorithm.
But it's interesting to read the code and to read the context of the code. 'fp' is a string guaranteed to be 64 bytes in length, because of line 5138. This checks the strlen(fp) and if it isn't 64, the code exits with an error.
So the math is: 'filename' is filled with a string that is at maximum 4096-64-2 bytes long == max 4030 bytes. This value (maximum 4030) is eventually passed to a strlcat() call inside of sc_get_cache_dir(), where it is used as the 'size' argument. This means that 'filename' has a maximum size of 4030 bytes when sc_get_cache_dir() exits.
Then the code strcat's one byte (for the '/'), onto the 'filename' buffer, then 64 more bytes (for the 'fp' string) onto 'filename' buffer, meaning the buffer cannot be overflowed because the maximum string length is now 4095 (the last byte being a null terminator).
So, anthropic's LLM probably spent 4 seconds 'finding' this 'vulnerability'. I spent about an hour looking at the code to debunk their finding (really 20 minutes looking at the code, followed by 40 minutes of hemming and hawing about whether I'm right and making sure there wasn't a signed size or some other gotcha, then being anxious about saying anything because golly human brains are funny machines).
I guess it's cool that opensc changed the calls to strlcat() now and that is probably a good thing, but this was never even a bug anyway so it didn't help anything today.
Unless the compiler magically omits instructions like the strlen() check some day. Which, if we really go down the path of compilers being generated by LLMs I guess that is a valid concern
.@reverseics I saw some posts about people beating themselves off about how good claudecode is at finding vulns, and how they have so many vulns for OpenSSL and I'm here rolling my eyes, after seeing how so many open-source projects have had to remove bug bounties and tell AI users to fuck off because their slop generators are making shit up.
-
R relay@relay.infosec.exchange shared this topic
-
Anthropic wrote a blog post about their LLM finding 0days in open source projects. I reviewed the first vulnerability they reported, in OpenSC, and as far as I can tell the bug isn't even real? I think so anyway, otherwise I'll eat some humble pie.
I was looking at the code at the time (so a few weeks ago, before opensc removed the strcat call and replaced with strlcat):
The offending lines are around line 5145. According to anthropic, this is a buffer overflow found by their predictive text algorithm.
But it's interesting to read the code and to read the context of the code. 'fp' is a string guaranteed to be 64 bytes in length, because of line 5138. This checks the strlen(fp) and if it isn't 64, the code exits with an error.
So the math is: 'filename' is filled with a string that is at maximum 4096-64-2 bytes long == max 4030 bytes. This value (maximum 4030) is eventually passed to a strlcat() call inside of sc_get_cache_dir(), where it is used as the 'size' argument. This means that 'filename' has a maximum size of 4030 bytes when sc_get_cache_dir() exits.
Then the code strcat's one byte (for the '/'), onto the 'filename' buffer, then 64 more bytes (for the 'fp' string) onto 'filename' buffer, meaning the buffer cannot be overflowed because the maximum string length is now 4095 (the last byte being a null terminator).
So, anthropic's LLM probably spent 4 seconds 'finding' this 'vulnerability'. I spent about an hour looking at the code to debunk their finding (really 20 minutes looking at the code, followed by 40 minutes of hemming and hawing about whether I'm right and making sure there wasn't a signed size or some other gotcha, then being anxious about saying anything because golly human brains are funny machines).
I guess it's cool that opensc changed the calls to strlcat() now and that is probably a good thing, but this was never even a bug anyway so it didn't help anything today.
Unless the compiler magically omits instructions like the strlen() check some day. Which, if we really go down the path of compilers being generated by LLMs I guess that is a valid concern
.@reverseics i did a similar trapeze act with their last writeup about china doing the bad with a three tier framework of recon/exploit/exfil, and spent like 16 hours trying to reproduce it only to learn how fucked and broken it all is, and how its functionally equivalent to the ea lawsuit regarding lootboxes and reward systems that get children addicted to gambling
-
Anthropic wrote a blog post about their LLM finding 0days in open source projects. I reviewed the first vulnerability they reported, in OpenSC, and as far as I can tell the bug isn't even real? I think so anyway, otherwise I'll eat some humble pie.
I was looking at the code at the time (so a few weeks ago, before opensc removed the strcat call and replaced with strlcat):
The offending lines are around line 5145. According to anthropic, this is a buffer overflow found by their predictive text algorithm.
But it's interesting to read the code and to read the context of the code. 'fp' is a string guaranteed to be 64 bytes in length, because of line 5138. This checks the strlen(fp) and if it isn't 64, the code exits with an error.
So the math is: 'filename' is filled with a string that is at maximum 4096-64-2 bytes long == max 4030 bytes. This value (maximum 4030) is eventually passed to a strlcat() call inside of sc_get_cache_dir(), where it is used as the 'size' argument. This means that 'filename' has a maximum size of 4030 bytes when sc_get_cache_dir() exits.
Then the code strcat's one byte (for the '/'), onto the 'filename' buffer, then 64 more bytes (for the 'fp' string) onto 'filename' buffer, meaning the buffer cannot be overflowed because the maximum string length is now 4095 (the last byte being a null terminator).
So, anthropic's LLM probably spent 4 seconds 'finding' this 'vulnerability'. I spent about an hour looking at the code to debunk their finding (really 20 minutes looking at the code, followed by 40 minutes of hemming and hawing about whether I'm right and making sure there wasn't a signed size or some other gotcha, then being anxious about saying anything because golly human brains are funny machines).
I guess it's cool that opensc changed the calls to strlcat() now and that is probably a good thing, but this was never even a bug anyway so it didn't help anything today.
Unless the compiler magically omits instructions like the strlen() check some day. Which, if we really go down the path of compilers being generated by LLMs I guess that is a valid concern
.@reverseics what you are looking at is already the revision that has one of the fix for the issue applied. The Anthropic reported this issue before the https://github.com/OpenSC/OpenSC/pull/3558 was applied. The fix https://github.com/OpenSC/OpenSC/pull/3554 was mostly hardening as the strcat() is really a magnet for anything security relevant (regardless if it is an static analysis or AI scanner).
-
R relay@relay.infosec.exchange shared this topic
-
@reverseics what you are looking at is already the revision that has one of the fix for the issue applied. The Anthropic reported this issue before the https://github.com/OpenSC/OpenSC/pull/3558 was applied. The fix https://github.com/OpenSC/OpenSC/pull/3554 was mostly hardening as the strcat() is really a magnet for anything security relevant (regardless if it is an static analysis or AI scanner).
@jakuje Oh interesting! I missed that this strlen() check was an intermediate fix.
After staring at ASN.1 parser code for a few minutes I guess:
If the asn.1 tag type 0xf3 has a very large length field
and if the url string itself contains the first slash quite early (e.g. 'http:///AAAAAAA....'
then fp can point to a long buffer, say 10,000 bytes
then integer underflow makes the call to sc_get_cache_dir() somewhat unsafe (depends what kind of control we can expect to have over the directory config/environment variables, possibly a buffer overflow inside this function though)
and yes then 'filename' buffer could then be overflowed
interesting bug then. this requires physical access? assuming that this asn.1 tag is only read off of a card itself? i am guessing crl or some other feature...
-
@jakuje Oh interesting! I missed that this strlen() check was an intermediate fix.
After staring at ASN.1 parser code for a few minutes I guess:
If the asn.1 tag type 0xf3 has a very large length field
and if the url string itself contains the first slash quite early (e.g. 'http:///AAAAAAA....'
then fp can point to a long buffer, say 10,000 bytes
then integer underflow makes the call to sc_get_cache_dir() somewhat unsafe (depends what kind of control we can expect to have over the directory config/environment variables, possibly a buffer overflow inside this function though)
and yes then 'filename' buffer could then be overflowed
interesting bug then. this requires physical access? assuming that this asn.1 tag is only read off of a card itself? i am guessing crl or some other feature...
@reverseics correct. This data is read from objects on card/tokens. Therefore it requires a physical access to the system, plugging (malicious) stuff to USBs and having the system running opensc handling the desktop login (in most cases). Most of the issues we had over last couple of years fall into this attack vector category (and this was certainly not the last one).
This part of the PIV specs is something ancient logic hopefully not used by anyone these days (but code is there), that was saving the space on cards by allowing to have a certificate saved externally on disk.
-
Anthropic wrote a blog post about their LLM finding 0days in open source projects. I reviewed the first vulnerability they reported, in OpenSC, and as far as I can tell the bug isn't even real? I think so anyway, otherwise I'll eat some humble pie.
I was looking at the code at the time (so a few weeks ago, before opensc removed the strcat call and replaced with strlcat):
The offending lines are around line 5145. According to anthropic, this is a buffer overflow found by their predictive text algorithm.
But it's interesting to read the code and to read the context of the code. 'fp' is a string guaranteed to be 64 bytes in length, because of line 5138. This checks the strlen(fp) and if it isn't 64, the code exits with an error.
So the math is: 'filename' is filled with a string that is at maximum 4096-64-2 bytes long == max 4030 bytes. This value (maximum 4030) is eventually passed to a strlcat() call inside of sc_get_cache_dir(), where it is used as the 'size' argument. This means that 'filename' has a maximum size of 4030 bytes when sc_get_cache_dir() exits.
Then the code strcat's one byte (for the '/'), onto the 'filename' buffer, then 64 more bytes (for the 'fp' string) onto 'filename' buffer, meaning the buffer cannot be overflowed because the maximum string length is now 4095 (the last byte being a null terminator).
So, anthropic's LLM probably spent 4 seconds 'finding' this 'vulnerability'. I spent about an hour looking at the code to debunk their finding (really 20 minutes looking at the code, followed by 40 minutes of hemming and hawing about whether I'm right and making sure there wasn't a signed size or some other gotcha, then being anxious about saying anything because golly human brains are funny machines).
I guess it's cool that opensc changed the calls to strlcat() now and that is probably a good thing, but this was never even a bug anyway so it didn't help anything today.
Unless the compiler magically omits instructions like the strlen() check some day. Which, if we really go down the path of compilers being generated by LLMs I guess that is a valid concern
.@reverseics Humble pie has been eaten
🧐
