There's a lot of discourse on Twitter about people using LLMs to solve CTF challenges.
-
There's a lot of discourse on Twitter about people using LLMs to solve CTF challenges. I used to write CTF challenges in a past life, so I threw a couple of my hardest ones at it.
We're screwed.
At least with text-file style challenges ("source code provided" etc), Claude Opus solves them quickly. For the "simpler" of the two, it just very quickly ran through the steps to solve it. For the more "ridiculous" challenge, it took a long while, and in fact as I type this it's still burning tokens "verifying" the flag even though it very obviously found the flag and it knows it (it's leetspeak and it identified that and that it's plausible). LLMs are, indeed, still completely unintelligent, because no human would waste time verifying a flag and second-guessing itself when it very obviously is correct. (Also you could just run it...)
But that doesn't matter, because it found it.
The thing is, CTF challenges aren't about inventing the next great invention or having a rare spark of genius. CTF challenges are about learning things by doing. You're supposed to enjoy the process. The whole point of a well-designed CTF challenge is that anyone, given enough time and effort and self-improvement and learning, can solve it. The goal isn't actually to get the flag, otherwise you'd just ask another team for the flag (which is against the rules of course). The goal is to get the flag by yourself. If you ask an LLM to get the flag for you, you aren't doing that.
(Continued)
@lina I feel exactly the same about academia. I dunno about anyone else, but I genuinely enjoyed learning new things in school, building new ways of thinking and new skills. Sharpening my mind was the best feeling in the world back then, when I was young and still had all that neuroplasticity. It's mystifying that anyone would rather subsume their thinking to the chatbot than build new skills.
But then again, lots of young adults in the US are ushered into university and told that it's their only option, even if they don't particularly care for the subjects they're ostensibly supposed to be learning. I was privileged enough that I didn't have to work through college, for instance.
-
This is, quite frankly, the same problem LLM agents are causing in software engineering and such, just way worse. Because with CTFs, there is no "quality metric". Once you get the flag you get the flag. It doesn't matter if your approach was ridiculous or you completely misunderstood the problem or "winged it" in the worst way possible or the solver is a spaghetti ball of technical debt. It doesn't matter if Claude made a dozen reasoning errors in its chain that no human would (which it did). Every time it gets it wrong it just tries again, and it can try again orders of magnitude faster than a human, so it doesn't matter.
I don't have a solution for this. You can't ban LLMs, people will use them regardless. You could try interviewing teams one on one after the challenge to see if they actually have a coherent story and clearly did the work, but even then you could conceivably cheat using an LLM and then wait it out a bit to make the time spent plausible, study the reasoning chain, and convince someone that you did the work. It's like LLMs in academics, but much worse due to the time constraints and explicitly competitive nature of CTFs.
LLMs broke CTFs.
@lina I mean, yes, but I don't know if complete pessimism is warranted. It's definitely broken a lot of public CTFs but I think society will find a way, and maybe it's not even the worst thing.
Forever ago, when I was in uni, a few colleagues and I would do this thing every semester where we'd do one of the nominally individual projects together, ahead of time. Technically, it was cheating, but we did it specifically so we could go "off the rails" and try things that were not in the guide that our TAs handed out to us.
For instance, the "rite of passage" for every 3rd year student in my generation was a transformer design. It was something you'd work on, on and off, for the whole semester (we're talking big three-phase transformers for power distribution here so there was definitely *a lot* to work on).
They'd give us a sort of step-by-step guide to walk us through the whole process (start here, compute this quantity, check it against this standard table etc.) and you'd consult with the TAs along the semester. It was definitely interesting, if tedious at times, but tediousness was the lesser problem.
The bigger deal was that these guides weren't updated very often -- because the associated industrial standards don't get updated that often.
So what we did was that those of us who actually wanted to be there in the first place got together and we tried to experiment with various things not in the guide. Different isolation materials that we'd just read about, different cooling methods and so on. Not that we could show those to the TAs (can't blame them but most of them weren't very interested), and we didn't always have a lot of time or access to all the data we needed (we were students and had student budgets to contend with -- we couldn't buy standards, for example, and this was before libgen).
The cool thing about it was that it removed any kind of metrics pressure from this process. We weren't going to be ranked by anyone, there were no arsehole TAs to cater towards and no obtuse professors whose personal preferences in the formatting of our reports who had to be placated.
We also didn't have to show our results to anyone who wasn't primarily interested in mentoring us. We worked *really* quickly because we had graded assignments to finish first and clung to whatever had remained of our social lives by the third year of an engineering degree, so "deadlines" were super tight.
That quickly removed any incentive to cheat. When there was no way around it (tl;dr outdated guides sometimes didn't work in the context we used them, I have some fun stories about that) we totally cheated on the "real" assignments -- but never on these ones. This was technically cheating, too -- in the process of working out these differences we'd obviously discuss how we'd gone through the "real" assignments, share results and so on -- but since we all had different design targets (tl;dr same transformer designs but with different target parameters, so you couldn't just copy your colleague's work) it wasn't really a big deal.
With no incentive to cheat and nothing to get ahead of other than the limits of our own knowledge and engineering abilities, we often found ourselves doing things we normally wouldn't do for our regular assignments. We couldn't try things out in a lab, so if we doubted our analytical results for some particular configuration, we'd compare it against general EM field numerical simulations. If we didn't have a good simulation package for what we were after, we'd try to work out different analytical solutions for related quantities and see if we got similar results were similar.
We ended up learning a lot more than we did from the "real" assignments, mostly because our priorities were different. With real assignment, your main objective was inevitably to get a high grade, and keeping the TAs and the prof happy were as critical as tracking the decimal point.
Whereas with our "social" assignments, our main objectives were 1. to learn new things and 2. to get something that looked like a workable design that was an improvement over the "real" one in some aspect of our choice (better efficiency, reduced size, less coolant, whatever). If you "cheated" your way through it, #1 was obviously not happening and you were never really sure of #2, so no one was motivated to do it.
I think this is what we're eventually going to converge towards in other spaces, too: CTFs organised in smaller circles, with fewer external metrics and motivators, and an emphasis on cooperation, shifting the "competition" towards external factors than competition among teams/team members.
When CTF scores matter because they could potentially get you ahead in the race for an intership, every twenty year-old will eventually give in to cheating -- if only because it's the only way to stay in the race with people who do it because it's the only way they *can* do it. But if you take out the cheese, it's not much of a rat race anymore.
I'm old enough to have seen this happen to hackathons to some degree. At first, after hackathons had grown into their "competitive" form from their "let's hack shit together" roots, everyone was super enthusiastic and people every age jumped in. After a while, when prep became intensive enough that the only way to a prize was to implement 90% of what you meant to do beforehand (e.g. in a library) and then show up on the day of the hackathon and just piece the frontend together, everyone who was in it primarily for the thrill of focused building noped out.
Did that stop hackathons? Not at all, it just "split" things into:
- Corporate-funded hackathons which almost no one attends after they finish school -- where people rarely produce anything of value, and it's fine, because everyone understands that's not what they're there for. The "cheese" wasn't explicitly removed here, it's just at some point almost everyone recognised it's unattainable and the amount of hoops you have to jump through in order to attain it just isn't worth it when you're programming professionally
- "Real" hackathons, where people get together to work on a real project together, and the only competition is maybe the how-much-wasabi-you-can-eat-without-crying competition when everyone goes out for sushi the next day. -
There's a lot of discourse on Twitter about people using LLMs to solve CTF challenges. I used to write CTF challenges in a past life, so I threw a couple of my hardest ones at it.
We're screwed.
At least with text-file style challenges ("source code provided" etc), Claude Opus solves them quickly. For the "simpler" of the two, it just very quickly ran through the steps to solve it. For the more "ridiculous" challenge, it took a long while, and in fact as I type this it's still burning tokens "verifying" the flag even though it very obviously found the flag and it knows it (it's leetspeak and it identified that and that it's plausible). LLMs are, indeed, still completely unintelligent, because no human would waste time verifying a flag and second-guessing itself when it very obviously is correct. (Also you could just run it...)
But that doesn't matter, because it found it.
The thing is, CTF challenges aren't about inventing the next great invention or having a rare spark of genius. CTF challenges are about learning things by doing. You're supposed to enjoy the process. The whole point of a well-designed CTF challenge is that anyone, given enough time and effort and self-improvement and learning, can solve it. The goal isn't actually to get the flag, otherwise you'd just ask another team for the flag (which is against the rules of course). The goal is to get the flag by yourself. If you ask an LLM to get the flag for you, you aren't doing that.
(Continued)
@lina most of the CTF include 6-7 challenges to be solved in 4 hours.
Those CTFs expect you to know a typical set of forensync tools managed by an external guy/gal/entity which is somewhat known to be able to do it in time.
It stops being funny when you stop learning by doing and starts being a "kill'em all" competition.
-
This is, quite frankly, the same problem LLM agents are causing in software engineering and such, just way worse. Because with CTFs, there is no "quality metric". Once you get the flag you get the flag. It doesn't matter if your approach was ridiculous or you completely misunderstood the problem or "winged it" in the worst way possible or the solver is a spaghetti ball of technical debt. It doesn't matter if Claude made a dozen reasoning errors in its chain that no human would (which it did). Every time it gets it wrong it just tries again, and it can try again orders of magnitude faster than a human, so it doesn't matter.
I don't have a solution for this. You can't ban LLMs, people will use them regardless. You could try interviewing teams one on one after the challenge to see if they actually have a coherent story and clearly did the work, but even then you could conceivably cheat using an LLM and then wait it out a bit to make the time spent plausible, study the reasoning chain, and convince someone that you did the work. It's like LLMs in academics, but much worse due to the time constraints and explicitly competitive nature of CTFs.
LLMs broke CTFs.
@lina Programming competitions are banning LLMs, see e.g. https://info.atcoder.jp/entry/llm-rules-en. How are CTFs any different?
-
@lina most of the CTF include 6-7 challenges to be solved in 4 hours.
Those CTFs expect you to know a typical set of forensync tools managed by an external guy/gal/entity which is somewhat known to be able to do it in time.
It stops being funny when you stop learning by doing and starts being a "kill'em all" competition.
@echedellelr The ones I've worked on are less about "forensic tooling" and more about diverse (reverse)engineering challenges. They also usually run for a couple days and ~16 chals.
It evens out the playing field because pre-prepared tooling doesn't help you as much, since the challenges tend to be quite novel. I much prefer those to "write a ROP chain and exploit this service" or "crack this password" (not requiring an inordinate amount of compute, no more than 1hr of CPU time on a contemporary PC, is also a hard level design rule). There's usually one or two more typical infosec ones but they aren't the majority.
One example is a CrackMe challenge that was written in Verilog (implementing a custom CPU to run the actual crackme binary).
-
I might still do a monthly challenge or something in the future so people who want to have fun and learn can have fun and learn. That's still okay.
But CTFs as discrete competitions with winners are dead.
A CTF competition is basically gameified homework.
LLMs broke the game. Now all that's left is self study.
@lina thank you for this excellent thread
-
@echedellelr The ones I've worked on are less about "forensic tooling" and more about diverse (reverse)engineering challenges. They also usually run for a couple days and ~16 chals.
It evens out the playing field because pre-prepared tooling doesn't help you as much, since the challenges tend to be quite novel. I much prefer those to "write a ROP chain and exploit this service" or "crack this password" (not requiring an inordinate amount of compute, no more than 1hr of CPU time on a contemporary PC, is also a hard level design rule). There's usually one or two more typical infosec ones but they aren't the majority.
One example is a CrackMe challenge that was written in Verilog (implementing a custom CPU to run the actual crackme binary).
@lina at least the ones performed by the National Police or any other national agency here is like that, and are the typical ones I see.
Prolly is a cultural thing by country
-
@lina at least the ones performed by the National Police or any other national agency here is like that, and are the typical ones I see.
Prolly is a cultural thing by country
@echedellelr CTFs run by organizations focusing on infosec and offensive capability would necessarily lean that way. That's not the world I'm interested in. There are many CTFs not associated with such organizations with different themes.
-
There's a lot of discourse on Twitter about people using LLMs to solve CTF challenges. I used to write CTF challenges in a past life, so I threw a couple of my hardest ones at it.
We're screwed.
At least with text-file style challenges ("source code provided" etc), Claude Opus solves them quickly. For the "simpler" of the two, it just very quickly ran through the steps to solve it. For the more "ridiculous" challenge, it took a long while, and in fact as I type this it's still burning tokens "verifying" the flag even though it very obviously found the flag and it knows it (it's leetspeak and it identified that and that it's plausible). LLMs are, indeed, still completely unintelligent, because no human would waste time verifying a flag and second-guessing itself when it very obviously is correct. (Also you could just run it...)
But that doesn't matter, because it found it.
The thing is, CTF challenges aren't about inventing the next great invention or having a rare spark of genius. CTF challenges are about learning things by doing. You're supposed to enjoy the process. The whole point of a well-designed CTF challenge is that anyone, given enough time and effort and self-improvement and learning, can solve it. The goal isn't actually to get the flag, otherwise you'd just ask another team for the flag (which is against the rules of course). The goal is to get the flag by yourself. If you ask an LLM to get the flag for you, you aren't doing that.
(Continued)
@lina I view CTFs mostly as a way to learn and think that still exists. If you LLM the whole thing, you just hamper your own ability to learn. Competition wise for jeopardy it's got some challenges. I think it may be interesting to see if there is a shift to more Attack and Defense, King of the Hill or different structures where LLMs would still help but one shot single solutions aren't necessarily the best possible approach.
-
@lina Programming competitions are banning LLMs, see e.g. https://info.atcoder.jp/entry/llm-rules-en. How are CTFs any different?
@abacabadabacaba It's much easier to parallel construct a CTF solution than a programming challenge. CTF challenges are all about having a series of realizations that lead to the answer.
If you ban LLMs in a programming challenge, you could conceivably detect signs of LLM usage in the program in various ways (not perfectly, but you could try). A CTF challenge just has one output, the flag. Everyone finds the same flag. There is no way to tell how you did it. You'd have to introduce invasive monitoring like online tests, and even if you record people's screens, they could easily be running an LLM on another machine to have it come up with the "key points" to the solution which you just implement. You can't prove that someone didn't have some ideas on their own.
-
@echedellelr CTFs run by organizations focusing on infosec and offensive capability would necessarily lean that way. That's not the world I'm interested in. There are many CTFs not associated with such organizations with different themes.
@lina sorry, I replied because you were generalising and was not my experience here.
-
This is, quite frankly, the same problem LLM agents are causing in software engineering and such, just way worse. Because with CTFs, there is no "quality metric". Once you get the flag you get the flag. It doesn't matter if your approach was ridiculous or you completely misunderstood the problem or "winged it" in the worst way possible or the solver is a spaghetti ball of technical debt. It doesn't matter if Claude made a dozen reasoning errors in its chain that no human would (which it did). Every time it gets it wrong it just tries again, and it can try again orders of magnitude faster than a human, so it doesn't matter.
I don't have a solution for this. You can't ban LLMs, people will use them regardless. You could try interviewing teams one on one after the challenge to see if they actually have a coherent story and clearly did the work, but even then you could conceivably cheat using an LLM and then wait it out a bit to make the time spent plausible, study the reasoning chain, and convince someone that you did the work. It's like LLMs in academics, but much worse due to the time constraints and explicitly competitive nature of CTFs.
LLMs broke CTFs.
@lina perhaps having separate categories for LLMs allowed vs. banned would help with 90% of this problem? So ppl who want to use LLM can do so at their pleasure, and only ppl who actively want to cheat (hopefully very few) will try to use LLM in the banned category.
-
@lina Programming competitions are banning LLMs, see e.g. https://info.atcoder.jp/entry/llm-rules-en. How are CTFs any different?
@abacabadabacaba @lina mostly because the incentive to cheat for time is so high and it places an ever increasing burden on the organizers to develop LLM detection methods that are prohibitively cumbersome.
Rules without the ability to enforce them effectively are just guideposts for bad actors
-
@lina I view CTFs mostly as a way to learn and think that still exists. If you LLM the whole thing, you just hamper your own ability to learn. Competition wise for jeopardy it's got some challenges. I think it may be interesting to see if there is a shift to more Attack and Defense, King of the Hill or different structures where LLMs would still help but one shot single solutions aren't necessarily the best possible approach.
@nightwolf Yeah, I'm thinking mostly Jeopardy, which is the style I'm most familiar with. It just sucks to see that competition format completely break. I used to write a lot of challenges for that.
-
@lina perhaps having separate categories for LLMs allowed vs. banned would help with 90% of this problem? So ppl who want to use LLM can do so at their pleasure, and only ppl who actively want to cheat (hopefully very few) will try to use LLM in the banned category.
@YaLTeR I promise lots of people would cheat. These are competitions with rewards (bragging rights at minimum, but often cash prizes, swag, invitations to events, etc.)
-
There's a lot of discourse on Twitter about people using LLMs to solve CTF challenges. I used to write CTF challenges in a past life, so I threw a couple of my hardest ones at it.
We're screwed.
At least with text-file style challenges ("source code provided" etc), Claude Opus solves them quickly. For the "simpler" of the two, it just very quickly ran through the steps to solve it. For the more "ridiculous" challenge, it took a long while, and in fact as I type this it's still burning tokens "verifying" the flag even though it very obviously found the flag and it knows it (it's leetspeak and it identified that and that it's plausible). LLMs are, indeed, still completely unintelligent, because no human would waste time verifying a flag and second-guessing itself when it very obviously is correct. (Also you could just run it...)
But that doesn't matter, because it found it.
The thing is, CTF challenges aren't about inventing the next great invention or having a rare spark of genius. CTF challenges are about learning things by doing. You're supposed to enjoy the process. The whole point of a well-designed CTF challenge is that anyone, given enough time and effort and self-improvement and learning, can solve it. The goal isn't actually to get the flag, otherwise you'd just ask another team for the flag (which is against the rules of course). The goal is to get the flag by yourself. If you ask an LLM to get the flag for you, you aren't doing that.
(Continued)
@lina I do feel like this is about how you use the LLM. I often find my self throwing something into my local llama to give me an ELI5 or what do these flags on this command do in combination.
But as someone who has Designed CTFs and watched someone fling through it without learning a damn thing, it can be hard to keep the faith.
When I took physics all those years ago my professor made us learn a slide rule before a calculator. If you skip over the basics and use a machine to do it..when the machine breaks or is wrong, who is gonna fix it and how?
-
This is, quite frankly, the same problem LLM agents are causing in software engineering and such, just way worse. Because with CTFs, there is no "quality metric". Once you get the flag you get the flag. It doesn't matter if your approach was ridiculous or you completely misunderstood the problem or "winged it" in the worst way possible or the solver is a spaghetti ball of technical debt. It doesn't matter if Claude made a dozen reasoning errors in its chain that no human would (which it did). Every time it gets it wrong it just tries again, and it can try again orders of magnitude faster than a human, so it doesn't matter.
I don't have a solution for this. You can't ban LLMs, people will use them regardless. You could try interviewing teams one on one after the challenge to see if they actually have a coherent story and clearly did the work, but even then you could conceivably cheat using an LLM and then wait it out a bit to make the time spent plausible, study the reasoning chain, and convince someone that you did the work. It's like LLMs in academics, but much worse due to the time constraints and explicitly competitive nature of CTFs.
LLMs broke CTFs.
Asahi Linya (朝日りにゃ〜), I really hope that LLMs are a temporary phenomenon. Sure the local ones will remain even after the bubble finally bursts, but they're ridiculously bad, you do need millions of dollars worth of GPUs to get to that "it's still bad but it looks plausible" level of output quality.
-
There's a lot of discourse on Twitter about people using LLMs to solve CTF challenges. I used to write CTF challenges in a past life, so I threw a couple of my hardest ones at it.
We're screwed.
At least with text-file style challenges ("source code provided" etc), Claude Opus solves them quickly. For the "simpler" of the two, it just very quickly ran through the steps to solve it. For the more "ridiculous" challenge, it took a long while, and in fact as I type this it's still burning tokens "verifying" the flag even though it very obviously found the flag and it knows it (it's leetspeak and it identified that and that it's plausible). LLMs are, indeed, still completely unintelligent, because no human would waste time verifying a flag and second-guessing itself when it very obviously is correct. (Also you could just run it...)
But that doesn't matter, because it found it.
The thing is, CTF challenges aren't about inventing the next great invention or having a rare spark of genius. CTF challenges are about learning things by doing. You're supposed to enjoy the process. The whole point of a well-designed CTF challenge is that anyone, given enough time and effort and self-improvement and learning, can solve it. The goal isn't actually to get the flag, otherwise you'd just ask another team for the flag (which is against the rules of course). The goal is to get the flag by yourself. If you ask an LLM to get the flag for you, you aren't doing that.
(Continued)
@lina I'm a geek... I like AI and all of that... but if I understood your post right, it's "complaining" of the consequences of the capabilities it provides and that reminds me of MMORPGs a long time ago where you could marvel at the deeds of someone while now, it's just google the setup and technique and just reproduce it... basically, humans are becoming less the center of intelligence and more cows following a line
-
R relay@relay.an.exchange shared this topic
-
There's a lot of discourse on Twitter about people using LLMs to solve CTF challenges. I used to write CTF challenges in a past life, so I threw a couple of my hardest ones at it.
We're screwed.
At least with text-file style challenges ("source code provided" etc), Claude Opus solves them quickly. For the "simpler" of the two, it just very quickly ran through the steps to solve it. For the more "ridiculous" challenge, it took a long while, and in fact as I type this it's still burning tokens "verifying" the flag even though it very obviously found the flag and it knows it (it's leetspeak and it identified that and that it's plausible). LLMs are, indeed, still completely unintelligent, because no human would waste time verifying a flag and second-guessing itself when it very obviously is correct. (Also you could just run it...)
But that doesn't matter, because it found it.
The thing is, CTF challenges aren't about inventing the next great invention or having a rare spark of genius. CTF challenges are about learning things by doing. You're supposed to enjoy the process. The whole point of a well-designed CTF challenge is that anyone, given enough time and effort and self-improvement and learning, can solve it. The goal isn't actually to get the flag, otherwise you'd just ask another team for the flag (which is against the rules of course). The goal is to get the flag by yourself. If you ask an LLM to get the flag for you, you aren't doing that.
(Continued)
@lina they're engineering their self-incapacitation. Or decapacitation i suppose, because they get flush some skills down the drain to do that.
-
@abacabadabacaba It's much easier to parallel construct a CTF solution than a programming challenge. CTF challenges are all about having a series of realizations that lead to the answer.
If you ban LLMs in a programming challenge, you could conceivably detect signs of LLM usage in the program in various ways (not perfectly, but you could try). A CTF challenge just has one output, the flag. Everyone finds the same flag. There is no way to tell how you did it. You'd have to introduce invasive monitoring like online tests, and even if you record people's screens, they could easily be running an LLM on another machine to have it come up with the "key points" to the solution which you just implement. You can't prove that someone didn't have some ideas on their own.
@lina There are programming competitions where participants run their solutions locally and submit the output. But they are usually also required to submit the code, even though it is not automatically judged. If cheating is suspected, the judges may look into the code. Also there may be automated checks for plagiarism etc. CTFs could do the same. There really isn't a good reason to keep solutions secret after the challenge concludes, and published solutions can serve as a learning material for future challenges.
