LLMs and CTFs

Opus 5.5 solves a CTF challenge I made, and does so in less than three minutes using following prompt:

The first attempt gets close to the solution but uses the wrong flag encoding, after which it suffices to reply:

…to then obtain the correct flag:

Some unstructured thoughts:

• How do CTF challenge designers balance the complexity of their challenges against solvability by coding agents, especially when trying to create original challenges for beginners?
• Do we even need to avoid agent usage? Is it patronising to participants to disallow these tools, even if the participants get no value out of “solving” the challenge (as their tool solves it for them)?
• Can you avoid usage of coding agents through challenge design?
  • I expected it to take more time for the challenge above to be solved, as the encoding of the flag is spoken out loud in the audio file, but “guessing” encodings until encountering something that made the flag “make sense” also worked out for Opus.
• Should you require participants to use a sandbox with a fixed set of standard tools? But if they have access to the internet to look stuff up, chatbots can be used as well, so I guess this would be useless.

These points all depend on what you believe the goal of a CTF is. My favourite challenges (mostly in the misc category) allowed me to solve some fun puzzles, to learn something new, to think outside the box without requiring lots of background knowledge, to try out some new tools, and most of all: to have fun, by myself and with friends.

If learning is the goal of the CTF, then coding agents take away the struggle (and this struggle, at least in my case, is what allows for learning). If winning is the goal, then it would make no sense at all not to use every available tool. I guess you can also translate these points to education (learning vs. obtaining a degree as soon as possible) and exams.

Please reach out if you have some thoughts on this!