legitbs_ctf

That’s actually untrue - they explicitly give permission to reverse engineer their software in the license.

Check restrictions in https://docs.binary.ninja/about/license/index.html

I don't think I can add much more than what was included in this very well thought out answer. I will double down on "Make it unambiguous and obvious when a challenge has been solved." ALL of our flags started with "The Flag Is: ", and that string was NEVER allowed to show up outside of the answer. It made some classes of file carving challenges not possible to do well (because they could be solved with grep...), but I'd argue that those aren't great challenges to begin with. When we did come up with challenges like that, it forced us to be more thoughtful about what made the challenge difficult through being clever, instead of just by brute force red herrings.

Hack.lu in 2018 had a pretty bad challenge that left a bad taste in a lot of players mouths. https://twitter.com/fluxfingers/status/1053279841578086406 describes some of the thinking in making that challenge. I disagree with things like that, though.

Also, meta, but on challenge design: if you're gating your challenges, keep in mind that your final challenges may need to be opened sooner than the last hours of your competition. There's an expectation that challenges get harder as the game goes on, but also, the teams are getting tired, and the time left in the game is going down. We viewed challenges that remained completely unsolved at game close as minor failures. I think one year we didn't let the final challenges open as there was too little time left at the end of the game, so we rolled them over to the following year.

I ran the DEF CON CTF from DEF CON 20 to DEF CON 25. Prior to that, I won a few.

I know order of the overflow is running a good game now. It’s different than what we did, but it’s a good direction, and I am hopeful it will continue to grow.

They (like we did) have a few selected prequalification events tbgat are generally pretty high quality. See https://www.oooverflow.io/dc-ctf-2019-quals/

PlaidCTF was an old favorite of ours as well. PPP has won several competitions, and seems to understand what makes a good challenge, what makes a challenge hard, and more importantly, what DOESNT make a good/interesting challenge.

Boston Key Party also runs a pretty good game. But a lot of them became Order, so who knows what will happen with them...

DARPA ran a CTF for autonomous systems called the Cyber Grand Challenge. It was neat, but I don’t know of any plans to do anything like it again. I believe they’re waiting to see what the community does to push the state of the art further. Source code for challenges and infrastructure are at https://github.com/cybergrandchallenge. There’s also a bunch of video at the darpatv YouTube channel.

We open sourced most of our challenges/frameworks after the fact. https://github.com/legitbs. This includes the compiler/emulator/manual for our 9-bit bytes custom architecture, clemency. That was probably our most ambitious year...

That’s all off the top of my head. If you care for more from a CTF has been, I’m happy to come back and answer more questions.