Would be interesting to write an algo to generate defensive puzzles, seems like you could look at positions where the delta between the best move and the next best move is very high in terms of centipawn loss.

I'm thinking something like this: Evaluation of the position is even. All moves except one lead to high centipawn loss. Some of the moves look solid when evaluated superficially (too simplistic evaluation heuristics / not thinking enough ply ahead).

Lichess, chesstempo etc don't do a lot on the "superficial evaluation" - that's generally a hard problem to solve algorithmically.

Instead the main approach is to just mix new problems into the pool and let users try them. If there are no plausible alternatives, everyone gets the right move and the problem quickly becomes too low-rated to be shown to most players. If there are tempting alternatives and the correct solution is hard to calculate, it will become high-rated and only shown to stronger users.

You may be right about this. In some cases to turn an attack puzzle into a defense puzzle, I think it would be sufficient to use the same problem, but one turn earlier and switching sides. Then it may be possible to prevent the attack and the challenge becomes to spot and prevent it.

It’s not necessarily a problem, but that kind of challenge would likely have many correct answers and one wrong one. It may be less clear when you get it right why that was the case.