FWIW Cloudflare offers lots of useful services beyond DDoS protection—that's just one of them. Once you use Cloudflare for one service, it's nice to have all of your domains going through their DNS at the very least even if you were to bypass their stack.

Aside from ideological preferences or a preference for some other service, I don't see what you gain by avoiding them.