Your comment about the logging is spot on as well. I think dnsmasq and I think "small flash device where the logs will never get stored anywhere and no one can or will view them anyway". As you can probably tell I've had issues getting meaningful logging and debug out of dnsmasq as well...
I don't envy your situation in the slightest!
Note to mods - I'm not complaining about downvotes. I'm using the downvotes as a sign there's little understanding on HN of network ops at scale. People may love dnsmasq on a home router (I do) but plenty of network admins like OP and I have watched it fall over in other scenarios.
dnsmasq has some really nice features that kea doesn't have. just finding a single blade in the datacenter, means a naming system, and it was easier with dnsmasq, and the config was easier to create. We ended up having to use ansible to generate the kea config file because we also keep machines subnet separated based on physical location.
HN has all sorts of people, I never try to quantify it. We don't know what we don't know.
Now that I've studied the source code of dnsmasq a bit more because of the issues we had, I have to agree with you. We had run it with about 3k blades and it was fine. It was fine at about 9k blades too... but once we hit around 10k, it just fell over.
The frustrating part was that it just didn't log what was up. We used netdata and discovered the UDP queues were filling up. Tried mucking with tuning the sysctls and that didn't work either.
Switched midstream to Kea, which was a giant pain because we had a hard time dealing with duplicate IP addresses (imagine rebooting 10k+ PXE booted blades during a UDP storm).