Agreed, I have come to believe strongly in the value of Fizz Buzz and other simple tests. Many candidates have strong resumes but large companies destroy their souls and skills by sticking them for 8+ hours a day in rubbish meetings. Go without coding for a few months of this and your ability to Fizz Buzz will be compromised. Most smaller companies are looking for active, hands-on developers and Fizz Buzz is a quick check to ensure that the candidate spends a substantial part of the day writing code and designing software.

I don't know why knowing modular arithmetic is a requirement for knowing the syntax and how to code in a specific language/on a specific platform?

Yes. Always have an easy problem in analysis & coding, like constructing a simple Boolean predicate from some comparisons. You will thank yourself later when you don't hire the fantastic-looking candidate who had absolutely no idea how to work the problem.