I think you can say “no religion” though and then not pay your tithing taxes, no? There’s no requirement that if you are Jewish you register as such (but then, your synagogue misses out on money).

Hmm I've read stories of people getting in trouble for being members of a church and not paying their church tax.

Yes exactly, you can easily opt out from the church tax (of course, ideal it should have been opt in).