With the right ARIA labels, accessibility shouldn't be too much of a problem.

I've found out that uBlock Origin has a setting that can block web fonts by default. This also allows me to unblock them on broken websites, so that's better than using the web font browser setting at least.