On the access point side, if you want more granular control to handle congestion, Ubiquiti Unifi and their various mgmt platforms work well. I can't recommend their Security Gateway Appliance (frequent reboots), but the backhauls, APs and CPEs are solid. That will allow you to visualize utilization using a gui, and adjust limits per device/group easily. If you already have the APs, then a gateway device may be better.
I've found over the years, and in the datacenter, that running pfsense on server hardware is very reliable. You can run on a single board computer kit but the CPU doesn't hold up to heavier traffic/vpns etc, worse remote management.
Also, is there a way to charge for traffic beyond a certain quota? There’s a product called Sputnik with some firmware you can download for various router brands which lets you run a hotspot/captive portal with billing
Is there a simple prosumer solution for this? My current path is leading me towards openwrt on a beefy router with some custom firewall rules.