Yep, I made permissions tables per-entity too with simple `CRUD` access a bit like RWX (it was an API). It's definitely fast enough. I was handling ~5000 concurrent users with queries returning in the ns range. To be fair, I skipped RBAC checks if the entity was marked as 'public'.
Basically, create three tables: user, user_role, role_permission. Each user can have one or more role, and each role has one or more permissions. Permissions could be things like "view_admin_panel" or even granular like "view_project_with_id_5".
Then, you can create a row level security policy that does the right look up in these tables. I've not run this in a production system yet, but did successfully build out a proof of concept that worked. Performance seemed reasonable.