(a bit late to the party)
Hi Fabian, At Cerbos we had to handle this issue as well and wrote a blog post about [1] how we can convert a policy into a generic AST that you can use in your data filtering logic on your data storage. This way you can empower your data storage queries to only fetch the relevant records.
To showcase how this works, we have released a Prisma ORM plugin [2] that converts our AST to Prisma filters - you can see a demo on Prisma’s YouTube channel[3]
[1]: https://cerbos.dev/blog/filtering-data-using-authorization-l...
[2]: https://cerbos.dev/blog/fully-featured-authorization-for-you...
Generally, this problem is called ACL-Filtering[0][1] and can be done in two ways: "pre-filter" and "post-filter". Sometimes you might even have to do both.
If you decide to use a service/database for permissions, similar to SpiceDB[2], there are often specialized APIs for directly listing the entities a subject has access to in various ways. You can take these results and feed them into a database query to select only the authorized content. This doesn't have to just be a list of IDs, but can also be datastructures like bitmaps, effectively providing your database with a custom index for your query. Systems that implement some of the novel parts of the Zanzibar paper[3] can also enable you to cache these values in your database until your application performs an operation that invalidates the results.
Filtering once you've queried all possible results from your database can also be more performant than you'd think, because you can amortize performance by lazy loading and performing permission checks in parallel. We have some pretty large systems that are purely using this strategy. The code for filtering can also be made extremely elegant because it can be hidden behind the iterator interface in whatever programming language you're using.
[0]: https://docs.authzed.com/reference/glossary#acl-filtering
[1]: https://authzed.com/blog/acl-filtering-in-authzed/
Apologies for the delayed response.
Instead, we went with Casbin (microsoft research) because it can push to your DB (including multi-tenant-sharing if you scale) and a legit modern policy engine - A(R)BAC, ACL, etc. Definitely warts, but pretty close to what I'd hope architecturally, meaning a clear path to prettier UIs, plugging into automatic SMT solvers/verifiers, etc, and till then, pretty easy from whatever backend lang + SQL DB you use.
Long-term, stuff like row-level security in your SQL / metadata store makes a lot of sense (people pointing that out in the thread below), but RLS is still awkward in practice for even basic enterprise RBAC/ACL policies. Until then, Casbin-style architectures are the equivalent of a flexible external policy decision point with the actual compute still being pushdown to wherever you want, including the DB: win/win.
I wish the VC money went this way instead, but I see why $ goes to simpler "google for everyone else" pitches, so here we are :(
We chose a more pragmatic approach with Aserto. We believe that most authorization problems can be expressed as a combination of rules and data. A system that is 100% rules or 100% data isn't pragmatic.
As others have mentioned, authorization often requires both a single method to authorize "can the user perform this action on this resource" as well as more flexible versions like "what are all the resources this user can perform this action on". That's one part of why authorization is hard, I wrote an article on this a little while ago [1]. At Oso (disclaimer: I'm the CTO), we solve this by turning authorization logic into SQL [2].
Supporting those APIs in a generic service definitely turns up the difficulty level -- you no longer have a single database to query.
The thing is, if you have multiple services, you might already be in that situation. If I need to query another service to ,e.g., find what projects a user belongs to, and then need to go combine that with data in my database, I'm going to need to start worrying about how to do that efficiently. In those situations starting to centralize that data + logic starts making sense -- we talk about this in [3]. So now there's a bunch of companies with different takes on how to best solve this, including Oso.
---
I feel bad about self-promoting so many links here... but we're passionate about this subject so we've been writing a lot about it!
[1]: https://www.osohq.com/post/why-authorization-is-hard
[2]: https://www.osohq.com/post/authorization-logic-into-sql
[3]: https://www.osohq.com/post/microservices-authorization-patte...
The biggest gripe we have, is lack of support for SQLAlchemy 2.0 style queries and lack of support for DB & Python enums as role names
We had a chat with Graham who told us about your upcoming cloud offering. Looking forward to that!
More seriously, I agree that there are a number of challenges, and different use-cases tend to require different approaches. Over time, we think there will be a set of common patterns that emerge, which will help the industry move towards a more consistent set of authorization experiences. And that will be great for everyone.
1. Gating the operation (whether it's retrieving a single resource, or creating / updating / deleting a resource). In this scenario, the application does need to call the authorizer before performing the operation on the resource, but the relationship between the authorizer and the application is at "arms length".
2. Filtering a list of resources. In this scenario, the authorization system can help you by running what in OPA is known as a "partial evaluation", which returns an abstract syntax tree. Your code would then walk that tree and insert the proper where clauses (if you're talking to a SQL back-end). As you mentioned, by necessity, your application needs to work more closely with the authorizer.
Using your DB's row-level security for data filtering is definitely complementary to API authorization.
[0] https://www.aserto.com/blog/modern-authorization-requires-de...
If I request a single resource, of course this can work if I ask a second API on whether the request is allowed or not. But if I query a database for a list of items, to add access control I need to modify the database query. I can't just filter after the fact, it's too easy to cause pathological performance issues there e.g. if the user has only access to a very small subset of a large list of results. How does this work with a separate access control API that can't directly modify the database query?