If you have a field like and acl_id or some other context information on the data that is linked closely to a user's files. You can pass in the user's set of those field values to the vector database to pre-filter the results and do a permissions post check with a fairly relevant set.

The vector db definitely has to do some heavy lifting intersecting the say acl_id normal index with the nearest neighbors search but they do support it.