Maybe just having some groups, like Firefox's containers. It's less secure, but a simple setup would drastically reduce the number of programs with access to important information.

So just don't do that. The application needs an "Open" dialog already so you can tell it what files to open, just have that dialog be a process run by the system instead of the application, treat the permission as implicit since the user is the one selecting the file, and pass a handle back to the application.

E.g. Flatpak has the concept of "Portals" for that: https://github.com/flatpak/flatpak/wiki/Portals