I've been using a .rsync-filter file for something like what you mean for ages for my homedirs backups. It's a bit tricky probably to make it right the first time but once it's there it just works.

https://manpages.debian.org/bookworm/rsync/rsync.1.en.html#f...