Somewhat similar to what you describe, the "Access Contacts" permission usually present in mobile operating systems essentially leaks the contact information of everyone stored in ones contact, if permitted.
The names and associated phone number can be used to build advertising/tracking profiles by apps.
The problem is that even if someone does not use those apps and instead chooses privacy-respecting alternatives, it only takes another person who does use these apps to have their information saved in their contacts.
I would imagine it to be quite similar with these photos and also with the facebook case.
It seems unethical to put other people who might not be on facebook and have not read the terms of service or privacy policy through these algorithms.
I think the contacts example illustrates how hard of a problem this is, I definitely don't have a viable solution. Even if you ask someone for permission, they might change their mind or not fully understand.
As I was walking around I realized that so many other people were taking photos where I was in the background that it felt like Facebook or Google knew I was there even though no one else did.
According to documentation, Google mentiones that face groups are deleted when you turn the feature off: https://support.google.com/photos/answer/6128838?hl=en
(And if they're not, winning a lawsuit against Google should be trivial.)