When you add a friend just append new pubkey encrypted symmetric key for each asset. Friend removal is even more expensive and requires the client side to reencrypt everything :(
I'm sure there are better cryptographic constructs out there with multi key encryption that can alleviate some of the insertion and removal costs.
You could do it with decent trade-offs for the content provider as well. You could obfuscate everyone else's face but yours and allow them to get access to that image and then just encrypt the map for your friends. Of course then it's all about just creating enough mock accounts to recover the full images throughout the graph. Or just take custody of a user's keys in exchange for features.
Still though, it's marginally more expensive to build a mass surveillance app!
If you accept that people do have this right the logical conclusion is then you are infringing and causing harm to them by taking their photo and sharing it with Facebook. If you don't think people should have that right, I am not convinced there is any infringement going on and this creepy database is just the inevitable end result.