Even with a small user base; if a user flags a photo for deletion -- it should get deleted. Period.
When a user presses the delete button, we...
1. Delete it from the public facing servers that we control. 2. Put a request in the queue so the backups, and thumbnails will be deleted. 3. Pray that the CDN will purge the picture eventually.
#3 is a big deal... we don't control the CDNs cache options, and can't push a 'delete' upstream so the picture might be gone 2 seconds after the deletion is completed on our end, or it might take 2 weeks.
I'd thought Facebook would have been able to engineer or negotiate better options than we do, but apparently not.
I don't really know how their photo system works - maybe it is a trivial thing to purge files from CDNs on demand - but I'm willing to give the benefit of the doubt and say never attribute to malice, what could be explained by earth-population-level scaling problems.
Also unrelated: Facebook has shown many times that they can't really be trusted on privacy issues - I never upload anything there under the illusion that I can freely delete it and it's really gone. But I don't personally believe this thing is a privacy issue they're intentionally balking on
You aren't familiar with Facebook's infrastructure and comparing working on small websites to working on something that scales to roughly 1/12 of the entire population of the planet is laughably naive.
Saying you use a "modul[ar]" approach is nothing but handwaving. You think that facebook doesn't use modules? Or doesn't have a service oriented architecture to some degree? They friggin' developed their own RPC transport!
Keep in mind, that when the photo features of Facebook were added, they probably didn't do it while thinking about half a billion users or whatever gazillion terabytes of photos they have to operate with. Heck, they may have even started out with simply storing photos on disk in their servers where deletion was easy as opposed to on a CDN.