The single source of truths, all the core Facebook users data (except messages) is stored in MySQL cluster. The issue you describe is that the same data is also cached in Memcached and by other means. But deleting is entirely possible, though it's against their business model, so they make real deleting as hard to find and do as possible, and scare people away with info like "you cannot register again".

Training of neural networks is probably a gray area, and a potential issue for many companies. Example: Think of the recorded audio messages from Siri/GoogleNow/Cortana that gets processes and potentially stored and not deleted thereafter on their (and third parties like Nuance) servers. If one deletes an account, one would assume it also and especially also deletes all private data like messages, voice and video recording.