The appropriate notion here seems to be Differential Privacy, which is a mathematical definition informally saying "a scrambling of the dataset that is information theoretically indistinguishable from that where one arbitrary person is added or removed". It's a surprisingly deep topic, with entire (very good!) textbooks dedicated to it.

PDF (entirely legal): https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf