To me the distinction is in whether model trained on my data can return it almost verbatim. Any statistical model, even linear regression can expose some of my data and preferences, but the difference is whether they can compromise the data they've been trained.

And frankly I see the distinction in only number of model parameters vs number of people in cohort it was trained. So 50k-param model trained on 1b emails from 1 million people is ok, encoding/compression loses anything personal. But huge LLMs are closer to lossless compressors that can just spew all my passwords, prescriptions etc verbatim.