If data is just stored and encrypted at transport and at rest I would say not necessarily.

But if I read about that deep-learning magic they plan to do, I'm seriously concerned. Especially since I find the already existing note context feature quite invasive. Companies should only be allowed to do with my data what I give consent to.

I'm all for privacy and tinfoiling, but I have to say they already can do only what we allow. And oh boy do we allow. It's very simple, if you do not want to allow, then don't use service X. To expect some kind of personalised privacy policy deal from these companies is not reasonable. It would be great, but it ain't gonna happen, even if they were willing. It's practically and economically impossible.