I've had the same thoughts, and I think you articulated them pretty well; maybe better than I could have. I.e., for me it's something like: I'm much more willing to share if I know the party I share with will share back, in turn.
I know it's more complicated than that, with economics and privacy factors involved. But if I operate on the assumption that real privacy is real hard I'd rather just be guarded and careful with my prompts and whatever output I give to the LLM and expect that it will be training on that, rather than spill all my darkest secrets to some other LLM provider that pinkie-promises privacy only to leak it publicly later, anyway.
I'd still prefer it wasn't like that, but I guess it's a compromise ultimately, at least I'll be able to run it myself.