I do trust my own server not to misbehave, but I probably would not trust some random server on the internet.

I've had personal servers compromised. I don't unconditionally trust them either.

Exactly. I'm usually worried about the opposite scenario (the client infecting the server). If you're copying files from a personal computer it might have all sorts of random software running on it that's accumulated over the years. Whereas on a remote server you tend to have a better security posture, and aren't installing random software and apps. Admittedly, that might just be my personal use case though.