Don't they? They certainly have access to the VPS file system.

But even beyond that, can they be compelled by a court order to install software on the VPS (and securing against someone with unlimited access to the VM host is… about as practical as securing against someone with physical access)? I'd assume so?

Probably?

The main benefit to me for runnign my own email is owning my own data/domain, and not having to allow third parties (aka google) full access to all my emails.

You just have to be sure and disable most logging if you're using something like mailinabox. Defaults are still pretty loquacious.