To be fair, triviality is relevant to the tooling you run and there are many downsides - extra negotiation and traffic, unsupported on older clients without using up an IPv4 address, certificate expiry, etc.

For most cases these are non issues, but there are many scenarios where those things can outweigh the potential of your ISP modifying/reading. If that's still a concern, you can tunnel through your ISP to a more trusted exit point.