Even if you grant "only the government has access," I think you still shouldn't be complacent, because what the government considers acceptable can change, and may not agree with what you think is acceptable.

E.g., maybe a govt comes into power that is strongly anti-gay, and punishes you for being pro-gay rights.

Or the opposite.

In general the idea that the government will always be reasonable and should have unlimited surveillance powers is unsound and unappealing.