Great question. I agree with your logic, and that on its face it isn't vitriolic, or even really offensive (or shouldn't be at least). However I've only ever really seen the term used by people that are using it as a pejorative. For that reason I take more of a structuralist view on the meaning, which is subjective.

I've never heard somebody (excluding trolls) describe themselves as a "climate denier" either.