I also switched my small and ad-free blog to https-only and I haven't noticed a drop in traffic. On the other hand, even if I had, I wouldn't have cared.

I'm proud to contribute to a saner Internet and it does matter even for small blogs because I noticed networks that inject content in websites - I don't know how this practice evolved in the US, but a couple of years ago while traveling there the Wifi networks in the 2 motels I stayed at were injecting ads in the websites I was visiting. I found that to be extremely distasteful.

For me HTTPS is a way of signing my content. Shameless plug - https://www.bionicspirit.com/ :-)