I'm the same way. Sometimes I'll try to ride slowly to avoid getting sweaty, but I'm rarely successful. I'm just used to riding faster, and it's more fun. As you said, it also can feel safer/more natural if you're next to car traffic. Luckily I've usually worked places that had showers if I really needed them.