My experience is that it's most important to keep your ears, hands, and torso warm, which I achieve with a windbreaker vest or jacket depending on how cold it is. Even if I'm a bit chilly when I leave, I'm certain to be warm when I arrive at my destination -- sometimes a bit too warm, but not nearly as sweaty as I would be if I were riding a regular bike instead of an e-bike.