In my experience, it makes a big difference. My swap is on a hard drive, which can handle the sequential reading of the swap quite quickly. Whereas the random accessing of it in normal use is much slower per byte.