The kernel write the pages into swap long before it decides they are rarely used. Otherwise rewriting them would be way too slow to be of any use. Any normal use will fill some swap space, and if the memory pages keep getting dirty (and you have spare disk IO), your computer will keep writing them to swap.

How much RAM you have is not really relevant.