In my experience building LLVM (not for Swift though), it depends on your make -j switch. With -j 16 I would run out of memory quickly (32 GB RAM, 16 GB swap), with -j 1 it would be quite conservative memory-wise, but take hours to build.

So I switched to ninja (cmake -G Ninja). It can use all the cores and not run out of memory.