If you need to use all the memory then swapping breaks down, best you can do is try to divide the work so half can be done, then you completely swap all the memory and do the other half. Also PCIe is (relatively) slow for moving gigabytes of data back and forth.

Swapping would be too slow.

Llama.cpp allows you to specify that n layers load into the GPU's VRAM, and the rest load into main memory.

Some frameworks have tried to do this automatically, but its extremely slow. You are better off just running whatever won't fit on the CPU (which is what llama.cpp does).