Virtual memory doesn't exist solely to support swap files. It also allows any program to assume it's the only thing running (which simplifies assembly code) and enables memory-mapping of files and devices.

(Yes, you'll still need to memory-map files when we have memristors. They still won't necessarily be contiguous on disk, and they still won't necessarily exist locally.)