Traditional swap is more complicated than that. The kernel can't map it directly as RAM, it has to be paged to and from the backing store and RAM - doing so is not just an optimization.