<guess>VM space is not free. You need pagetables and various other structures to support it. The larger the VM space, the more of them you need, which costs real actual RAM. ASLR would make it worse as it will scatter VM allocations around the address space more, forcing the creation of a lot more 1st and second stage pagetables (and possibly other structures)</guess>