Virtual address translation can introduce logarithmic slowdowns for certain memory access patterns. Other than that I don't know why you would want to eschew it.