I'd imagine it's relatively easy for the OS to just swap out the memory from the inactive user. Unless they're actually running a task in the background, but how often does that happen?