If they have access to the account that is being used normally, they can modify the (user-accessible) settings to trick the user into running malicious code and giving them access (or causing trouble even without access to the root account).

If you lose physical control over the machine, all bets are off because an attacker can modify the hardware to do nefarious things.

I know the theory, but practically there's a huge difference between that type of physical access and "the victim left the room to go to the bathroom for 2 minutes" type of physical access

ok, sure, but in practice that is pretty unlikely.