My not particularly well informed understanding is that it is more a of a practical exception than a technical one. So a patent holder could try to use the legal system to stop personal use, it just isn't worth doing.