I hear you, and I've restarted the display server plenty of times on my computers instead of restarting them, especially on my main desktop where I have lots more stuff running independent of the display (daemons, etc). But, I feel like particularly on a laptop, if you're restarting the display server, it's not much different than just restarting the whole machine. Certainly that's the case for the large majority of users. Anyway, I think the point is that it's a hassle, which it is, not that you literally "need to reboot".