How much of that 1 million lines of code actually gets executed? Also, everyone runs the X server, so its code gets a lot of testing. This isn't true for window managers -- there's a long tail of them. This is just one data point, but in my experience I've had window managers crash far more often than X servers (since they get less love).