It's not just the area where the first stage might touch down, it's how far large pieces of debris might be thrown in a worst case RUD.

Plus, the Secret Service may be feeling far less tolerant of "eh, it's less than a 1 in BigNum chance of anyone getting hurt" odds than your average Tom, Dick, or Harry.