100% agreed, and it's the first setting I change.

In addition to that I have found that using the setting:

    AllowUsers "[email protected]" 

(If you are connecting from a static IP) Will cut down on log spam and connection initializations by a tremendous amount.