I feel similarly. Switching ports is no real defense, but it at least means you are eliminating the drive-by attacks who are only interested in the trivially exploited. Such a simple thing to do and sharply reduces the log volume.

The next trick I think of implementing is port knocking. Should drop log noise to zero unless someone starts targeting me specifically. In which case, my goose is already cooked.