Totally agree. Sometimes I watch my live logs of auth.log of about 100 public servers. I am probably see 10 port scans or ssh attempts per second from random machines. Nothing to worry about if machines setup correctly. More worrying are the people who try to break our public APIs. Either by letting it to crash, brute force operations etc. The security flaws are often inside the software domain layer.