None of those were 'small security holes'. SQL injection on your website? Unnecessary ports open and known vulnerabilities on a public facing server? This is embarrassing for a company that apparently focuses on security.

typically the infrastructure which is supposed to be "uber sekure" has been well vetted, and is relatively secure.

The problem is, there is almost always some "trivial" system (public web site, severely outdated wordpress blog, or worse) that some poor fool in marketing/product "HAD TO HAVE YESTERDAY". The admins knew it wasn't mission critical, and would only be "temporary". So they spent minimal effort to set it up, skipped over all of the process and security hardening they would do for a proper release, and left it.

Of course, we know what happens: some hacker finds the exploits, then pivots to explore the internal network.

You will find most big enterprise-y shops build networks with hard exteriors, and soft interiors. Very few of their security plans are capable of a threat from inside the network.

I was always baffled by the notion of "internal network". Why do so many admins think that it is secure, that the device on it should be trusted more than some random PC on the Internet?

Usually there are PCs and mobile platforms on it, handled by more or less naive users... many of them could be / are turned into unsuspecting adversary to attacks.

One should always treat internal devices as potentially compromised.

"Small" meaning easy-to-mitigate. I was expecting something along the lines of, "I spent months probing buffer overflows to leak security credentials." Not, "I spent three seconds and nearly fell out of my chair when I realized they don't sanitize database queries."

How would you know for sure that it didn't dump the database to somewhere in Asia once "they" have your server under control? Serious question, because how can you trust the logs? (Mind you, I'm not that technical)

Once the servers been comped, you can't tell what's been accessed on that machine. You could possibly find out if it accesses other machines within your network (logging depending) - however if someone were to root a public facing server that had a bunch of files on it, you have to assume they've been seen/duplicated.

That was exactly my thought so there is no "lucky" after all. At least not for sure.

And, also, we operate in a low-tech service industry where simply having a database of customers is considered moderately cutting edge. We're not a software company producing hacking tools for evil governments and their puppets. There's nothing interesting on the server for anyone save our competitors. That leads me to logically deduce that the "hacking" attempts the internet-facing servers experience simply fall into the net of trolls searching for more machines to add to their botnets.

All the logs over the years simply show spam from bots idly probing for pirated SIP lines/extensions on our VoIP box, attempts to send mail through our mail server, and open PHP MyAdmin/Django/Wordpress login pages--none of which are present because none of that software's in use.

Because they never gained root access. I trust the logs in this case because their actions were immediately made known: Access logs show failed login attempts from that ip for a range of usernames, a successful login on the compromised account, then nothing but reams of mail being churned out.

If it had been done for more nefarious purposes, wouldn't "they" have been more discreet, carefully wiping traces of their activity from the logs? Not doing something that immediately throws red flags like sending thousands of email messages?

In all honesty, I certainly don't have the skills to detect an NSA-level attack that doesn't involve brute-force attempts on accounts. I can erase or alter logs, but then there are logs logged of me vi'ing logs, so I erase the shell history, but then that gets logged when I log out. It's a weird loop I don't know how to defeat, but some people do.

The heart of our problem was a misconfigured sshd that permitted remote logins (not root logins) on all user accounts. A disaster in the making. We got lucky that it was a spammer who compromised the system and not a competitor.

> I can erase or alter logs, but then there are logs logged of me vi'ing logs, so I erase the shell history, but then that gets logged when I log out. It's a weird loop I don't know how to defeat, but some people do.

This is trivial, but you need to be familiar with the environment variables used by bash. unset HISTFILE

Or kill the shell from within, avoiding history write:

    $ vi /var/log/*.log
    $ kill -9 $BASHPID

The solution for log issue can be a remote specialized machine that does append-only logs and nothing else - it should be possible to lock down such a service so that if you're compromised, then at least you have unaltered data from the initial part of the attack, before they disable all logging.