News for hackers. Stuff that matters.
Re:Stop using Digital Ocean Now For the greater good by TylerE on Tuesday September 24, 2013 @02:52PM (#6439926)
> I sent a few notices before and after shutting them down, but their mail was hosted on the down VPS, so they never got it. Man, they were angry.

Isn't this why most competent hosts request/require a secondary contact e-mail, not hosted by them?

Re:Stop using Digital Ocean Now For the greater good by lsc on Tuesday September 24, 2013 @02:58PM (#6439965)
>Isn't this why most competent hosts request/require a secondary contact e-mail, not hosted by them?

Enforcing that requirement would be... non-trivial, though. I mean, I could look at the MX records for the domain in question, but those quite often don't point at the final delivery servers.

Still, I should at least make that a rule, even if I don't enforce it.

edit: Also note, even when you contact folks successfully, sometimes they refuse to deal with it, as, well, properly dealing with a compromise involves formatting and re-installing, then restoring from a backup taken before the thing was compromised. Most people at this level are unwilling to go through the effort, meaning that they will remain compromised for as long as you leave them on your system.

Often I'll setup a new domain for them, with the old domain read-only, so they can pick through the data. More than half the time, within 24 hours, they are compromised again.

Re:Stop using Digital Ocean Now For the greater good by dllthomas on Tuesday September 24, 2013 @03:29PM (#6440150)
"Still, I should at least make that a rule, even if I don't enforce it."

"You might not get notified when shit hits the fan" seems like some natural enforcement :-P

Re:Stop using Digital Ocean Now For the greater good by telephonetemp on Tuesday September 24, 2013 @03:12PM (#6440056)
>More than half the time, within 24 hours, they are compromised again.

Is that because they don't upgrade their software in response to being compromised?

Re:Stop using Digital Ocean Now For the greater good by lsc on Tuesday September 24, 2013 @03:22PM (#6440105)
>Is that because they don't upgrade their software in response to being compromised?

Recovering from a competent compromise is... more complex than that. You pretty much have gotta read all the code you move from the old (compromised) system to the new system. As part of that, you want to minimize the code you move from the compromised to the new system; so you should re-install as much as you can from scratch, then move over your custom stuff, one file at a time, after reading it carefully.

If you have a good pre-compromise backup, and you have some idea of where the hole was and that hole was in one of the packages/support libraries, and that hole has been patched, then you can restore and upgrade.

As I'm running a no-support service, I'm not digging in deeply enough to tell you exactly what happened, but I believe that in most cases, users are just copying over their documentroot wholesale, and at best upgrading over the top of the (possibly compromised) copy, which quite often won't help you.

Re:Stop using Digital Ocean Now For the greater good by telephonetemp on Tuesday September 24, 2013 @03:41PM (#6440221)
> You pretty much have gotta read all the code you move from the old (compromised) system to the new system.

If they don't have a know-good local copy of the code to upgrade the stock components of and redeploy on a fresh VM that just sounds like asking for trouble.

What do you do with such customers? If you "fire" the repeat offenders would you say it's been worth the forum backlash from them (which this story seems like an attempt at so far)?

Re:Stop using Digital Ocean Now For the greater good by lsc on Tuesday September 24, 2013 @05:50PM (#6441010)
Yes, I fire them. The form backlash varies a lot depending on the customer and how I handle it. I'm rather smaller than Digital Ocean (I recently dropped below 2000 customers) so I have something of a more personal touch, which usually helps more than it ought to help.

I also strongly select for customers who have some sysadmin experience (e.g. I turned someone away this morning who wanted a VPS but didn't want to authenticate with an OpenSSH public key) which helps a lot (but also vastly decreases my customer base.)

Worst case is when someone knowledgeable setup someone else on my service, then broke off contact; I'm now expected to step in and essentially be their sysadmin for $12/month. - That's the problem with unsophisticated users in a unsupported environment; they don't know enough to know if the problem is a hardware problem (which really is my responsibility) or a configuration on their own VPS.

but yeah, occasionally I get someone really, really angry. It's no fun.

My least favorite part of the job is firing customers who are the /target/ of DDoS attacks. I mean, if the attack is smaller than my pipe, I can tolerate it, but I've had customers hit with 10 gigabit+ attacks. Few providers can deal with that, so you are forced to get rid of the customer; It's really sad and messed up, because sometimes it's not even the customer's fault. Someone on the internet who controls a botnet doesn't like them. It's not fair that you finish the job, but there often isn't a whole lot of choice in the matter.

I have a whole lot less sympathy for people who allow their domains to be compromised and used in those attacks against other people.

Re:Stop using Digital Ocean Now For the greater good by livingparadox on Tuesday September 24, 2013 @02:58PM (#6439958)
I believe he means they setup their own mail server on the VPS, not that the mail itself was was provided by the VPS provider.