News for hackers. Stuff that matters.
Re:An update on last week's customer shutdown incident by marcinzm on Tuesday June 04, 2019 @08:12PM (#20100416)
Reading this response it seems that crypto-mining is not allowed on digital ocean as they have checks against it. The TOS doesn't say so explicitly but does note that:

>violation of any of these Terms of Service or any law, or if you misuse system resources, such as, by employing programs that consume excessive network capacity, CPU cycles, or disk IO

By my reading that seems to mean that you're not allowed to use your VMs to their full capacity due to them being over-provisioned. This is in contrast to AWS who are more explicit on which instances (T instances) are over-provisioned and exactly how they're throttled.

Re:An update on last week's customer shutdown incident by bcooks on Tuesday June 04, 2019 @09:06PM (#20100666)
If you want to do cryptocurrency mining on DO that is actually okay with us. Some of the other respondents are correct the behavior we were looking for was really around fraudulent accounts being created and performing cryptocurrency mining. This is why the trigger that flagged this account was using payment history as a key factor in the triggering.
Re:An update on last week's customer shutdown incident by weaksauce on Wednesday June 05, 2019 @08:49PM (#20110556)
The thing that has me scratching my head is how this chain of events unfolded.

I get that your fraud algorithm flagged it because of lack of established payment. how is this possible if what the tweet referred to as "locking us out of all of our backups and work"? surely an account history of any significance would have an established payment record. From their tweets they mention that they had 5 droplets and some storage of a not insignificant number of records (~500k) and that a script is required to be run every 2-3 months to process some data and that script spins up 10 droplets during that time. seems like it will take 13 hours to process the data based on row count and per record time.[0] I am struggling to see how they didn't have payment history. can you elaborate?

In addition another thing I'd think would help assuage fears of a complete lockout is some process where you can request and download the db or a snapshot of the virtual machine.

[0] https://twitter.com/w3Nicolas/status/1134529322902007809

Re:An update on last week's customer shutdown incident by wyldfire on Wednesday June 05, 2019 @09:12AM (#20103974)
> If you want to do cryptocurrency mining on DO that is actually okay with us

Do you disclose this anywhere? Are there any special steps one could take to avoid issues while doing legitimate mining?

Re:An update on last week's customer shutdown incident by chris_wot on Tuesday June 04, 2019 @11:51PM (#20101314)
Your post-mortem implies this is not allowed at all.
Re:An update on last week's customer shutdown incident by Johnny555 on Wednesday June 05, 2019 @02:17AM (#20101929)
Your post-mortem implies this is not allowed at all.

Not sure why you were downvoted, I had the same impression, after reading:

...an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). These signals, coupled with a number of account-level signals (including payment history and current run rate compared to total payments) are used to determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers

This sounds like they don't permit extended high CPU loads due to the impact it can have on other customers.

Re:An update on last week's customer shutdown incident by solidasparagus on Wednesday June 05, 2019 @04:34AM (#20102488)
Cryptojacking is a well-known, major problem for cloud compute providers. Catching and squashing new exploits that allow people to create a fresh account, run up compute bills and then abandon the account without paying is very important.

My guess would be that this is such a well-known problem (within the field of cloud compute at least) that they just didn't think they had to state that normal crypto mining by paying customers is completely fine.

Re:An update on last week's customer shutdown incident by mr_toad on Wednesday June 05, 2019 @06:11AM (#20102874)
Is ‘normal’ crypto mining in the cloud even profitable, compared to custom designed hardware?
Re:An update on last week's customer shutdown incident by AgentME on Wednesday June 05, 2019 @06:50AM (#20103063)
Depending on the cryptocurrency's proof-of-work algorithm and new-ness, it can be profitable to mine in the cloud. I've done it briefly in the past. But generally it's not profitable.

In every cryptocurrency (the popular and functional ones anyway), there's a set global rate of mining rewards. All miners compete for a slice of that reward, so as more people mine, each individual miner gets less reward. (This causes an equilibrium to be reached where more people mine until it's no longer profitable for more people to start mining. If mining becomes unprofitable, some miners will drop out, and the remaining miners will each make a little more.) If masses of people realize that cloud mining for a particular cryptocurrency is profitable, then what generally happens is that lots of people pounce on cloud providers to mine, it becomes barely profitable, and then people operating their own hardware that's cheaper than cloud providers come in and push the mining rewards down to where it's no longer profitable for people to cloud mine.

Because cloud mining is never profitable in the long run, most cloud mining that happens is fraudulent activity using stolen cloud accounts or payment info. (If you're not paying for it, then making any amount of money from it is profitable.)

Re:An update on last week's customer shutdown incident by vthriller on Wednesday June 05, 2019 @07:07AM (#20103141)
It depends on what crypto is going to be mined an how many accounts can be stolen given the fact that there is already plethora of bots that look all over GitHub for accidentally committed credentials. Heck, just a year ago people did scans for outdated WordPress installations to inject, among other things, some JavaScript (!) Monero miners [0]…

[0] https://arstechnica.com/information-technology/2018/01/more-...

Re:An update on last week's customer shutdown incident by nostrebored on Wednesday June 05, 2019 @10:52AM (#20105000)
No. Cryptomining represents an arbitrage opportunity such that the spot price of the instances should be adjusted. In the long run it should not be profitable.
Re:An update on last week's customer shutdown incident by fhars on Wednesday June 05, 2019 @02:32AM (#20101992)
No, that quite clearly states that they treat high CPU loads as suspect on accounts without an established good payment history or if it significantly deviates from previous usage patterns.
Re:An update on last week's customer shutdown incident by bgirard on Wednesday June 05, 2019 @02:46AM (#20102059)
The keyword here is 'fraudulent'. High-cpu-loads is allowed, but an automated service monitors for fraudulent activity.
Re:An update on last week's customer shutdown incident by dlubarov on Wednesday June 05, 2019 @03:00AM (#20102107)
What would "fraud" mean in this context? Are they talking about customers who don't pay their bill to DO? (If so, seems like the account should just be temporarily suspended until the bill is paid.) Or are they talking about fraud to other parties, like phishing sites? (If so, I don't see the connection to crypto mining.)
Re:An update on last week's customer shutdown incident by hoseja on Wednesday June 05, 2019 @03:22AM (#20102192)
My understanding is that they're trying to prevent users from creating new accounts, running 100%CPU until it's time to pay the bill and then just not paying, moving on to another new account.

edit: from elsewhere ITT it seems they're doing this with stolen credit cards.

Re:An update on last week's customer shutdown incident by chris_wot on Wednesday June 05, 2019 @03:34AM (#20102255)
Obviously they don't verify if the load is fraudulent. Otherwise this whole debacle couldn't have happened.
Re:An update on last week's customer shutdown incident by rajaganesh87 on Wednesday June 05, 2019 @12:17AM (#20101435)
Blog post did mention that accounts with high CPU usage and payment history won't be flagged.
Re:An update on last week's customer shutdown incident by ac29 on Tuesday June 04, 2019 @10:11PM (#20100947)
Here was my key takeaway:

"Cryptocurrency mining mitigation detects suspicious behavior, including very high CPU utilization on an account with no payment history, which results in an account lock"

Lots might have been done wrong here, but it sounds like they had an account with trial or promotional credits - I can see how this could easily be abused.

Re:An update on last week's customer shutdown incident by mdip on Tuesday June 04, 2019 @10:55PM (#20101107)
Completely. At the same time, those promotional credits are going to be used by guys like me who will have to decide if their services are worth having to spend an extended period of time explaining why "we're not recommending AWS/Azure/Goober"

An account shutdown, or enough complaints from verifiable sources, and I'm not going to the trouble. Not to pull out an old trope "Nobody got fired for picking IBM". But that's the case: pulls a move like this and the entirety of the customer, who likely came in with AWS in mind (in some cases, was advised against it and insisted on it) is going to shrug their shoulders. Pick a provider that the customer hasn't heard of and I'm going to get a phone call that goes something like "You're the one that said we should use that basement-operation!" with raised voices. Heck, the last time there was an Azure outage, we didn't hear from most of our customers. It was so impacting that even customers well outside of software development/technology read news articles and connected the dots. I had one customer tell me he thought it was just their corporate internet connection; they assumed it was working[1].

[0] Plus, too lazy to put in the research; sorry.

[1] They were a customer who insisted on doing the app monitoring, themselves -- that guy was getting the alerts and similarly assumed it was the network since that happened regularly with another application they developed -- the monitoring server was on-prem.

Re:An update on last week's customer shutdown incident by phire on Tuesday June 04, 2019 @10:44PM (#20101071)
Sounds like it's designed to counter stolen credit cards.

An attacker might load a stolen credit card number into an account and only use enough resources to generate a few dollars worth of billing. The owner of the credit card might not notice the small charge.

Then after a few months of low billing (to bypass a previous heuristic), they ramp up the utilization, mine a bunch of coins then the holder gets a massive bill.

The holder does a charge back and DO is left holding the bill.

Re:An update on last week's customer shutdown incident by kijin on Wednesday June 05, 2019 @02:05AM (#20101878)
It's also designed to keep everyone relatively happy in a shared-CPU environment. "Standard" droplets share CPU with others on the same node, so one droplet pegging the CPU 24/7 can be problematic.

AWS doesn't have this problem because either your instance is allowed to use all the CPU that's allicated to it, or else (t2 & t3) the platform will automatically limit your CPU usage. You don't have to care about how your usage affects other people. It's one more thing that AWS abstracts away. DO's abstraction, on the other hand, is rather leaky in this area. That's a problem in and of itself, in addition to the matter of credit card fraud that every company has to deal with.

Re:An update on last week's customer shutdown incident by xvf22 on Tuesday June 04, 2019 @08:14PM (#20100429)
I don't think that it isn't allowed but that they have seen that fraudulently acquired resources are generally used for crypto-mining so they felt that this was a good signal to look for.
Re:An update on last week's customer shutdown incident by duskwuff on Tuesday June 04, 2019 @08:32PM (#20100515)
Right. Reading between the lines a bit, it's not the activity itself that DO is worried about, but a pattern of usage that suggests that the account may have been created fraudulently or compromised.
Re:An update on last week's customer shutdown incident by bcooks on Wednesday June 05, 2019 @05:43PM (#20109478)
This is correct. This was the primary thing we were attempting to solve for in this case and the bug in the algorithm started the chain of events documented in the postmortem.
Re:An update on last week's customer shutdown incident by Twirrim on Tuesday June 04, 2019 @11:31PM (#20101235)
Cryptocurrency mining is a favourite of fraudsters. Get a fake or stolen credit card number / identity, sign up for an account on a cloud provider, and spin up instances to do your mining. Depending on how quickly the provider reacts to the pattern of behaviour, or identifies the account as fraudulent etc, you can have earned a reasonable pay-off.

If the provider isn't paying attention and/or doesn't have good fraud detection in place, it may be a few months before your account and resources are terminated (assuming the payments eventually bounce, and the provider gives you a chance to fix it)

Re:An update on last week's customer shutdown incident by Shoop on Tuesday June 04, 2019 @09:06PM (#20100667)
I believe the purpose of these checks is for when people's outdated wordpress install or whatever gets compromised by script kiddies. Generally, the scripts install crypto miners to mine for the hacker until your account gets shut down, running you up a huge bill in the process.
Re:An update on last week's customer shutdown incident by treis on Tuesday June 04, 2019 @09:35PM (#20100795)
I think it's moreso people continually creating new accounts to get free credits. Then bitcoin mining turns the credits into actual money.
Re:An update on last week's customer shutdown incident by jimmaswell on Tuesday June 04, 2019 @08:21PM (#20100461)
Anecdotally, I had a python script unintentionally using 100% for a month or two and never heard about it or noticed until I happened to look at top.
Re:An update on last week's customer shutdown incident by xnxn on Tuesday June 04, 2019 @08:54PM (#20100613)
Counter-anecdote slash venting:

I recently logged into my long-dormant DO account to kick the tires on their now-GA managed Kubernetes service and to contribute some CPU cycles to a distributed computing project I'm interested in (not cryptocurrency, I swear).

I first requested and was approved for a droplet limit increase (to 25). I started 20 nodes, deployed my very CPU intensive workload, and 12 hours later my account was locked and my nodes all went NotReady.

I immediately replied to the abuse ticket explaining my usage. 4 days later, my account was unlocked and received the "allow high CPU" flag... but they billed me for the nodes as if they had been running that whole time. I asked for a credit (5 days ago) and they haven't replied yet. Probably a little busy over there right now.

So... I'm not too thrilled with DO. I get that cryptocurrency ruined everything, but this has been a frustrating experience and I'm glad I was only using it for a pet project.

Re:An update on last week's customer shutdown incident by zacharybk on Tuesday June 04, 2019 @09:30PM (#20100765)
Hey xnxn - Thanks for raising this issue. I'm Zach, Director of Support at DO. Please send me an email with your account detalis, first name at, and I'll take care of this.
Re:An update on last week's customer shutdown incident by sneak on Tuesday June 04, 2019 @10:37PM (#20101045)
Customers should not have to resort to shaming you publicly to get timely support responses.

Two points establish a line.

Re:An update on last week's customer shutdown incident by zacharybk on Tuesday June 04, 2019 @10:45PM (#20101072)
Hey Sneak - I totally agree* :) We've already started efforts between Support and Security leadership to leverage the 24/7 structure of Support. Our goal is that no one will ever need to use social as an escalation path, and our new Support Engineers who are joining in mid-June and early July will be part of this a reality for our customers.
Re:An update on last week's customer shutdown incident by metildaa on Wednesday June 05, 2019 @05:59AM (#20102829)
Fyi, I'm planning to migrate off of DO due to the crummy payment options. With every other recurring service I can have a PayPal subscription, DO doesn't support PayPal properly though.

Also, your billing system is incredibly spammy (1 to 2 emails a day at times). The bills I get are always for partial months despite the VMs on the account running for a full month.

Re:An update on last week's customer shutdown incident by zacharybk on Wednesday June 05, 2019 @06:55AM (#20103080)
Hey metildaa - Zach here from DO support. It is accurate that that do not currently support PayPal subscriptions, this certainly isn't the overall experience that you should be having. Can you shoot me an email (first name at) and I'll look into this for you?

Thank you! Zach

Re:An update on last week's customer shutdown incident by xnxn on Tuesday June 04, 2019 @09:32PM (#20100773)
Thanks Zach, will do.
Re:An update on last week's customer shutdown incident by zacharybk on Tuesday June 04, 2019 @10:34PM (#20101031)
Email received, followed up on, and account credited.

-Zach

Re:An update on last week's customer shutdown incident by chris_wot on Tuesday June 04, 2019 @11:55PM (#20101344)
Really? What is so bad about your support that you needed to hear about this from HN before you did anything?
Re:An update on last week's customer shutdown incident by heyoni on Wednesday June 05, 2019 @12:15AM (#20101426)
The postmortem...they admit that they need to hire more support staff.
Re:An update on last week's customer shutdown incident by chris_wot on Wednesday June 05, 2019 @03:35AM (#20102260)
That's my point. Their support got so bad that they had to be called out on in publicly before they did anything. That's hardly what you want to hear from a company like DO.
Re:An update on last week's customer shutdown incident by mschuster91 on Wednesday June 05, 2019 @05:35AM (#20102724)
> That's my point. Their support got so bad that they had to be called out on in publicly before they did anything. That's hardly what you want to hear from a company like DO.

That's what you get with most providers these days. Ever tried reaching Google or Amazon?

Re:An update on last week's customer shutdown incident by glenneroo on Wednesday June 05, 2019 @07:23AM (#20103247)
Actually I have complained to Amazon... and they have always responded within a couple days and in one case where I complained about Prime Video, they called me on a Sunday, which shocked me because everything is otherwise closed in Austria, except gas stations, restaurants and hospitals.

Google on the other hand, I make complaints or suggestions once every couple months and 99% of the time I don't even get a boilerplate response.

Re:An update on last week's customer shutdown incident by chris_wot on Wednesday June 05, 2019 @06:12AM (#20102883)
I don't believe that Google or Amazon have ever done anything like what DO did before. However, if you notice their support guy Zach was responding to a completely different incident that is not related to the one being addressed with their post-mortem.
Re:An update on last week's customer shutdown incident by nostrebored on Wednesday June 05, 2019 @10:56AM (#20105048)
If you pay for enterprise support with Amazon you can open a case with a less than 15 minute SLA [1]. With business that goes down to an hour SLA. With both plans you can create chats or calls that typically get answered within a matter of minute with and assigned to an engineer with a background in the service.

[1] https://aws.amazon.com/premiumsupport/plans/enterprise/

Re:An update on last week's customer shutdown incident by jasonlotito on Wednesday June 05, 2019 @10:15AM (#20104567)
Amazon and AWS. I've always been happy with the support I've received from them. It's always been easy.
Re:An update on last week's customer shutdown incident by marcinzm on Wednesday June 05, 2019 @09:43AM (#20104228)
Amazon support is from everything I've heard and experienced great.

So don't lump everyone together in an attempt to paint DO in a better light when it's not true.