We are considerably smaller than DigitalOcean in staffing head count. But we have a network that is spread out geographically across five states and 30+ cities and towns, built a combination of third party lit L2 transport, dark fiber IRUs, and fiber we built ourselves. We're a facilities based WAN provider.
There are so many different possible types of ISPs. For a small organization it only makes sense to decide whether you want to go after a huge number of $5 a month customers, or if you want to focus your time and effort on customers that spend anywhere from $250/month upwards for last mile broadband services, colocation/hosting services, etc. As a generalization, the higher the dollar value of the customer, the less of a headache they are, and the higher the clue level of the customer is.
I concur with 100% of what the above poster says about the hosting business.
Bulk hosting/VPS/VM hosting is an incredibly brutal race to the bottom in pricing. Extensive well crafted automation tools and massive economies of scale are the only thing that will save you. I truly feel sorry for the people who are working (mostly entry-level) jobs doing first tier technical support/customer service for 5 dollar a month VPS customers.
If somebody wanted to hire me to work for a consumer-facing hosting company I would run away screaming. It's my idea of a personal hell in the ISP business. Those who have found a way to make it work, not go bankrupt and not have mental breakdowns are a rare breed.
Ironically, this same strategy is what makes cloud providers more interchangeable to their users, and drives down prices.
I've had a "shared hosting" setup on Dreamhost for at least 15 years, also using pennies per month in capacity, at most.
I get that it can be a brutal, low-margin business, but I also wonder how many "small" customers are extremely high-margin like me, and whether that can aggregate into a better overall margin than you might guess?
Or will you always have a few outliers running at capacity and calling the help desk and blowing out your margins?
The 80/20 rule in in full effect here. The 'needy' customers are extremely resource intensive.
The key to remaining competitive in cut-throat industries is knowing where to spend your limited time and money.
How does this apply to Planet Fitness in any way?
one decent incident can cost a multiple of a year's revenue for the account.
That 5/mo customer is much more likely to disappear than pay for the data they used.
Is there no such thing as prepaid, "the VM stops when your credits run out" VPS hosting? Seems to work well-enough for Twilio.
5 weeks. Deleted everything.
Yes, I screwed up. But I would have happily paid them. They deleted the backups too.
When we inquired as to whether the backups could be restored now that we’ve paid them, they said it was impossible.
Blame me if you want and say it’s my fault. It certainly is; I admit that. But why was I paying them for backups that they wiped along with my server?
Meh.
(The card expired.)
The issue seems to have been that you weren't.
Lost everything when they deleted the backups. They turned off the server on the same day, and it was running fine right up until then. Zero warning other than the emails I didn’t see.
How should they get ahold of you? A call that you don't answer? A letter that goes unopened? African swallow laden with a note that you don't see?
Keep backups elsewhere. Regardless of cloud provider or payment terms or whatever. 3-2-1 backups.
I had my wallet stolen on one of my work trips, and had all of my credit cards deactivated and new numbers issued. Digital Ocean's emails went to my "bulk" folder, and I missed the notifications saying that they were going to disable, then deactivate my service.
What caught my attention was one of the support guys emailing me and saying basically, "Hey, I see that you've been a customer for a long time -- we're going to give you 30 more days to fix this (and btw, you still owe us for your services so far) before we delete everything."
That support interaction -- that wasn't flagged as spam -- is literally what kept my data, and it keeps me using Digital Ocean: amazing support.
What they could do is work out some high restore cost which makes the numbers work on holding all that data for longer.
The hold happens right away and prevents you from creating new billable resources.
Then a suspension happens several weeks later, which powers off your servers.
You don’t get terminated until a couple weeks after that, at which point the data is irretrievable.
The time between each stage can vary depending on how long you’ve been a customer or what your monthly payments in the past have been, but that’s the gist of it.
A lot of people seem willing to blame you without knowing the answer to this question.
On a different note, this kind of thing is probably one of the top threats to the survivability of a cloud-powered site or service. Alongside vandalism by disgruntled (ex-)employees, I imagine it happens far more often than outages in cloud providers' infrastructure.
In contrast, I haven’t paid Dropbox for years and my data is all still there.
Is there a simple solution for this?
DO reserves say $20 (just an example) when one spins up a new droplet, and if one forgets to pay — then, DO shuts down the server. But keeps the backups, until the GB-month cost is $20.
One could choose how much money to reserve, depending on how important the data one stored on the droplet, was. If it's just for running test: $0. Customer data: Maybe $$$ instead.
There could be a default that made the data and backup last for 2 or 3 months?
And any remaining money would be refunded, if one closes one's account or sth like that.
* * *
Actually I'll look into implementing this, in my own SaaS (which, like DO, takes monthly recurring payments and stores customer data).
It's five weeks really poor form but eight acceptable? Seems a bit subjective.
DO reserves say $20 (just an example) when one spins up a new droplet, and if one forgets to pay — then, DO shuts down the server. But keeps the backups, until the GB-month cost is $20.
One could choose how much money to reserve, depending on how important the data one stored on the droplet, was. If it's just for running test: $0. Customer data: Maybe $$$ instead.
There could be a default that made the backups stay for 3 months?
Any remaining money could be refunded, whenever one wants.
(also "cross-posted/replied" here: https://news.ycombinator.com/item?id=22094218 )
All in all, from the moment of non-payment to the moment that everything you have on their servers is gone shouldn't be less than 90 days in my personal opinion. If that raised the cost of my backups by a few dollars, so be it. The backups are there to account for things I may not have been able to consider.
I would love to live on the planet you live on
If there were, for any reason, an internal lapse in communication or processes (imagine small companies...) that could result in a pretty sour situation.
5 or 6 weeks vacation per year, + about 1 or 2 more weeks in total, because of Christmas, NYE etc, is how things work in Scandinavia.
(I feel curious about how things are, where you live )
Sounds like you weren't paying for backups.
tl;dr - customer stops paying service provider, and service provider stops providing services. Customer is upset.
This one's on you.
It's a completely different story if you were late by 1 day and they sold all your stuff the next day.
I guess the question here is if DO waited long enough before deleting their droplet. They were over a month late, and I'm assuming they sent notifications....
For 6-7 years now I'm using DO for free but my only resource-intensive instance is a personal VPN server that I use from time to time(a few hours a day maybe?).
At first, DO use to give referral payouts in cash, then they limited the payouts to credits, then changed the structure and introduced expiration date to these credits.
I would guess that this referral program that probably helped them a lot with the growth at first is now a burden.
I don't understand
a) How does this possibly make them money?
b) If they want signups that much, why is the flow for joining the referral program so buggy?
(It's Mailgun, btw)
Although some undoubtedly pay more than the lifetime value of a customer as a "growth hack". This of course can't go on forever.
They take in less than the referral payout in revenue for a customer who sends tens of thousands of emails a year for five years based on their pricing page.
b) Because as long as some signups happen, noone is testing it.
My interpretation of their marketing pages is that they pay out referral payments in cash (actually SWIFT, but...) once a signup has something charged to their credit cards. So they can't rely on unused credits. And they lose money on every referred signup that makes them less than $120 in profit.
This seems insane to me, because someone sending tens of thousands of emails to hundreds of people a year gets them less than half that in revenue.
IMHO referrals work amazing when people you refer mostly want this problem to go away and it's not their primary occupation.
This makes it sound like you could get a big win in the VPS-provider space by drawing an ROI line at ~$20, and making all instances below that size diskless, with their rootfs being either a tmpfs overlay of a shared SAN-mount of a base image (like a LiveCD environment), or a tmpfs into which was dumped a PXE initramfs image (as e.g. CoreOS does in its idiomatic deploy style.)
I feel like many customer use-cases would still be satisfied by such instances (especially if you also offer local object-storage for the diskless instances to interact with.) It'd sort of be a hybrid position between ephemeral PaaS containers, and actual persistent VMs.
Anyone know of a provider that provides low-cost long-running diskless VPSes like this?
Of course that already exists, and DO offers it for $5.
Well, to be honest, Vultr has 2.5$ (IPv6 only) and 3.5$ plans. So, if they're getting by, so could DO.
https://www.vultr.com/products/cloud-compute/#pricing
I actually migrated from DO to Vultr because at the time DO offered 512 MB RAM for $5, while Vultr offered it for $2.5. And Vultr gave me $50 bonus platform credit on sign up, valid for about 18 months (accounting for possible overage fees).
I'm still on Vultr, 3 years on. No problems at all, other than billing issues (accidentally was assigned Australian VAT despite living in Serbia), I had no support tickets. After some time I started using more instances, and more powerful instances, and more services (block storage, "portable" IPs, object storage, internal networks, etc).
I've had a lot of problems with DO's Object storage which was also one reason to move away from them. Problems were quite catastrophic in nature, i.e. the files were unavailable for a few hours every few weeks.
Recently I ave got into Upcloud.com, they have [1] flexible plan that you could mix and match resources, allow me to spin up 20x vCPU, 1GB RAM, 10GB SSD for $168 / month, or 4x vCPU, 128GB RAM, 10GB SSD for $550 / month.
Pretty damn good if you ask me.
And we have no idea if a similar action by Vultr is imminent (I'm not saying it is -- but we don't know).
They will, and you will tell them to leave. Unmanaged VPS' are unmanaged.
> Those pwned customers are complaining and demanding that you fix it Tell them to leave, they are paying $5/mo after all.
> regular customers are also upset because of slowness due to the "noisy neighbor" problem inherent to all VPSes Throttle abusive users, and hand-wave it via AUP, TOS, etc.
> neither gives you great tools for dealing with disk bandwidth That could, should and will be fixed sometime.
I would definitely have signed up for a competitor if DO droplets cost 400% what they do now. As other commenters have said, this space is a brutal race to the bottom in pricing. DO is great, I might pay 50% more for their current services, but not 300% more.
As someone doing proofs-of-concept and experimental work, the resource I'm trying to conserve is my own time, and Digital Ocean's value proposition on that is great. Everything works really straightforwardly and I can concentrate on my own code.
I wouldn't suggest Digital Ocean change their high-volume pricing, but the experience they provide to the small-scale user is worth a premium.
I remember at the time I suggested $5 plan should be limited to 1 per account or only for non- public internet facing usage. But the $5 plan made lots of headline and new customers during the growth at all cost stage.
So if $5 plan were really the problem that it was really their own making. Having said all of that I dont think $5 is really their concern. Hardware is cheap, and those plan with vCPU are shared and always over sold. The number of bad actor within the lowest plan are statistically quite small.
I actually think the future should be more like Render[1],
I've used other hosts in the past and had nothing but trouble. Joyent ended one of their hosting plans and I had to migrate EVERYTHING which took forever. Then Rimuhosting had an actual hardware failure that resulted in non-reproducible errors happening very frequently - that company nearly brought down my whole business. Then there was Serverpronto which had too much downtime.
by comparison, DO has been much much better, always up, always trouble free.
"Legacy Service End of Life" here are the details: "We've been analyzing customer usage of Joyent’s systems and noticed that you are one of the few customers that are still on our early products and have not migrated to our new platform, the Joyent Cloud.
For many business reasons, including infrastructure performance, service quality and manageability, these early products are nearing their End of Life. We plan to sunset these services on October 31, 2012 and we'd like to walk you through a few options."
So, I had to migrate everything to a new host which was a huge pain at the time because i was running a live Strategy game which are very time consuming to move with a minimal amount of down time, disruption and risk. not to mention everything that goes to changing over the DNS servers to point to the new address. Facebook login also added another layer of complication. It's all doable, but it takes a lot of planning and risk mitigation to get it 100% right with an absolute minimum of downtime.
Other than that, joyent uptime and performance was fine. But, having to move hosts is a big deal breaker.
I wonder if the Always Free tiers of GCP / OCI have helped DO in this regard. You can get a lot of free VMs these days, so maybe other cheapasses like myself have left the DO / Linode / etc platforms.
I feel like GCP/AWS free tier have probably hurt the $5/mo hosting business a lot.
A customer's need for additional resources should translate to a price-point question, rather than to uncertainty about what they've already paid for.
> SSDs have a limited lifetime measured in writes, and some of your customers will leave broken programs running that chew through this precious resource for no reason. If you throttle them, they'll complain.
High IO doesn't always mean an instance was compromised.
There should be clearly defined limits, and/or a clearly defined throttling policy, and the customer should have the option to buy their way out. Amazon gets this right. There should be no guessing game about reasonable use, or goodwill.
> Those pwned customers are complaining and demanding that you fix it, and the regular customers are also upset because of slowness due to the "noisy neighbor" problem inherent to all VPSes.
High CPU load doesn't always mean an instance was compromised. If you've sold CPU resources, the customer is entitled to use them. Obvious example: build servers.
If other customers experience unacceptable degradation, that means you overpromised, or else your isolation solution isn't fit for service.
Again, Amazon gets this right. They're criticised for their complex billing schemes, sometimes rightly, but it clearly makes sense to measure and be explicit about all resource-consumption. They even have an elaborate scheme to incentivise customers to tame down their CPU usage, in the form of 'burstable performance instances'.
> Anyway, I do hope they can become profitable!
Agreed. It's good to have smaller players, not just the big three of Amazon/Google/Microsoft. Competing on price-point without having the same scale, must be really tough.
they are heavily throttled on disk I/O, so much so that I had to switch to AWS and pay per I/O for one project.
also the network seems throttled to 100Mbps up/down, and a few TB/mo, something which Scaleway for $3/mo is unlimited TB/mo and at certain times 2.5Gb/s
Nope. Any hosting provider will be located at a carrier neutral datacenter or the like. At any of these you will have access to low cost IP transit providers and Internet exchanges. You can buy 100G IP transit for $5k per month, so Internet cost isn't really an issue.
I'd be curious to learn more, even Cogent is in the $0,20/mbps ballpark which would be ~$20k/month for 100G.
If you can't get proper quotes, hit me up. I can always use the residuals for brokering a 100G sale :)
Side note: I took that cluster and split it into 8. Then moved them to different geographical locations and where I could use friend or families residential connections and a cloudflare cluster to offset my cost in exchange for unlimited hosting. Very similar to Ceph actually
The founders of NordVPN have recently invested in Hostinger[1], which has successfully adopted their extremely profitable pricing model: charging for 2-4 years in advance, by default. This way, even those who would have paid $5 / month and cancelled and a few months later, end up spending $100+ for 24-48 months at once, often without having a clear need for it, thus leaving a lot of resources underutilized – and available for overselling. The company has more than doubled in size in the last 3 years, more than a decade after its inception.
And they are competing with AWS Lightsail that have similar prices and offers Windows instances for people who want it.
But even though I am very steeped in the AWS ecosystem and the price of Lightsail is competitive, if I just needed a VPS I would still go with Linode. I can’t imagine AWS’s support being good for anyone who doesn’t have a business support plan.
It's one of those situations where it's a death by a thousand cuts. When you're cost provisioning, you can figure out the big stuff, i.e., we need 25 EC2s m4a.2xlarge instances with 40TB of S3 storage, and a 2TB Aurora instance. But once you get the bill, you start seeing the costs of ELBs, NAT gateways, inter-region transfers, etc. Individually, these costs aren't significant, but in aggregate, they can make up a health chunk of your monthly bill.
Plus, their managed solutions are fucking expensive. We moved a self-managed ELK cluster to an AWS-managed one and the costs went up by a factor of two.
If DO or Linode were an option to consider, we might be able to save so much money but our own customers use and believe in AWS so we develop and test on AWS. It's a bit of a vicious cycle.
DO on the other hand from my experience has been catered towards small businesses or hobbyists who simply just don't bring that much money vs even a one giant company will bring. And since DO doesn't have all the goodies of AWS, it can't really directly compete with it for those big customers.
My impression is that DO is doing just fine. I get the sense that the layoffs are just to add to the profitability, not necessarily a sign of weakness. The article did mention that they were still growing revenues very aggressively.
http://calpaterson.com/amazon-premium.html
And they have a lot of lock-in too.
But DO cannot have the monopoly margin enjoyed by AWS and alike.
One example, the hardware cost for AWS probably will be significantly cheaper than DO. That alone can sentence Do to death.
And frankly, DO is better at UX, its technology is not innovative in any measure. By definition, that's a death penalty to a firm of its size.
Cloud hosting is low margin. You sell IaaS at about the cheapest possible price point you can. That's the very definition of low margin. Economies of scale don't enter in to whether or not it's a low margin business, they only define how competitive you can be in a low margin business.
IaaS is not where you make the money. The margins have to be tight to be competitive because that's the dollar value people see first when evaluating your cloud platform. It has an immediate effect from day one.
The profits are not made on IaaS, but on the PaaS and SaaS solutions that you, as a cloud provider, build on top of the IaaS. Things like your DBaaS, Streaming, Functions, Load Balancers, Data warehouse etc. products.
Once they're on your platform, that's when you try to get them to pivot. "Why spend engineering effort on running and maintaining database servers, when we can do it for you immediately?" Of course, then once they're using your value-added solutions, they start to get towards vendor lock in, every business's favourite situation. A customer that can't leave!
It's a difficult balancing point, you want to make it seem to the customer like they can realistically leave any time they want, but you don't want them to so you do just the absolute bare minimum you can get away with to make it seem like they're not locked in to your platform.
It seems like it took Digital Ocean a long time to realise that they need the SaaS and PaaS components if they're going to be in this for the long haul. When we launched Oracle Cloud Infrastructure some 3 1/2 years ago, we launched with features that Digital Ocean hadn't yet bothered with, and we were trying to launch with what was seen as the bare minimum to be a viable cloud product.
DO only added load-balancers in 2017, https://techcrunch.com/2017/02/14/digitalocean-launches-load..., and a Block Storage service in 2016. That's (in both cases) 8 years after AWS launched EBS (2008) and ELB (2009).
Prior to those services existing, it was relatively easy for any customer to just drop Digital Ocean for another cloud provider, but even those services aren't a big lock-in for customers.
I sincerely hope it's not too late for them. I like Digital Ocean. They really shook things up when they first hit the market, by bringing something a little different to the plate, but that was never enough to survive and it seems like they only relatively recently realised that.
That $5/mo has to cover the hardware costs: you're buying expensive physical servers to put the VPSes on, and lots of SSDs too. SSDs have a limited lifetime measured in writes, and some of your customers will leave broken programs running that chew through this precious resource for no reason. If you throttle them, they'll complain.
Then there's the support. Handling a support ticket costs you at least $3 in salary and benefits (remember, your typical customer pays $5/mo), and people will demand that you help them fix their broken MySQL server or whatever. They'll yell and threaten when you tell them that this is outside the scope of what you can do.
And don't forget security. Your customers will install broken-ass Wordpress sites and forget to upgrade them for 5 years. Then a worm sweeps through and now a whole bunch of them have been pwned and are mining cryptocurrency. Those pwned customers are complaining and demanding that you fix it, and the regular customers are also upset because of slowness due to the "noisy neighbor" problem inherent to all VPSes.
Speaking of which, preventing one VPS from hogging all the CPU or disk bandwidth is harder than it looks. The two dominant software platforms are Xen and KVM, and neither gives you great tools for dealing with disk bandwidth. Limiting CPU is much easier, but there's still the problem that you're overselling. Which is fine until half the VPSes on your machine are trying to mine Ethereum.
On the bright side: half your customers will buy the VPS, leave it running, and forget about it for years at a time. That's what makes the $5/mo business model work out.
Anyway, I do hope they can become profitable! They run a much better operation than the incumbents they replaced (slicehost, etc).