News for hackers. Stuff that matters.
Re:A million-dollar engineering problem by nik736 on Thursday March 16, 2017 @01:54PM (#13887218)
Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.
Re:A million-dollar engineering problem by rbranson on Thursday March 16, 2017 @02:49PM (#13887837)
It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case.

Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows down the decision-making process and seriously complicates experimentation. The number of people who know how to build out datacenters at the scale of thousands of machines is vanishingly small.

We'd also need to replace IaaS services like ECS, ELB, ElastiCache, RDS, and DynamoDB. There are certainly off-the-shelf replacements, but we'd need to build-out the expertise within our teams to operate these systems. We're talking roughly a dozen or so engineers working full time for many, many months to get these systems in place from scratch, on top of the even larger effort to design and build out datacenters. I'd much rather plow those cycles into efforts like expanding to multiple regions and improving reliability of internal services. That's a much better return on investment for our customers.

Right now we're in the sweet spot for the cloud. We're way too big to run on trivial amounts of hardware, growing at a rate that makes it difficult to stay ahead of demand in a datacenter-centric world, and too small to justify investing in a scalable hardware build-out.

Re:A million-dollar engineering problem by QuinnyPig on Thursday March 16, 2017 @03:41PM (#13888440)
> It could be if our workload was relatively stable

+1. If your entire environment is steady-state, it may be worth considering a migration to bare metal.

That said, you can expect:

* Worse uptime

* A fair bit of retraining / hiring different skillsets

* A loss of engineering focus as they work on the migration instead of feature expansion

It can make sense economically, but that discussion goes well past "the monthly AWS bill."

Re:A million-dollar engineering problem by user5994461 on Thursday March 16, 2017 @04:41PM (#13888976)
Try all the same on Google Cloud, you'll save half the money compared to AWS, probably more.

Disclaimer: I am NOT affiliated to Google in any way.

Re:A million-dollar engineering problem by patrickg_zill on Friday March 17, 2017 @10:50AM (#13894354)
I would bet you 1000 USD that it would be eye-opening to buy a 64gb RAM, 8 core CPU server, put 4 SSDs in it in RAID10 config, and test your entire stack on just that box using Docker, KVM, whatever. Just running it beside your desk and having your other devs beat on it.

You might have to stub out some things using e.g. Cassandra or another open source alternative; just use a Docker or VM image without the production redundancy.

You did admit, that you have no idea how much performance is being dropped on the floor due to not using dedicated hardware. So why not test it?

My belief: you would find you are paying AWS far more than they are actually worth.

Re:A million-dollar engineering problem by rbranson on Friday March 17, 2017 @11:22AM (#13894628)
I've operated on-prem and colo physical data centers. I've also been part of larger companies where someone else deploys baremetal on my behalf. I've been running infrastructure on top of AWS off-and-on since 2008. I've done a live migration of one of the largest Internet sites from AWS to baremetal in data centers. I feel pretty good about keeping our infrastructure in AWS for the foreseeable future.
Re:A million-dollar engineering problem by patrickg_zill on Friday March 17, 2017 @11:49AM (#13894877)
I apologize; I didn't read that part of your bio on the site.

Your post seemed to indicate that you hadn't done any calculations for the bare metal server with your own vms scenario. But clearly you have, if only mentally.

Re:A million-dollar engineering problem by rbranson on Friday March 17, 2017 @02:55PM (#13896852)
No worries! I'm not super into self-promotion, but I do think my background is relevant in this case. Either way, the decisions that go into a strategy like this involve a mountain of variables.
Re:A million-dollar engineering problem by lafay on Thursday March 16, 2017 @04:06PM (#13888671)
This article makes a good case for having at least one early team member who knows how to build out physical infrastructure at scale, even if you initially decide on AWS / GCE / Azure:

http://firstround.com/review/the-three-infrastructure-mistak...

Re:A million-dollar engineering problem by rbranson on Thursday March 16, 2017 @04:29PM (#13888879)
This might sound a bit extreme, but contrary to most FRR pieces this is garbage advice. Startups should focus first on getting their product-market fit before trying to future-proof themselves from a future that doesn't yet exist.
Re:A million-dollar engineering problem by user5994461 on Thursday March 16, 2017 @04:44PM (#13888995)
Register your startup to Amazon or Google, you'll get 100k of free credits.

Keep a look on your bill because you don't want to run anything. But take advantage of having no hassle for your first years.

Re:A million-dollar engineering problem by asptimothy on Thursday March 16, 2017 @02:05PM (#13887330)
I was curious about that myself. What's the best way to model the costs associated with maintenance overhead that comes with bare-metal, vs the savings from managing bare-metal servers using a service (like packet), co-locating your own hardware or running your own datacenter.

My gut feeling is that you have to get to an extraordinary size to realize any meaningful savings, but that's primarily based on Dropbox's migration off AWS (https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...).

Re:A million-dollar engineering problem by user5994461 on Thursday March 16, 2017 @04:50PM (#13889040)
It's rather simple, you count the days you are wasting on handling the low level hardware.

The entry cost is £500 per day or $1000 for an engineer.

5 hours on the phone to find the hardware and agree on the order with DELL + an afternoon with customer support because they shipped the servers without hard drives + your project is delayed by an entire week because you don't have the resources => 1 day + 1 day + half a week.

These things would have been 5 minutes on a cloud.

Re:A million-dollar engineering problem by QuinnyPig on Thursday March 16, 2017 @04:04PM (#13888654)
It depends upon your constraints.

As a for-instance, I was asked this exact question a year or so ago about an on-prem object store instead of S3.

The break-even price to go multi-region was ~15PB on paper; that included DC space, hardware, software (build vs buy was another factored discussion), and the staff to run it.

That assessment was delivered with the caveat that their uptime was not going to approach S3's by any stretch of the imagination-- and their infrastructure outages weren't going to sync with "most of the internet's."

It's a complex topic, and there are many hidden costs...

Re:A million-dollar engineering problem by GABeech on Thursday March 16, 2017 @02:18PM (#13887454)
That's a big giant "it depends."

It depends on what your service looks like: CPU intensive, Memory Intensive, Storage intensive? (In reality some unique mix).

You probably won't see a huge savings year one, as you'll be spinning up a lot of new things and have a fairly large CapEx expenditure. Now if your growth pattern is steady/predictable then you should be able to plan out your hardware buys or do a hybrid solution to handle traffic bursts.

One of the nice things about running your own hardware is that there are some costs that are easier to control. Don't need new hardware? Don't need to spend on new hardware for example.

You also have much more control over your environment so you are able to really optimize your code, and infrastructure so that you don't need to scale as large system wise.

But, back to the question on how to model it? You just gotta dig in, and make some educated guesses about performance,test and repeat.