1. Will Dell servers magically appear in your racks all mounted correctly? Shipping delays is a very serious and real problem.

2. Will Dell servers arrived with 0 problems on its components? Any time I ordered meaningful amount of servers, I usually get about 5% fail rate. VMs aren't perfect but you can destroy and recreate in different region almost real time.

3. Ever had to deal with difficult-to-work-with network admin? The cloud is significantly less pain in the ass.

#3 is why we went to Amazon, and its one of the reasons we were successful. Sad as that was. No weeks/months of delays/hassles/fights/sabotage. Just success as fast as we could do our part and figure out the few small hurdles.

...That sounds more like an HR problem than a technical problem.

There are network engineers in the world who don't live in caves and gnaw at the bones of administrative assistants.

There are, but they are hard to find.

Networking is a great place for assholes to build empires and exert control. In 20 years of professional engagement with distributed and data center networks, most of these guys (and they are almost always guys) running network orgs are an impediment and spend more time helping out their vendor of choice than anything else.

I've run into awesome network guys in position of power only a few times, and they are 100x employees. The last major rollout of a service that I did was literally 3 months ahead of schedule solely because of the efforts of this awesome network dude snowman as recently put in charge.

We saw some good people, but they generally left because they didn't want to work in that environment.

Eventually replaced with someone helpful, it was amazing what we started to get done. Got lucky there we found the replacement and he stayed long enough to be there when needed.

I wonder if all these 10x network admins are now working at gcp/AWS/Azure.

Oh it was, but it was the situation we had to operate in.

And Amazon allowed to to achieve amazing things that never would have been possible and that's without the idea of things like Kinesis or Redshift that we couldn't have achieved easily/at all even without personnel problems. You simply could never get 3 new servers or drastically respec an existing one in under a day let alone minutes. We were a small shop so it's not like we had spare servers ready to go.

Please send me their resumes..

Is sabotage something you are worried about in general. Just seems like there might be deeper and serious issues if that is a primary​ concern.

It was sabotage in the 'oops this project isn't working' or 'oops this really import thing came up, no time' or 'didn't I do that?' sense. Having some plausible deniability. Work slowdown/stoppage stuff. Not actively breaking things (mostly). And it was all directed at maintaining control/power.

Worst person I've ever worked with. When I tell stories at future jobs I fully expect people to not believe me. Hell, people who weren't there for earlier incidents often didn't believer them.

Oh I believe you. I've heard of it before. Was trying to say that if that's an issue, it should be an HR fix more than an infrastructure fix. But I also imagine often that person also convinced management the are an "11x rockstar programmer"...

They were one of the hardest working people in the company. Partially that's because they worked on nonsense projects they made up, and partially that's because they were always putting out fires caused by the fact they did things poorly in the first place.

That's scares me cause I've seen that bit way too many times. There are huddled over the terminal in the evening as all managers leave "Oh look so and so is working so hard" and I am thinking "they are fixing the bug they put in just last week because they didn't bother testing it enough".

Tons of that. Each server was a literal copy of a previous one then modified to fit its new role. So we never got two servers that were the same. They'd have extra junk on them too.

Just basic docker use (or something similar) would have made a HUGE difference. Parts of the department were sent to training but it didn't happen because it would have made him less indispensable.

At one point someone higher up (but not high enough) asked how many servers we had in Amazon and the answer like 10x our physical set.

"And only one of you manages them?"

"On the side, we don't usually have to touch them."

"Then why does it take 4 people for the other servers?"

(Collectively) "Yeah. Why IS that?"

Change control was a big problem too. "Random" changes, things that didn't change at all (even when we could prove it), stuff like that causes SO MANY issues, basically all avoidable.

A corollary to #1, oops, mistakes were made, these server specs are wrong, I actually need more/less CPU/RAM/storage.

When we account for cost of,

1) remote hands

2) network switches

3) hardware components you'll need for redundancy, I've personally replaced network cards because fans burned out or HDDs because of usual failure rate.

4) when I'm done using a n1-highcpu-32 I stop using it and not pay for it, or when I need 5 of those for 2 hours I provision them and stop when I'm done running a workload.

5) upgrades to hardware or additional hardware, intel released skylake, should I buy those and try to sell my Haswell on eBay?

I'm not sure it becomes a simple price of server + colo cost comparison.

With GCP (I use daily), AWS (I used to use) or Azure, what I'm paying for is someone else to focus on things like colo leases, capacity expansion, redundancy, physical security and provide higher order services - load balancers, BigTable/BigQuery, CloudSQL/RDS, GKE, DataProc.

This allows me to run all of infra as close to capacity as possible while avoiding any wasted cycles

However I do miss going to data centers and hear my machines hum and blink, that is quite the feeling.

> You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month.

And then magically maintain it for free?

hopefully with your spare 3437 dollars a month you could figure out how to ever maintain such a thing.

... per server.

It really, really depends on on your use case, but there are some scenarios where bare metal just makes more sense. Also there are ones where cloud makes everything much easier.

North America-oriented B2B app that's a JavaScript SPA witch connects to some lightweight services fronting a database? I think that's a great contender for the cloud.

500-node Hadoop cluster running 24/7 with heavy load? You are going to see some huge savings by maintaining a Datacenter. You can use the extra 1.7MM/mo. to set up some robust networking equipment and have few people on-site.

This.

Like every IT trend, "Cloud ops" is the trendy mind-slug that eats everyone's brain and makes them say that everyone who doesn't jump on the bandwagon is going to die broke and unloved in the gutter.

Remember how XML meant massively reduced integration costs and firing all your developers because GUI tools would let business managers connect pretty boxes and then kick back to watch the graphs go up and to the right as business exploded?

Yeah, so AWS is great, if you're building something that works well on AWS. And that's a large class of apps - if you're throwing up a Drupal front end to a CRUD line-of-business backend, need to push notifications and send some email, sure. You're dead in the middle of their target market.

Doing something interesting? Doesn't even have to be as massive as the parent suggests - if you're doing something that blows any of the billable parameters out of the sweet-spot, like bandwidth, storage, latency requirements, things that depend on system-locality, etc., you're much better off building, and using AWS for the pieces which can be broken off that don't hit the pain points.

Correct!!!

Where can I find competent sysadmins capable of managing a system like that on their own for $40,000 a year?

Doesn't make much sense for one server... but if you have 5, then you are looking at $200k. Then as you go up the numbers start making more sense.

That might get you one admin (we office workers have a mistaken belief that our salary is most of our cost. It's somewhere just north of half.).

What do you do when they're on vacation/sick/sleeping?

Thing is, I really want us to host our own stuff. For one thing if you don't host anything you lose all competence and then you really are beholden to your cloud provider.

But it seems to be that we are in this situation until someone figures out how to comodify servers - and server orchestration - to the point where this stuff gets cheaper to manage again.

This pendulum always swings. No sooner will we control our own hardware again than someone will come up with a new way to centralize it.

Totally, the thing that's super cool about the amazon instances you don't have to do anything with users, authentication, authorization, networking, dns, backups, monitoring, logging, firewall rules, security patches and upgrades.

With a bare metal server you have to manage all of that.

Oh my goodness man, have you managed Amazon instances? Setting up VPCs, IGW, IAM, DNS, EBS Snapshots (and deletions until recent lifecycle), custom metric in coudwatch, and Security Groups is a seriously challenging task just to learn the basics, much less do it right, much less automate it.

It's not a bed of roses in the physical world either, but you're simply wrong to say it's easy in the "Cloud" and hard in a DC/Bare Metal.

(it was a joke.) There are a zillion things to think about if you're going to have even one machine public on the internet, regardless of the hosting solution. At a small organization there's always that one programmer that's like half sysadmin.

I find it amusing that people pretend without bare metal, you'll get all of that person's time back.

I missed it. Thanks.

It's not as if using AWS etc eliminates the need for a good sysadmin...

You can buy sysadmin services. This isn't a full time job at that level.

Then you are just trading one fee for another, but the "sysadmin services" you are ordering most likely has slower response time, less resources, less knowledge, and worse uptime guarantees than someone like aws.

Yeah, at the larger sizes it might make sense, but not for everyone.

Nope you have a number for the person that provides the service and you can call her/him. Unless you are at several million per month spend at AWS have fun getting a hold of someone when sh#t hits the fan.

Good luck hiring someone on a demand hourly basis that knows your stuff and won't break your stuff.

Hmm never had an issue, but to have people at AWS that know my stuff would start somewhere north of 100 million a year spend on AWS.

At smaller sizes too. If you just have one server, it's unlikely that the slower response time will loose you 3000 dollars per month which is the savings of dedicated service versus aws. Many small businesses run on leased hardware with on site support. Rackspace for instance made their fortune in that sector.

Well you're not comparing to AWS without ops, you're comparing to hiring someone for non-AWS vs managing AWS on your own. Outsourced ops know your systems. They have contracted guarantees. They can manage your own systems, hosted systems or even AWS. Using not-AWS saves you enough to pay for that.

> Outsourced ops know your systems.

I worked for a MSP for many years, and was well acquainted with our regional competitors. Our uptime and theirs was nowhere near AWS-level.

You're getting a different engineer every time you pick up the phone. They most certainly do not know your systems like an internal team would. I regularly saw cascading/circular issues caused by lack of familiarity and/or poor change management.

Owning and administering your own iron makes sense at a certain scale, but it's a much bigger scale than most companies will reach.

That means you only have about $40k/yr to hire a sysadmin assuming they need no supplies or tools...

$40k/yr/server

With correct pricing (3-year commitment), you're actually looking at about $8k/yr/server difference, for 64-core servers. If you're at the point where your workload requires ~12 64-core servers, that difference will net you $100k, which you can spend on the ops staffer.

Can probably hire a someone at minimum wage to watch it 24/7 in case it spontaneously combusts.

Probably combusted because he plugged the 230v power supply to the 600v plug.

Minimum wage => You get what you pay for.

I'm not sure which region you priced, but an m4.16xlarge in us-east-1 (Virginia) is $3.20/hour, or $2304/month.

But if you are comparing it to a purchased server, you ought to be looking at Reserve pricing, a 1 year reserved instance is $16K/year, 3 years is $32K (or you can pay monthly if you want).

Still not as cheap as owning the server, but not quite as bad.

I had selected a dedicated instance. It's been my experience the difference in performance between dedicated and non dedicated is dramatic.

edit: I had selected a dedicated instance in order to compare apples to apples. By definition you don't share a colo server.

With an m4.16xl, there's no effective difference between a dedicated or default tenancy instance - only one .16xl can run on a physical hypervisor, which is reflected in the pricing, it's $3.20/hour whether it's dedicated or not:

https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst...

For smaller instances, there is around a 10% increase for dedicated, for example, for an m4.4xl, the price goes to $0.88 for dedicated tenancy versus $0.80 for default.

There is a $2/hour fee for each region in which you have dedicated instances, but that gets lost in the noise if you're using significant AWS resources. (it's $2/hour regardless of how many instances you're running)

We haven't noticed any significant difference between performance on dedicated and non-dedicated instances, and only use dedicated instances where required for compliance/regulatory reasons.

An m4.16xl has 8 physical cores; what hypervisor does AWS buy that only has 8 physical cores?

My smallest physical servers have 8 physical cores.

An m4.16xl has 32 cores / 64 threads.

okay, my error - thanks!

Why get a dedicated instance and not get reserve pricing? You can get those instances for a fraction of what you're currently paying

OVH has absurdly low prices on both. If you have a baseline load take a look. Their enterprise offerings are pretty rock solid and they have multiple data centers with direct fiber backplane interconnects.

IMHO the big cloud providers only make sense if you have a highly elastic load or if you are making extensive use of their as-a-service higher level offerings. If you're running baseline load bare metal destroys them.

I used to use OVH on occasion, but I was really disappointed with their North American-based tech support. Didn't seem particularly plugged-in and English proficiency was shaky. (At this point I'm in AWS because I don't really care about low-point performance and would rather spike up when needed.)

Unfortunately OVH doesn't have any data centers in the US although they seem to be working on them.

Montreal has low pings to the US.

You're paying for the staff time to set up the lower-level infrastructure and keep it running at _least_ as reliably as if you had your own staff on bare metal, quite likely more so. (including scaling it when you need more etc).

It is interesting that they end up charging you this for _bandwidth_ though. Their staff time doesn't actually probably scale with bandwidth, but their prices do.

I think it's probably kind of a way of pricing based on a proxy for customer size, likely revenue, and ability/willingness to pay. Rather than as a % margin on actual costs.

I just landed a few racks of bladecenters and storage. High 7 figure deal. This was probably the best implementation project ever... everything ran on time, the vendor delivered everything on time, facilities was 100% ready and pre-cabled, etc. I'm incredibly proud of our team -- our biggest issue was someone ordered the wrong SFP. Butter.

Timeline: 120 days procurement (including product evaluation and PoC), 30 days install activity, 10 days buildout. 10 days software build, 15 days testing.

That's a little over 6 months. A ton of vendor/build engineering/tester time. And we'll be at 50% capacity for another 4 months. Or... we could spin the whole thing up on AWS/GCE/Azure in < 30 days and only consume what we need when we need it.

Which hardware vendor did you go with?

> A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx)

Cherry-picking a bit there. You're entering a 3-year commitment with the Dell lease, but not with the EC2 lease. Take a 3-year RI on that EC2 instance, and the price drops considerably. If you also take away the 'dedicated' requirement, the 3-year RI is about twice as expensive, not ten times.

Edit: correction from Johnny555 above: this top-tier server doesn't have a 'dedicated' option, so there's not an 'extra' cost for that. It's just twice the price, if you do a 3-year commitment (as you are doing for the dell)

you can also pickup an enterprise flash device and have a few terabytes of 500k+IOPs for a few grand whereas reaching a similar performance profile via provisioned IOPs will cost you 10x as much every month.

You're comparing apples to oranges -- if you want a couple TB of locally attached storage, you'd get an i3.2xlarge for $450/month with 1.9TB of NVMe SSD (plus 8 cores CPU + 60GB RAM).

Using EBS volumes with provisioned IOPS give you some advantages that you don't get with a bare SSD -- like snapshot support and the ability to easily detach it and move it to another server.

you're right, the performance of the local storage on i3 instances is a lot better than normal EBS but still nothing close to what you would get from just buying an Intel P4500 or similar though.

http://www.aerospike.com/benchmarks/amazon-ec2-i3-performanc...

Double the P4500. The raid is not gonna store data out of thin air.

And add a third one while you're at it.

Given the price of the server and the data you have on it, it's important enough to warrant a hot spare.

It's not the hardware you're paying for, it's the services. How much does it cost to employ a systems administrator? A lab tech? And so on. You see exactly the same pattern in software services. People pay more to buy products that are "turn key" versus having to hire someone to do tons of work building out systems or services or what-have-you.

Also, hiring people is a risk, and it doesn't happen instantly. When you have a gap of months without a sysadmin things are bad. Hiring a bad sysadmin is even worse. Not to mention the impossibility of hiring 1/2 or 3/4 of a sysadmin. There's scaling limits there. If you have the resources to DIY, that can be a key competitive advantage, but not everyone does.

Often, you still need a "systems administrator" with the cloud. Except, now he calls himself a DevOps guy and charges a lot more. ;) AWS isn't going to configure itself.

Use google cloud or softlayer and the same machine is $1500 a month.

It's no secret that AWS is a rip off on instance pricing. Use the competitors if you worry so much about pricing.

How does that work out? Looking at the per-hour figures in AWS and GCE, they're similar for a 64-core, quarter terabyte RAM machine ($3.20 AWS, $3.40 GCE). Both vendors have ways to get your bill discounted (with google's being easier). AWS's instance pricing has been fairly comparable to other vendors for a couple of years now; it's their bandwidth that's costly.

What are most people running anyway? Using this kind of behemoth server as a 'typical' example seems out of line.

Google n1-standard-64 => $1710

AWS m4.16xlarge => $2592

Google base is slightly lower and it gives you 30% off automatically on instances that run 24/7 for the month.

That's typical of on premise deployments. You get big servers because they are so annoying to deploy you might as well fill everything possible in the box.

On the clouds, you would create a group of VM per service, with appropriate size for the service. It is significantly more efficient.

Isn't that $1710 already including the 30% discount? (It works out to $3.4 x 30ish days x 70%). You get similar discounts on AWS, but you have to commit for a year - given the OP's intent of leasing a Dell server for three years, this shouldn't be dismissed.

[Off Topic] I have a chrome extension that replaces all occurrences of "the cloud" with "my butt".

It is the best extension.

Does it also replace 'millennials' with 'snake people'?