How and Why Swiftype Moved from EC2 to Real Hardware

Re:How and Why Swiftype Moved from EC2 to Real Hardware by stephen-mw on Monday March 16, 2015 @02:59PM (#9213507)

Here's my anecdotal, one-data-point experience from moving a giant EC2 environment to datacenter:

1. Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers.

2. Any weirdness you experienced with AWS infrastructure will be replaced with weirdness in your own environment, except now you're on the line to troubleshoot and fix it yourself.

3. Operation staff will immediately start guarding the food bowl as resources become finite. Server provision waits start to seem like breadlines. Power is consolidated with Those Whom You Must Ask.

4. Your cost will decrease, sometimes significantly so.

5. You'll have more hardware flexibility to run your app just the way you want to (Stack Overflow's mega databases come to mind[0]).

In the end I think this type of transition is for stable companies that don't mind or even prefer strong divisions of labor (coders who code, sysadmins who sysadmin, testers who test), but it's not for startups or companies that hope to move with any kind of strong velocity.

[0] http://highscalability.com/blog/2014/7/21/stackoverflow-upda...

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by Negitivefrags on Monday March 16, 2015 @04:29PM (#9214171)

Contrast that to my experience with renting from Softlayer:

1) The operational overhead is the same as Amazon. Actually I think it's less because everything is so predictable. Every machine we get has identical performance to any other. We still don't have to care about the exact same list of things that EC2 provided but we also don't have to care about weird cloud issues.

2) See above answer but also Softlayer are happy to provide actual support for any issues in a timely manner. In general everything has been much more reliable that you actually have to think less about High Availabilty technologies that make your stack more complex. In the years I've been using them we have only had a few hard disk failures that were replaced in around an hour and we just failed over to a replicated slave manually for around 2-3 minutes of downtime.

3) Resources are no more finite than EC2. The only difference is that the provisioning time is 1 hour rather than 1 minute. That has still been perfectly fine to respond to unexpected load events.

4) Absolutely

5) Also great.

People often compare Cloud Hosting with Colo. What they should compare cloud hosting with is renting servers. You get 95% of the benefits of cloud with none of the drawbacks.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by snewman on Monday March 16, 2015 @06:46PM (#9214994)

Can you walk me through a concrete example of cost savings? Every time I look at something like SoftLayer, it doesn't seem to offer much of a savings compared to a corresponding EC2 reserved instance. Sure, you can do way better than EC2 on-demand instances, but the reasonable comparison is to a reserved instance.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by zaroth on Tuesday March 17, 2015 @02:25AM (#9216563)

I think a big part of it is that dedicated metal and "reserved instances" are not apples-to-apples. You want absolute control, consistency, and customization of the build, you can buy/lease metal and co-locate it, or you can rent it from someone like SoftLayer, and you still don't ever have to walk into a data center.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by Sidis on Tuesday March 17, 2015 @09:08AM (#9217601)

Inter-DC network traffic is not rated/charged for SoftLayer. The key cost differentation is network charges when contrasting SoftLayer other services.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by chx on Monday March 16, 2015 @03:02PM (#9213525)

> Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers.

Why? What are you talking about? You are hiring servers you are no colo'ing them. Networking them is not your problem. Your responsibility still starts from a root prompt just there's no VM layer between that and the physical server.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by kovyrin on Monday March 16, 2015 @03:18PM (#9213661)

Yes, it really does look like @stephen-mw is talking about a migration to an owned/colocated hardware. And that's a very different kind of hairy mess I'd prefer to stay away from as long as humanly possible.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by chx on Monday March 16, 2015 @03:29PM (#9213737)

OMG, colo. I've been the architect of a Top 500 website acquired by a Top 100 and even for that we just rented a few servers (you'd be surprised how few -- a dozen or so). I do not really know how big you need to be make colo a sensible choice but quite probably much, much bigger than most think of. We ran the numbers and they weren't pretty.

And for cloud: we purchased video conversion as a service, that one ran in the cloud. I can see how that makes sense.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by merb on Monday March 16, 2015 @03:34PM (#9213770)

and if you have more than one? or more than one dc? somebody needs to connect it, or you will need a VPN. it's not that easy without Clouds if you need connected servers. We switched to AWS since connecting servers in a dc isn't as cheap as people think of.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by corford on Monday March 16, 2015 @08:18PM (#9215485)

I host all my stuff with Hetzner (who are at the cheap end of the market compared to Softlayer) and even they provide pre-configured VLANS and private switches. For intra-DC you can just setup some redundant openvpn links or pay the dc to configure a hardware tunnel between both sites.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by kovyrin on Monday March 16, 2015 @05:07PM (#9214397)

We get free inter-DC connectivity for free with Softlayer. There really is no difference for us between connecting to a server within a DC or between datacenters (aside from added latency, of cource) – the same private address space, the same access rules (we see all of our machines in a private backend network from any datacenter).

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by Negitivefrags on Monday March 16, 2015 @04:19PM (#9214093)

This article was talking about renting servers from Softlayer. All servers on your account in Softlayer have a private VLAN that connects them, even between data centers. Private networks for your servers are a feature of many server hosts.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by nasalgoat on Monday March 16, 2015 @05:57PM (#9214717)

Every enterprise router in existence has built-in tunnel support. Even the open source ones like PFSense.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by twblalock on Monday March 16, 2015 @03:50PM (#9213899)

Operation staff will immediately start guarding the food bowl as resources become finite. Server provision waits start to seem like breadlines. Power is consolidated with Those Whom You Must Ask.

This is the worst part about moving out of the cloud, especially since cloud computing has moved a lot of ops and deployment responsibility to developers.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by rodgerd on Monday March 16, 2015 @09:05PM (#9215642)

I would say an absolute pre-req for moving from Amazon to own servers would be tooling like VCAC or OpenShift to give people a nice self-service experience.

[ Parent | on HN ]

Re:How and Why Swiftype Moved from EC2 to Real Hardware by wmf on Monday March 16, 2015 @09:16PM (#9215698)

Then you get "out of capacity" errors even sooner. You could have a pretty bad experience with an internal cloud that was self-service but wasn't run like a service on the business side, so the fixed annual budget was exhausted immediately.

[ Parent | on HN ]