News for hackers. Stuff that matters.
Re:We're moving. Goodbye Rackspace. by adriand on Monday November 08, 2010 @08:38PM (#1884903)
This is a rather unpleasant review of a service we were about to move over to.

For companies that have moderate performance requirements (e.g. visitors in the range of 30k or 40k per day across a range of web apps and sites), reasonable but by no means expert level server administration skills, the need for a redundant environment to satisfy SLAs with clients (e.g. two app servers + load balancer + master/slave db servers), and the desire to focus mainly on software development instead of server admin, what companies does the HN community recommend?

We've been considering Rackspace Cloud and Linode, but are open to any suggestions. We also have a quote for a standard, managed four-server + hardware load balancer deployment in front of us but it is pricey ($3000+/month).

Re:We're moving. Goodbye Rackspace. by nwilkens on Monday November 08, 2010 @09:38PM (#1885067)
Softlayer (or AWS) + outsourced server management is great fit for your scenario. Softlayer has great HW / network, and can provide a good base for a scalable infrastructure. AWS EC2 also sounds like a great fit in your environment.

Regardless, having a great server admin guide this implementation will be a key part of the chosen architecture success.

I run a server management company, so I have a bit of a bias ;)

Re:We're moving. Goodbye Rackspace. by sanswork on Monday November 08, 2010 @08:58PM (#1884955)
I've used Rackspace, Amazon and GoGrid. I'm currently almost entirely on GoGrid and just finishing up migrating back off of Amazon for the second time(Migrated some services back to them when they opened west coast).

There have been some stability issues in the past year(instances going down for maintenance regularly) but haven't had a problem with that in a while and was advised it was due to them upgrading all the servers in their cluster.

The ability to mix dedicated and cloud systems easily has been the killer feature for me. We also prepay and the number of servers we get for the cost is pretty good.

Re:We're moving. Goodbye Rackspace. by MichaelGG on Monday November 08, 2010 @09:23PM (#1885032)
Have you looked at just buying the hardware and colocating it? $400 should get you half a cabinet. Bandwidth can be found for a few dollars a meg. As you're buying redundant servers, you can get refurb/ebay stuff and save on the upfront costs (who cares about next business day service replacemnt if you have spare parts?).
Re:We're moving. Goodbye Rackspace. by lsc on Monday November 08, 2010 @09:42PM (#1885072)
I wrote a thing about the pros and cons of doing that, but instead I thought it would be better to give you some objective numbers.

1 full rack with 15a of 120v power (remember, don't exceed 75% of a circuts rated potential, so that's really 11.25a) at he.net in freemont is going to set you back $400/month. $500/month with 100Mbps bandwidth. (more than you will need for now)

Now, you are talking between 5-7 low-power servers there.

You want to buy a metered/switched PDU ($400 or so new, $150 used)

one of my 8 core, 32GiB ram opterons with 4 disks is about $2500 in parts, new. I can't imagine anyone charging more than $500 to put the thing together.

Note, unless you have really cheap hardware guys and downtime costs you little, I'd avoid used hardware. I started on used servers, and I no longer buy used hardware that is user-serviceable. (I avoid used servers. But I may buy used switches.)

If you need racker/stackers I can point you at experienced guys who will be happy to work for $50-$60/hr, and people who can get the job done at half that. If you want more advanced hardware troubleshooting, double that for someone really good. but for that kind of money they should be able to set you up with an automated provisioning system. for comparison, I'll work for $100/hr if you are willing to deal with my inconsistent schedule.

Setting up a server should not take more than a few hours, including hooking up the serial console, etc... And hardware problems? figure on once every two years per server.

a lot depends on the quality of your provisioning setup (something like cobbler is good) and your serial console/rebooter setup. But it /can/ be a lot cheaper than 'the cloud' in many instances.

edit: added the bit about used hardware.

Re:We're moving. Goodbye Rackspace. by MichaelGG on Tuesday November 09, 2010 @01:48AM (#1885637)
Right, so, if they were quoted $3K a month, in a couple of months of self hosting, they'd have definitely recovered their investment, and then start saving a few K a month.

Your point about onsite hands is something I forgot. In the companies I've had, I've always had offices adjoining the datacenter. (I've been lucky to end up with such great spaces.)

If you don't have onsite hands, then fixing even minor hardware issues can be a major pain. If you do, then cheap users servers are fine, if your software is fault tolerant. If a machine goes down, others takeover in a minute and you have slightly less capacity. No big deal.

I just see these crazy high numbers for hosting, especially "cloud" stuff and don't get it. Cloud in particular seems to only make sense if you're actually elastic, or scaling up very fast. Amazon and so on's pricing, for "always on" servers, does not look that appealing.

Re:We're moving. Goodbye Rackspace. by lsc on Tuesday November 09, 2010 @04:39AM (#1885928)
Personally, I think the very best use case for "cloud" is backup servers. Think of it, run a 'small' instance to slave your database and other stuff to EBS storage, and once a month practice bringing up production on "the cloud" - cheapest and most complete backup system I can think of.
Re:We're moving. Goodbye Rackspace. by lsc on Tuesday November 09, 2010 @04:44AM (#1885940)
As for onsite hands, 95% of what you would go to the co-lo for can be done from your comfy chair if you have a good serial console/rebooter setup. Also, the he.net price includes "remote hands" service. Note, don't expect more than "remote hands" and make sure you clearly label everything... but you can usually avoid leaving the office.
Re:We're moving. Goodbye Rackspace. by adriand on Tuesday November 09, 2010 @09:16AM (#1886448)
I really appreciate your insight on this. I'm going to take this back to my team and discuss. Thanks.
Re:We're moving. Goodbye Rackspace. by 3pt14159 on Monday November 08, 2010 @09:33PM (#1885060)
Don't. It isn't worth it. Just use AWS or Rackspace or Linode. Outsource hardware management to people that do it for a living.
Re:We're moving. Goodbye Rackspace. by lsc on Monday November 08, 2010 @09:57PM (#1885099)
why not outsource your OS management and use shared hosting, engine yard or appengine? hardware guys willing to do the work of amazon can be had for very little money, even on a short-term contract basis.

there are situations where different levels of outsourcing are appropriate. And certainly, if your cost per compute node doesn't matter, few will fault you for going with amazon. But you should be aware that you are paying for that, and in some situations, having a high cost per compute unit will kill you.

Re:We're moving. Goodbye Rackspace. by 3pt14159 on Monday November 08, 2010 @10:15PM (#1885142)
This is a fair question, so I don't see why it should be downvoted.

The reason outsourcing hardware to AWS or Rackspace makes sense versus hiring hardware guys (usually) doesn't is that you introduce one more link in your failure chain. In My opinion it is far, far more likely that you and your hardware guy are going to mess up ZFS configuration somehow and have it fail when your app gets black swan traffic than Amazon going down. When it comes to Engine Yard, I'm all for that too, in the right circumstances, but you get fairly high lock in. I agree that you get that with AWS too, it just seems more likely that AWS will be around 10 years than it does Engine Yard. But sure, Engine Yard and Heroku are great.

Re:We're moving. Goodbye Rackspace. by lsc on Monday November 08, 2010 @11:08PM (#1885279)
filesystem user error can be just as deadly on aws as it can on your own hardware. either way, you need good backups.

While I agree that amazon itself is unlikely enough to go down that you can pretty much bet the company on it, any one server at amazon may go down at any time. Now, amazon has the huge advantage of being able to give you another server at any time, but you need to be prepared for that. any data on your local disk would be gone, and you need to be able to re-connect to your EBS store from only one instance. (many, many of the 'admin error' cases of data loss i've seen stemmed from mounting a network block device read/write on two different servers without a clustered file system.)

My point is just that amazon doesn't free you from the need for (and mistakes of) *NIX admins (like heroku, app engine and engine yard should.)

Re:We're moving. Goodbye Rackspace. by count on Tuesday November 09, 2010 @08:06AM (#1886273)
That's a good point - you could argue it doesn't free you from the need of a good admin, and it actually makes it more difficult to hire someone who knows what they are doing; good cloud SA skills are much more rare than good traditional SA skills.

When you can just drive to the datacenter and toss in some more disks or drop your new F5 on there (or hack it together with CARP), but instead have to be familiar with exactly what the restrictions and use cases around the cloud providers offerings are; the field of candidates just got painfully smaller, and probably more expensive.

Re:We're moving. Goodbye Rackspace. by jfine on Tuesday November 09, 2010 @03:38PM (#1887817)
Heroku and Engine Yard (for their cloud service) are just front ends to AWS anyway. Your basically paying them for the luxury of not having to deal with AWS. So as long as they don't get undercut by a competitor and all their customers flee, they should be around for a while. That or their owners get greedy, sell the companies, and the new owners move to "more economical hosting". Na, that never happens. ;-)
Re:We're moving. Goodbye Rackspace. by lsc on Thursday November 11, 2010 @01:40AM (#1893110)
right. Just like you are paying AWS for the luxury of not having to deal with hardware.

Both can be rational decisions (or not, depending on many factors) I'm just pointing out that the people who deal with racking and stacking (the sort that amazon allows you to fire) are rather a lot less expensive than Linux SysAdmins, the sort who app engine, heroku and engine yard allow you to fire.

Re:We're moving. Goodbye Rackspace. by jackowayed on Monday November 08, 2010 @10:33PM (#1885186)
Depends what your app is written in.

If I were doing this with a Rails app, I'd use Engine Yard for sure. Engine Yard Cloud makes stuff like load-balanced app servers and master/slave replication really easy (you can do it from the web interface), and they really know what they're doing and have tuned the stack based on that knowledge.

I don't know if there are similar offerings that are reasonably priced yet awesome for other languages though.

Re:We're moving. Goodbye Rackspace. by gommm on Tuesday November 09, 2010 @01:09AM (#1885561)
I found engineyard very unflexible... Sure you can create custom recipes to get around that but since they update their system without warning, the custom recipes end up becoming stale...
Re:We're moving. Goodbye Rackspace. by nivertech on Tuesday November 09, 2010 @02:58AM (#1885774)
Both EngineYard and Heroku run on top of Amazon EC2
Re:We're moving. Goodbye Rackspace. by gommm on Tuesday November 09, 2010 @06:21AM (#1886104)
Yes, but the problem is that they use their own gentoo based environment that they update as time go...

To do anything custom that they didn't plan beforehand, you need to create custom chef recipes that will be executed whenever a new instance is provisioned or if you deploy with their web interface (but not if you deploy with capistrano).

The problem is that it's hard to make sure that the custom chef recipes will work when the underlying environment changes and you are not told ahead of time of those changes...

Re:We're moving. Goodbye Rackspace. by stcredzero on Tuesday November 09, 2010 @09:32AM (#1886506)
What sort of stuff was in those recipes?
Re:We're moving. Goodbye Rackspace. by madaxe on Monday November 08, 2010 @09:09PM (#1884991)
Linode are wonderful - best support I've had from anyone, ever. Nice instances, too!
Re:We're moving. Goodbye Rackspace. by tlack on Tuesday November 09, 2010 @05:26AM (#1886022)
+1. In the past 24 months I've used Rackspace, ThePlanet, SoftLayer, Slicehost, Linode and the terrible Mediatemple. I'm doing all future development on Linode. Their basic VPS is pretty quick for the price and it's online almost instantly. It's still just basic Linux, so you can rig everything you want - no awkward managed interfaces to worry about.

I would consider switching to a different company, but for now Linode scratches my itch better than the rest.

Re:We're moving. Goodbye Rackspace. by mt_Sara on Tuesday November 09, 2010 @03:51PM (#1887860)
I'm sorry to read that you've had a less than stellar experience with us. May I ask what happened?

-Sara at (mt) Media Temple

Re:We're moving. Goodbye Rackspace. by tlack on Tuesday November 09, 2010 @05:35PM (#1888247)
Create an account. Open a service ticket. Wait hours for response. Plus no firewall options on your Nitro servers, at that price range, is ridiculous and way below the standards of the industry.
Re:We're moving. Goodbye Rackspace. by catch23 on Monday November 08, 2010 @09:32PM (#1885055)
another +1. we moved to linode from rackspace. They don't have that crazy backup limitation that rackspace does, and they're always open to custom inquiries. Rackspace has the big company mentality -- if our interface doesn't have an option for it, it doesn't exist.
Re:We're moving. Goodbye Rackspace. by chopsueyar on Monday November 08, 2010 @10:49PM (#1885222)
I second. Moved from Rackspace to Linode.

The bandwidth overage charges are much more reasonable, too.

Re:We're moving. Goodbye Rackspace. by easp on Wednesday November 10, 2010 @12:41PM (#1891143)
I like Linode, but they are wearing a little thin for me.

In general, uptimes have been excellent but the physical server hosting one of our VPSs had three emergency reboots in a little over a month before they replaced the hardware. Perhaps they'd already planned to do a replacement after th e second incident but I had no way of knowing.

Which brings me to my second beef, which is that they don't share much when closing a ticket. Their explanations are pretty prefunctory. In some cases, like transient network issues I generally can't tell if they've investigated the problem and done something about it or if the first-tier tech just takes a quick look sees it's working now, and punts it back to me to let them know if there are more problems.

Similarly, after a major screwup that took down most of the virtual servers they host (including the ones that hosted their own website and web-based management tool), all we got was a brief explanation of what had gone wrong and a status blog hosted on someone elses infrastructure.

What I didn't get was a sense of how they were going to guard against anything similar happening again.

I'm sticking with them for now, but I'm also feeling like the effort I've put into automating the configuration of our application and all it's dependencies has been time well spent. I can boot a new infrastructure on another service in an hour or so and migrate to it with minimal downtime.

Re:We're moving. Goodbye Rackspace. by boyter on Monday November 08, 2010 @11:09PM (#1885284)
What are they like for CPU usage? I have been looking at VPS.NET and other such providers to ensure I get a reasonable slice of CPU which I can use for other things.

Most VPS providers I have come across dont like it if you start working out the millionth digit of Pi, but sometimes you have a task that takes a while and you can't run offsite.

Re:We're moving. Goodbye Rackspace. by easp on Wednesday November 10, 2010 @01:19PM (#1891292)
I believe each physical host has at least 8 real cores. They give each instance 4 virtual cores. As long as CPU time is available, an instance can use the full capacity of four cores. If there is contention for CPU time, every instance gets an equal share of CPU, which is also their fare share of CPU, because all the instances on a box are the same size/pricepoint. Further, larger instances are shared with a proportionally smaller number of other instances.

My experience, which seems to be shared by just about anyone who has published benchmarks, is that Linode is appropriately conservative on the degree to which they oversell CPU, because generally, it appears that a task that wants 100% of four cores does in fact get close to 100% of four cores.

The only thing to look out for is that there is quite a difference in single threaded performance between their newer hardware and their older hardware. Our app is written in one of the popular interpreted dynamic languages. Generic benchmarks for that language showeda 2-3x difference on some tests when run on the different hardware, and our app showed similar differences on CPU bound tasks.

There are a few implications of this:

First, it complicates things when you try to make your staging environment mirror production. The quick check is to look at /proc/cpuinfo and see if clock,cache and model line up. If not, open a ticket and ask them to migrate instances so things match.

Second, while they try to size things appropriately so that your guaranteed CPU Is the same, regardless of which generation of host you are on, your peak CPU is going to vary dramatically since you get 4 cores, irregardless.

Re:We're moving. Goodbye Rackspace. by madaxe on Tuesday November 09, 2010 @04:30PM (#1887983)
Great - we've got two moderately large (12+ instances) clusters with them, one in the US, one in the UK - both are fast.
Re:We're moving. Goodbye Rackspace. by shazow on Monday November 08, 2010 @09:27PM (#1885048)
+1 to Linode.
Re:We're moving. Goodbye Rackspace. by schammy on Tuesday November 09, 2010 @12:53AM (#1885520)
I hear lots of awesome things about Linode, however they are actually hosted by Rackspace so with Rackspace's known reliability issues... maybe an issue.

Another service to consider is VPS.net. We have 6 virtual servers with them. They are cheap and perform well, their control panel is great and they have tons of great pre-configured OS installs. However downtime can sometimes be an issue. Usually it's not major, e.g. a server will just be unavailable for say 30 seconds or maybe a few minutes, but this happens probably an average of once or twice a month for each of our servers so it can get annoying. Like I said though the outages are usually minor. There's only been one that I remember that was major, and lasted several hours. We've been using VPS.net for about 8 months and overall are very happy with them. Not sure I'd recommend them for your setup but we only use them for a custom built CDN and for that it works very well.

Re:We're moving. Goodbye Rackspace. by bombs on Tuesday November 09, 2010 @01:21AM (#1885593)
Are you confusing Linode with Slicehost? Slicehost was acquired by Rackspace.
Re:We're moving. Goodbye Rackspace. by revoltingx on Tuesday November 09, 2010 @02:29PM (#1887602)
So, to confirm, Linode is NOT hoste by Rackspace?
Re:We're moving. Goodbye Rackspace. by Perihelion on Tuesday November 09, 2010 @04:32PM (#1887991)
Nope :)
Re:We're moving. Goodbye Rackspace. by revoltingx on Tuesday November 09, 2010 @05:27PM (#1888217)
Phew, thank you.
Re:We're moving. Goodbye Rackspace. by mdgeek on Wednesday November 17, 2010 @01:24PM (#1915130)
VPS.NET servers hosted on Softlayer data centers. Midphase,Anhosting & 100TB.com (all in UK2 group) also on Softlayer.

Hostgator,Linode,SiteGround,Site5 hosted on ThePlanet. JustHost on Liquidweb dedicated servers.

Hostmonster on Bluehost servers.

Re:We're moving. Goodbye Rackspace. by mdgeek on Wednesday November 17, 2010 @01:17PM (#1915096)
linode servers are hosted on The Planet data centers. Check this link: http://serversiders.com/linode.com