Ok, as far as I can tell, the real story is that Engine Yard relies on a GFS/SAN setup that doesn't scale in the unique way that Github needs.

If you think about it, Github is one of the few sites that actually directly uses the filesystem heavily. Everyone else hits scaling issues on the DB first.

  The sad thing of all of this is it's not really a matter  
  of scaling, and it never has been. Our bottleneck has 
  always been the file system. GFS just... sucks. I'm sorry, 
  but I have to say it. Case in point, your graph. The first 
  rebuild I ran timed out because of GFS. The second one ran 
  fine, took maybe a minute to process, if that. GFS impacts 
  everything... gem build failures due to cloning... GFS. 
  Network graphs taking long time to build... GFS. Caching 
  jobs not completing... GFS. I think you see where I'm 
  going here. There's no plans to deploy the new code to the 
  live servers, and I think the reason is that we're afraid 
  it'll make GFS performance worse, not better. But on the 
  new servers where we don't have to fight GFS, it's 
  amazing.

Funny thing is that we told github that gfs would not scale for them over a year ago, we also outlined how to move to a shared nothing chunk server architecture. They didn't take our advice so it's mostly their own architecture decisions that were holding them back with regards to gfs.

Anyway there seems to be plenty of airchair quarterback on this one. The real story is that we can't afford to host them for free anymore.

FWIW - thanks for hosting them for so long. GH is a wonderful service, and I'm sure EY contributed greatly to it's success.

Is it so hard to imagine that they weren't jumping at the chance to write their own custom non-filesystem storage backend?

Well, EY could make Github fast again by providing more resources. But then, someone would have to pay for the additional resources. EY doesn't think it's worth it, and Github can get the additional resources elsewhere for cheaper (EY is expensive, because you pay for premium support).

It seems like it's more about the fit between the companies, than any particular performance issues.

It's also inaccurate: GitHub doesn't offer the largest free storage quota. That's still Google Code, which offers 1024 mb/project, rather than 300 mb/user.

Google is also happy to bump up your quotas if you have a genuine need.

As are we, disk space should never be an issue with us unless you're doing something very wrong (hosting videos, etc).

The GNU Emacs repository uses almost all of my (paid) storage space. I am not doing anything wrong, either, Emacs just has a lot of history and a lot of code.

Emacs officially still uses CVS, right?

bzr.

But throwing everything on github is a lot easier than using bzr, so I do that. There is no bzrhub, and I don't have commit-access to GNU Emacs.

> It sounds odd Engine Yard hosts an entire for-profit company for free just in exchange for some complimentary accounts.

why? its fairly high quality, targeted advertising with a built-in demonstration.

> It sounds odd Engine Yard hosts an entire for-profit company for free just in exchange for some complimentary accounts.

Well, sort of. It's really a value trade-off. EY offers its customers something that would cost them $50 a month, so it's costing Github (customers) * $50 to offer that. They exchange the lost value for added value in hosting (i.e., they're "losing" a few thousand a month in free accounts for our customers, so we'll offer them free hosting to continue to offer that).

It's simple barter.

Very excited to see GitHub continuing to grow and EngineYard and GitHub handling their barter deals so cordially. Both companies are class acts!!!

That assumes that Github is slow because of their hosting (Engineyard) and not because of how it's written.

I dunno, it could go either way for me: "Engineyard drops Github for being too costly" vs "Github needs better hosting, so we're dropping Engineyard"

Its probably more about costs for both sides. I doubt that EY is _slow_ without regard to cost. Just that instead of rewritting GitHub's web app to deal with inherent performance issues, it makes more sense to throw more hardware at the problem. There does come a point where owning more of your server stack makes more financial sense.

It is going to be interesting to see how the performance of Github and their freemium offering changes when they start paying for infrastructure somewhere else. Unless they can smooth talk another host into giving them free infrastructure.

Based on replies from my support queries about performance, they are aware that paying customers expect better/more-consistent performance than what we've had of late.

I see, so even if they did pay, it still wouldn't be fast enough. So free even has its costs, in terms of dissatisfied customers.

It's like when your boss let's you "resign" so you don't have to tell a future employer you were fired. EY was fired.

How can github fire EY when EY was giving them free stuff. Why doesn't github just pay EY for the services they are using?

This is another failure of Freemium. GH is only able to provide such large repos because they get free infrastructure.

I can't really blame EY for bailing. The partnership is no longer equitable. GH is getting something for nothing. I shouldn't say nothing, it's something, but it's not enough.

If your service doesn't meet my needs, it doesn't matter how cheap it is - including $0. According to my understanding, EngineYard doesn't meet GitHub's needs anymore.

I don't think that's totally fair to say, we've put a considerable amount of effort into GitHub over here at EY, across all of our departments. GitHub offers a great product and it's really amazing to see them grow so fast, but that success also introduces a whole lot of new problems to the situation, many of which can be quite costly to solve.

It's just a conclusion that makes the most sense for both sides of the arrangement.

EY is worth every penny they charge because they provide an incredible quality and commitment of service. They are without a doubt the best hosting company I've ever had the pleasure of working with. Not only are they quick with assistance 24/7, but they all know what they're doing.

This really shows poorly on the guys at Github. It sounds like EY went out of their way to accommodate the growth rate (I have yet to hear the Github guys dispute that) and just because EY was asking to be paid for all the hard work, Github was unwilling to, for once, be paid...Github ditched them.

If the expectation was to be paid, then EY would have offered a contract with a future non-free date. Nobody expected this eventuality, or if they did both sides were OK with reaping the benefits in the short term.

What about appreciation for a company that essentially helped sponsor your company's incubation?

One of the things I appreciate about EngineYard, and the Ruby development world in general, is that they appreciate the benefit of helping to incubate open-source and small startup operations. It's about re-investing back into the community. They did this when they helped incubate a startup company with some great programming minds behind it, which they knew would assist in the rapid development of open-source Ruby libraries.

But it seems GitHub hasn't absorbed any of that mindset by their move to ditch their supporter as soon as they're in a position to show appreciation for the help by actually paying for the service they receive for once.

And what if GitHub can't afford EngineYard?

I find that hard to believe. They're one of the most popular and successful companies in the Ruby community and they have paid accounts that they've been collecting on for quite some time now. EY has thousands of customers who pay them every month and can afford it. How would it be that one of their most popular customers who doesn't pay anything for hosting wouldn't be able to pay them for once?

EY is super expensive. Also most of the cost is for support techs who can read/understand ruby code and gems. Chances are the github guys don't really need that.

How many of those customers have VC funding? Github is completely bootstrapped.

scott_s is scott shacon and works for github

No he isn't. http://news.ycombinator.com/user?id=scott_s

Indeed, I am not Scott Shacon. Too many Scott's! One time in undergrad three of us sat in a column and confused the hell out of our compiler's professor.

No, it's not. Scott Chacon's surname starts with a C, not an S.

http://github.com/schacon

http://progit.org/

Do you know if RackSpace will be giving GitHub free infrastructure? If they're not then do you expect them to fail?

FWIW, I first heard about Engine Yard through GitHub, and I'm sure I'm not the only one.

I do not expect rackspace to give them free infrastructure, no. I also don't think this will cause them to fail. They're pretty smart folks and they have revenue coming in, so they'll probably be fine.

After reading more into the thread and the issue being mainly with GFS, then perhaps it won't be that difficult to improve performance at Rackspace because they'll control the hardware and be able to use any file system they want.

Knowing why they can't run at EY clears up a lot of the questions. The scalability at EY is simply too limiting for GH, paid or not. It probably doesn't make sense for EY to install a new file system simply because GH wants it and GH probably can't afford, or doesn't want to afford the amount that it'd take for EY to move all their customers to a more scalable file system.

If you look beyond all the drama here, this really is a rational decision on both sides. It's a testament to the community here that two groups parting ways in such a dramatic way can both participate in this very comment thread together and with civility. It speaks volumes and I think everyone involved has handled themselves very well.

I can't imagine this is a pleasurable experience for either group, but when one party wants something the other is unwilling to provide and in this case, it's true of both EY (different file system) and GH (compensation for the change) it's rare for both to understand the other side's perspective and avoid irrational name calling and bickering. I'm glad this breakup hasn't gone down that route.

Come on, its not fair to say something for nothing. As much as you can't belittle the importance of infrastructure, you also can't act as if its the only thing that matters. GitHub obviously has a product outside the raw hosting they provide, and each additional user brings on extra work on their end.

For all we know it could have been equally lopsided in the other direction: if EngineYard was sending them tons of traffic through free accounts, it could have proved costly in terms of support and negative effect on paying customers.

We of course don't know.