If true, this is a pretty big blow to the viability of RoR as a large-scale platform, at least as it is right now.

I've always thought RoR's tight coupling with traditional SQL would be a serious scaling issue. RDBMSs in the traditional sense just dont scale easily. Google figured this out a while ago (hence BigTable). And RoR reads and writes soo much that you are left with a burden on the DB that might have been better mitigated by doing more of the processing client side and persisting with a simplified cloud-based service rather than a single, central RDBMS.

Not to stir up a hornets nest, but I really think this is a limitation of RoR that should be addressed in the future. Abstract out the SQL so other methods of persistence are possible.

EDIT: I am not trolling RoR. I am simply proposing that it might be better to be able to use it with something other than traditional SQL for persistence of data, so that traditional RDBMS can be replaced with a simplified cloud-based service such as SimpleDB.

I speculate this might be possible now using Ruby -> JRuby -> JPA -> SimpleJPA (or any other MySQL/SimpleDB wrapper) -> SimpleDB. I dont want to even get into how much work that would be though. It needs to be easier I think.

Startup idea maybe?

Rails, and the alternatives mentioned don't really make sense for a messaging application. You wouldn't write the back end for Gchat or AIM in Rails (or PHP, or Django), either. Rails would be fine as the web front end that talked to another messaging system. I've interviewed with the twitter guys and been to their talks and they definitely are NOT doing the right thing. However, "the right thing" is not easy and there are few people around and available who can do it.

edit: their best bet is to try to find someone who is burned out from working on wall street applications.

Twitter isn't a messaging application in the traditional sense (as in IM or phone networks) because it doesn't involve real-time synchronous communications. All communications in Twitter are asynchronous as in most database-backed webapps and I think it should be possible to scale Twitter with Rails assuming you know how to scale the database tier.

Having said that, it would still be fun to rewrite Twitter in Erlang... :)

>Rails would be fine as the web front end that talked to another messaging system.

Which is exactly what they're doing:

http://rubyforge.org/projects/starling/

Wall street types are used to working on serious hardware though (I am not a wall street person, but work for a fortune 500 type place).

One of our Oracle databases sits on a 24 core itanium box

That sort of hardware + Oracle can munch through some serious work - perhaps that is the database solution, but I believe it doesn't come cheap!

Just curious, what would you write the backend for AIM in? I don't have any experience developing messaging-type apps.

Erlang is a fine option. So is C.

Dang - I was going to say Erlang. This is an ideal application for Erlang... scalability and uptime. There's a reason it's heritage is in the teleco industry.

I went and had a look at the slides for their 'scaling twitter' presentation, and it looks like they had a look at Erlang and reacted with some silly "ooh, it looks ugly" sort of comment. Seems pretty silly to reject something out of hand that is very much along the lines of what you need.

Java would work extremely well.

C.

The scaling problems have little to do with rails. They admitted their problems were at the database level eons ago. I imagine "the switch" has more to do with the new chief architect switching to his favorite horse. Two whipping boys for the price of one: the old platform, and the old architect.

The new rewrite will give the new architect time to vest, until it fails, like big rewrites usually do. If you're not sure how to fix things, the big rewrite is a perfect solution, really.

I know rails itself is sound, my comments are directed more towards rails/DB integration. The database always seems to be the real bottleneck..

The rails DB integration lets you use the database directly whenever it suits you. At that point it's more a matter of arranging your data so that it lets you work around the key pain points. Since you can access the data directly when needed in rails, how will a new framework help if your problem is in the database?

Since Ruby gives you such great flexibility, it wouldn't be that hard to hack things around a bit to redirect data requests to some other source.

I don't see that as the problem.

Twitter always struck me as a little lower on the complexity scale and a little higher on the throughput scale than where RoR's sweet spot lies.

Since their requirements seem like they're pretty static now, I would think writing a backend in C wouldn't be out of the question.

The question is, do they have the skillset in-house for that sort of undertaking?

Even if they don't, how difficult will it be for them to bring in a great hacker who can?

I'm sure there are enough skilled C hackers out there; but for them to bring in a new face to re-architect the system now (which is to say, after all the scalability problems have started) seems risky.

For them to not do so is riskier.

skillset is a really good word

i am really surprised how facebook, so quikly found the skillset to build a 10,000 server site and how come, twitter didnt! just why ...

its the skillset, but again why was facebook successful in aquiring the skillset when twitter wasnt! ... why is twitter considering a rewrite and facebook didnt ... they just grew !!

According to various scalability articles on FaceBook, they continually rewrite their software. It's probably been rewritten far more than Twitter has, you just never hear about it because TechCrunch doesn't have a source inside.

In my experience, that's really the only way to do software. It's completely unrealistic to think you'll get it right the first time, and yet completely destructive to go and start from scratch. So you pick your worst problem, rewrite it, then hook it back into the working system without touching the rest. Then pick your next-worst problem, rewrite it, etc.

You've got a point regarding the inside scoop however, Facebook doesn't suffer a global outage on an almost weekly basis either.

Having 10,000 servers doesn't say 'skillful' to me.

Facebook's system is also 1000x times more complicated than twitter's simple message passing system!

For me, Facebook often seems very sluggish and slow - I have a pretty useful 4Meg connection and others sites are fine - maybe 10,000 servers is part of the problem - it seems like a ridiculously high number. I wonder how many transactions/pasge views per second they get ...

Employing people with the skillset to build 10,000 servers and having the money to do so are two separate things.

Twitter has millions of dollars.

The bottleneck/stress-point is almost certainly the unique IO patterns and latency expectations. I doubt CPU is the issue or that C is necessary; instead they need to figure out the right fanning/batching algorithms, and buy gobs of RAM and use it efficiently.

With their recent fundraising, they might even be able to run a couple parallel fixup projects in competition, and let the best win.

Yeah for people having never heard of Erlang.

When I wonder why Facebook doesn't have problem scaling their chat? Oh, erlang + flat files.

I can't wait for Twitter to move to Java or PHP so people finally STFU about Rails.

Twitter moving away from rails doens't make rails look bad, it makes everyone at Twitter look like the biggest jackasses in history.

"We can't get this wrench to hit this nail hard enough, so lets buy another brand of wrench instead of a hammer and see if that works."

Do you have any more info on Facebook's chat architecture or were you just speculating?

A quick Google of "erlang facebook chat flat file" returns your initial post, would love to read more about that if have anything.

My knowledge of Facebook's chat architecture comes from drunken conversations with Facebook engineers at bars. So no, I don't think I know any more than that. hiccup

Picking between the wrench and hammer is never a problem.

All tools tend to look like hammers when all problems are seen as a subset of nails.

ev: FWIW: Twitter currently has no plans to abandon RoR. Lots of our code is not in RoR, already, though. Maybe that's why people are confused.

http://twitter.com/ev/statuses/801530348

(Ironically, Twitter was down when I first tried to get the URL for this.)

Currently: 500 Internal Server Error.

(With an HTTP Status header being dumped to the screen instead of, you know, sent as a header.)

Do you obsessively check the headers for proper status codes whenever you land on a page? I'm always amazed how people find out header problems like this just by browsing.

He probably has firebug.

Michael Arrington loves taking things out of context.

I hate it when non-programmers write about programming! It's a giant festival of stupid.

"I hate it when non-programmers write about programming!"

It doesn't matter what he says. All that matters is that you notice.

After 30 minutes, you are the #1 response to the #2 thread.

So people click through.

Cha-ching.

True journalism, indeed.

Michael never claimed to be a journalist. You're likely mixing up the professions because you think TechCrunch has gotten so big that it has and cares about its reputation. That wouldn't be the case, he just writes whatever he can to continue to generate pageviews.

For what it's worth, I contributed to the article and I call myself a programmer.

Do you and Arrington have some kind of confidential knowledge about Twitter's implementation that allows you to conclude that its problems are directly attributable to Rails itself and not how it was implemented, or something that has nothing to do with the programming language at all (the hosting facility they use, their operating system, ...)? Or are you just trolling?

Yeah I think your question is a fair one.

But I don't think it's ridiculous to assume that the stability and operational functionality of a website is primarily dependent on the web framework the website uses. The data center is probably the second to blame, but I don't think that Twitter's problems are related to continual issues with their hosting (though i'm sure some are). I doubt the blame lies on Debian, or whatever their using.

It is possible to write buggy, unstable, unreliable, unscalable applications in any framework or language and just because you use a certain technology doesn't mean anything about whether you'll create an app with a good experience.

Handing a sniper rifle to someone does not make them a sniper. Snipers have personal qualities and training that make them what they are; their weapons are just tools that help them realize their inherent abilities as well as possible.

Why are you and other TechCrunch people so eager to assign blame when you don't actually know anything about what you're assigning blame for?

Because TechCrunch thinks our industry is so great that it deserves its own National Inquirer. Seriously, these people should be marginalized and put in the spotlight for the trolls that they are. I'm so sick of the bottom-feeding rumour mongering that scrapes by as journalism these days. Seriously people, stop reading TechCrunch and go checkout a book on O'Reilly Safari or something...learn something worthwhile with your spare time.

"Handing a sniper rifle to someone does not make them a sniper. Snipers have personal qualities and training that make them what they are; their weapons are just tools that help them realize their inherent abilities as well as possible."

This is a good point. But, in the interests of balance, the twitter situation could also be compared to arming a sniper with a broadsword and then being surprised he fails. The choice of analogy makes a surprising difference in how you think about the situation.

"Why are you and other TechCrunch people so eager to assign blame when you don't actually know anything about what you're assigning blame for?"

I would rephrase the question to "what do you and the other tech crunch folks know that gives you confidence in your statements about decisions at twitter".

Point being no one knows what the folks at tech crunch know (or not).

Just to keep the conversation going: what do you think Twitter's main issues are ? (Or do you think their downtime is not worth mentioning?)

Scalability problems are usually endemic to a particular setup. There is always another bottleneck. The way you're viewing scalability is different than the reality of it.

My previous company (a web testing company) did a load test for a large tax return company. When testing started, their app could support 3 simultaneous users. When the bottlenecks were removed, one by one, the same app (framework, codebase, language and all) supported north of 10,000 users. As you test, there is usually something like a database lock problem, high cpu from a loop that needs to be optimized, bad query/data layout, memory usage maxing out, or something you are running out of, one way or the other.

The people who point out that twitter is not a standard web app and might be better served by messaging have a point also.

What the parent says is that a proper assessment will require an in-depth knowledge of how it is actually implemented. Taking random shots like this article does is like a blind man trying to grope an elephant.

" Taking random shots like this article does is like a blind man trying to grope an elephant."

Groping an elephant seems like a bad idea even for the sighted.

I would like to second the call for an immediate stop to elephant groping.

Elephants are a dear and treasured species. This continual assault by blind people and various other gropers have too often left our Loxodonta Africanan friends cowering in terror.

As for RoR, I'll simply quote a good friend of mine. "RoR rocks! Scaling isn't that big a deal. Took us over six months to get the stack just right for a recent customer."

Ugh.

Actually, that is ridiculous to assume.

Well it's getting to the point where it's like a tabloid now.

I'm embarrassed by a lot of comments here. I thought people on this forum would know better. It's like I'm watching a car, with smoke and flames pouring out from under the hood, pull over to the side of the road and people running up and checking if maybe it needs new tires.

You could probably rewrite the front end in applescript and you'd see little difference, and if you did just throw more servers at it. The problem is probably that theres a DB at the heart of all the message crazyness, you can't do that and expect to scale.

I am Ruby learner. The way ruby works with DB is problem for novice. If programmer don't do careful coding for caching then it is bound to cause problem. Ruby hype is all around auto code generation and simple coding/scripting. But when you are writing enterprise level apps then there is more to consider.

Totally amused he threw Zed Shaw in there. From what I remember, Zed trashed several members of the rails core team, not the framework itself or even the entire community.

Surely they are just rewriting the slower core areas of the API in something faster/more distributed. There is no reason they'd have to rewrite the entire thing from scratch.

Zed certainly did trash the community. Take a look at

"This is exactly what makes Rails a ghetto. A bunch of half-trained former PHP morons who never bother to sit down and really learn the computer science they were too good to study in college. [...]

After this, I started thinking, since it was the first realization that I picked the absolute last wrong community to make my living in. They were all pricks, morons, assholes, and arrogant fucks who didn’t care about the art or the craft.

They were all a bunch of little ghetto fabulous princesses trying to make it in this tiny little Rails world and not enough brains between them all to make it happen."

and about the framework

"Now, DHH tells me that he’s got 400 restarts a mother fucking day. That’s 1 restart about ever 4 minutes bitches. These restarts went away after I exposed bugs in the GC and Threads which Mentalguy fixed with fastthread (like a Ninja, Mentalguy is awesome).

If anyone had known Rails was that unstable they would have laughed in his face. Think about it further, this means that the creator of Rails in his flagship products could not keep them running for longer than 4 minutes on average."

Although he does add that

" [...] Rails is a pretty nice idea, and it demolished the Java world I hated so much."

It shouldn't be an all-or-nothing situation. There are obviously parts of Twitter that would work best as non-Rails. Ruby is too slow for certain things, and so you determine which of those certain things are in your application and you build them in C or Java.

For Twitter, I think some parts of the API need to be in C. But the whole service? I doubt it.

Twitter has components in several languages (even erlang for xmpp)

They say most of their traffic is API requests, and I would expect that they would want to optimize that porition of their system. Rails is composed of many components, many of which you don't need for a REST service that talks json/xml. Blaine and other twitters have talked about their experimentments: other app servers (Thin), queueing systems (before building starling), ...

I seriously doubt they are actually going to rewrite Twitter in another language. That's a pretty dumb statement and prediction, and Michael Arrington just lost another notch of respect from me (he lost the previous one for his vicious attacks on that Blaine dude who archtected Twitter).

Sounds to me like Mr Arrington has an axe to grind and is putting that ahead of objective journalism.

Daniel

Why not? It'd probably take a week or so to rewrite in a different language, similar to Reddit when they rewrote in python :/

Has ror become a cult or something? It's just a language, and if it doesn't fit well with the application, best to move on and try something else.

Rails is probably not their problem. The way twitter works is the problem. (In other words, they need a better backend, not a better frontend.)

I was under the impression rails was the backend and was tightly coupled to a db. Is that not the case? :/ By 'better backend', what do you mean?

I assume the difficulty is storing the data and then retrieving the right data when generating a page. Yeah, you can use a SQL database, but that doesn't seem to be working very well. So they probably need to write some custom storage backend that is actually designed for the load that is put on it. Relational databases are general solutions. General solutions are good for general problems, but not necessarily specific problems.

Anyway, if the problem happens to be that the Rails frontend is too slow, speeding that up isn't going to buy them anything long-term. Eventually the database is going to bog them down even if the web GUI is hand-optimized assembly.

JMHO, obviously I have no idea what the twitter codebase looks like or what specific problems they are having. I also don't use Ruby or Rails, but I doubt that's their problem.

Don't underestimate the rewrite of high traffic site like twitter. Rewrites are among the top mistakes that kill startups, and are risky.

What they should do instead is to slowly rework their code, instead of doing a massive rewrite.

That's the theory. Sometimes the practice isn't so simple, when you have 32423432234 lines of spaghetti code, all tangled up. Surely twitter isn't that bad, but I've been in situations where someone cited that very idea and it just wasn't feasible.

Unless you have massive engineering resources that's not dedicated to feature enhancements and new developments of your main product (ie. a small startup environment), a full rewrite is just too expensive.

Regardless of the reason, if something doesn’t work for you, you need to identify the root cause of the problem and don’t be too arrogant to move forward even if it questions your core beliefs

A bike shed (any colour will do) on greener grass...

not true: http://twitter.com/ev/statuses/801530348

+1 for PHP + memcache + Thrift.

Arrington is a bag of douche.

People still consider PHP and Java for rewrites? Usually that's what you rewrite out of, not in to.

Java works pretty well for highly scalable network backends actually.

"X doesn’t scale" is a myth.

http://news.ycombinator.com/item?id=160730

Stop. There's nothing wrong with PHP, and it's proven to scale.

Well, yeah. What would prevent a language from scaling?

The reason you'd ditch Java or PHP is because they are lacking a lot of higher-order constructs. (PHP epsecially. Statically-scoped variables? First class functions? An object system that isn't a hilarious joke? Unicode support? PHP doesn't have 'em.)