Let's compare this with Facebook. In October 2009, Facebook announced they had 30k servers[0]. In September 2009, there was a rough estimate that Facebook served 200 billion pageviews per month[1]. That implies 73k pageviews per second, or 2 pageviews per second per server. Clearly the pageviews are a rough estimate, but even if facebook served 1 trillion pageviews per month they still wouldn't be beating reddit for efficiency.
I have a feeling if you run the numbers for any other highly dynamic site at scale, you'll find that amortized over every server in use, you won't get a lot better than 10 pageviews per second.
[0] http://www.datacenterknowledge.com/archives/2009/10/13/faceb... [1] http://www.businessinsider.com/googles-estimates-for-faceboo...
10 pageviews per second is pretty lame IMHO. Crazy crazy server costs. Amazon is crazy expensive! Why use them? Either Reddit need to drastically change the way they do things, or they deserve to die.
Facebook also doesn't count all webserver HITS as pageviews, whereas most of Reddit's do count.
Beyond that, pageviews for Facebook may eat more resources (CPU/RAM) than Reddit on average due to photo uploading and other misc. things on Facebook. This means a server is working harder at 2pageviews/sec/server for Facebook.
How is that even relevant? Images cost in bandwidth, not in computing power.
> Does Reddit find things only specific to your account
Uh yes, on every single submission and comments. The admins clearly stated that what used the most resources was the voting system. As well as every single user (whether you marked them as friends) and the list of links itself, which is extracted from the user's own list of reddits.
There is barely anything on a logged in user's page which isn't at least in part specific to that user's account.
> I could really go on for a long time
No, I don't think so, you've been 0/everything so far, it's time to stop.
Sounds like you need to do some STFU yourself, having clearly never come anywhere near this problem domain. Using standard tools to process and resize user-uploaded images can easily soak up all the CPU time and memory you could throw at it.
I recently spent a bunch of time rewriting a client's image processing pipeline fromt the usual O(n)-space ImageMagick crap that loads the full uncompressed image into memory a few times over to be O(1)-space doing streaming downsampling by exploiting the compression implementations of JPEG and PNG. It was more than worth it — even with the PNG implementation being a bit more CPU intensive now, it's a lot nicer having it operate in constant space without fear of swap and the OOM-killer.
Your personalized news feed and what you can see depends on the settings of all your friends and those friends.
Reddit is just a simple forum site...
Facebook have revenue, and profit. They can afford to buy more servers and have them sitting idle.
Reddit, should be trying to run on an optimal number of servers. Firstly, cheap hosting (Which is NOT amazon), and secondly less servers. 80 is just crazy.
As someone who's worked in a data centre before, I can say hardware problems only increase as you scale up. We had around 350 servers with an average of three drives each, and we probably went to the data centre to swap dead hard drives on average three times a week - SAS, SATA, or SCSI. Nevermind the time I spent testing motherboards, swapping CPUs out, etc.
Owning your own servers means dealing with maintenance and downtime. Virtualizing 10 servers on one physical server is great until that one physical server's RAM starts acting flaky, and then you have to take those 10 servers offline, or shift the VMs onto your other hardware.
Renting your servers typically means paying a monthly cost for the hardware, including RAM and disk, long after the fees have paid the DC back their costs.
For Reddit, being able to bring N extra nodes for X purpose (mysql, cassandra, web serving, etc.) with a few clicks in a few minutes is likely a huge draw for them. It means they can grow gradually (instead of having to shell out another $10-20k for a new server), and it means they can dynamically adjust to meet their needs. For example, if Monday is a busier day than Saturday by a significant stretch (and if their architecture allows it), they can bring more nodes online early Monday morning to handle the load. Take them offline until Thursday evening to handle the 'It's Friday, to hell with doing work' rush I'm sure they get, and so on.
That said; this looks, sounds and smells like database scaling issue more than anything. It is interesting to ponder the long-term costs of running a site like this on EC2 versus your own maintained server farm.
That web server is running separate from databases and other resources. It never sees load. We're using a static language, but even if ruby was 10x slower, it wouldn't matter.
On the other hand, change the contents of a stored procedure so that it doesn't line up with indexing properly, and page load goes from >0.1s to <4s.
What I'm really trying to say here is... prove it.
Actually, you do! :) http://code.reddit.com
But to help you out, I'll tell you that a good chunk of the expensive loops are written in C.
This also calls into question the original assumption, that posters believed Reddit had far too many wasteful servers to handle their service, and that it was because they use a dynamic language.
Scaling is caching and architecture, not writing your app in Java.
I'm also not sure what makes a Facebook game's domain more "widely cachable". 100% of users are logged in. The vast majority of actions taken are changing state. Any app touchpoint is writing to the database. Page caching is nearly impossible.
It's a lot less different than you think.
You obviously know how to scale an app if you were pulling 16 million pageviews a day, and I don't intend to discount that at all. I just mean to point out that while fundamentally the same problem, reddit has to deal with a version of that particular problem that most applications don't begin to approach.
I never understood this. Does reddit really need to spend the capital making sure I see a stranger's upvote the moment it occurs? A 60 second delay to refresh pages in batches seems perfectly reasonable. Perhaps with a client side script to mark my own upvotes so the system doesn't look like is losing my selections.
Mibbit is written in Java (Custom written framework and server), and handles traffic just fine. Java is insanely efficient for network IO.
It's an apples to frogs comparison, but if you measure 'page views' then Mibbit does bajillions, on a handful of servers.
Warbook was a Facebook application written in Ruby on Rails I ran by myself in late 2007 - 2008. It grew to over 16 million pageviews a day. At the time it was more pageviews than Twitter.
I scaled it using the following stack: Perlbal for load balancing, LightHTTPD for static assets, Mongrel for dynamic requests, Memcached for caching, and MySQL for relational data storage.
I used two medium instances for load balancing, one medium instance for asset hosting, 15 small instances for mongrel, one XL instance for memecached, and one XL instance for MySQL.
I used memcached as a "write-through" cache. Everything in cache was considered fresh. Every write of a cachable object would write to both MySQL and memcached. Every read of a cachable object would start with memcached first and failover to MySQL. This reduced reads on the database by 95%.
Total hosting costs were ~$2,000 a month.
Facebook giveth and Facebook taketh away.
That's cute, but was your web app as hard to cache as Reddit?
Scaling a fully static website is trivial, and the more dynamism you include, the harder it becomes.
There is barely anything static on a logged in Reddit user's page.
USERCLASS = SHA1(SUBREDDIT1 XOR SUBREDDIT2 XOR SUBREDDIT3)
For logged in users, you're still going to need the voting status for the current user on every single submission as well as that submission's hidden status for the current user to decide whether or not a submission should be displayed in the listing.
I believe they almost never hit the DB directly, so these are probably recached immediately (or submitted to both the cache and the DB at the same time), but that still means quite a lot of traffic.
If it's truly a python issue, it should not be taken lightly by entrepreneurs. I hope it's not...I'm an avid user of both ruby and python, and want to believe they can both be used to create successful and maintainable (in terms of effort and cost) sites.