News for hackers. Stuff that matters.
Re:Reddit costs $33k/month to operate? by teej on Monday July 26, 2010 @09:58PM (#1549997)
I ran a web app with identical pageview numbers using Ruby on Rails and EC2. My server costs were 10x cheaper than what reddit is paying.

Scaling is caching and architecture, not writing your app in Java.

Re:Reddit costs $33k/month to operate? by cheald on Monday July 26, 2010 @10:35PM (#1550072)
It's a bold boast, but honestly, it doesn't mean much if the problem domain isn't similar. Reddit in particular runs into issues because so little of what they do is widely cachable. The site is heavily customized per user, and experiences an awful lot of cache churn. I don't disagree with your sentiment (design matters more than choice of language), but I could serve 16 million static pageviews/day off of a $20/month Linode. That doesn't mean that reddit could do the same.
Re:Reddit costs $33k/month to operate? by teej on Monday July 26, 2010 @10:50PM (#1550092)
Reddit has 8 million unique visitors, but only 300k subscribers to /r/pics, the top auto-subscribed subreddit. Generously assuming half of people manually unsubscribe, that's at most 7% of their user base that are logged in. I know full well about cache churn with logged in users, but that's a lot of lurkers.

I'm also not sure what makes a Facebook game's domain more "widely cachable". 100% of users are logged in. The vast majority of actions taken are changing state. Any app touchpoint is writing to the database. Page caching is nearly impossible.

It's a lot less different than you think.

Re:Reddit costs $33k/month to operate? by cheald on Monday July 26, 2010 @11:55PM (#1550186)
I can't claim to know the specifics of your application, but consider that a single upvote invalidates however many thousand copies of a given page. An action in a Facebook game might invalidate a page for 10 or 20 or 50 users. An upvote, (or a comment, or an edit, or an upvote on a comment) invalidates that page for upwards of 300k+ users. Combine that with the rate at which reddit's pages change, and you're talking an obscene amount of CPU time spent building fragments/pages to stick into your cache. Scope of visibility of a change means a lot.

You obviously know how to scale an app if you were pulling 16 million pageviews a day, and I don't intend to discount that at all. I just mean to point out that while fundamentally the same problem, reddit has to deal with a version of that particular problem that most applications don't begin to approach.

Re:Reddit costs $33k/month to operate? by parfe on Tuesday July 27, 2010 @11:12AM (#1551457)
>a single upvote invalidates however many thousand copies of a given page

I never understood this. Does reddit really need to spend the capital making sure I see a stranger's upvote the moment it occurs? A 60 second delay to refresh pages in batches seems perfectly reasonable. Perhaps with a client side script to mark my own upvotes so the system doesn't look like is losing my selections.

Re:Reddit costs $33k/month to operate? by axod on Tuesday July 27, 2010 @04:54AM (#1550677)
I don't know what the "writing your app in Java" bit is about... Care to elaborate?

Mibbit is written in Java (Custom written framework and server), and handles traffic just fine. Java is insanely efficient for network IO.

It's an apples to frogs comparison, but if you measure 'page views' then Mibbit does bajillions, on a handful of servers.

Re:Reddit costs $33k/month to operate? by awa on Monday July 26, 2010 @10:09PM (#1550019)
It would be nice if you elaborate a little.
Re:Reddit costs $33k/month to operate? by teej on Monday July 26, 2010 @10:32PM (#1550067)
I'm not sure what you want to hear, so I'll elaborate a lot.

Warbook was a Facebook application written in Ruby on Rails I ran by myself in late 2007 - 2008. It grew to over 16 million pageviews a day. At the time it was more pageviews than Twitter.

I scaled it using the following stack: Perlbal for load balancing, LightHTTPD for static assets, Mongrel for dynamic requests, Memcached for caching, and MySQL for relational data storage.

I used two medium instances for load balancing, one medium instance for asset hosting, 15 small instances for mongrel, one XL instance for memecached, and one XL instance for MySQL.

I used memcached as a "write-through" cache. Everything in cache was considered fresh. Every write of a cachable object would write to both MySQL and memcached. Every read of a cachable object would start with memcached first and failover to MySQL. This reduced reads on the database by 95%.

Total hosting costs were ~$2,000 a month.

Re:Reddit costs $33k/month to operate? by mdolon on Monday July 26, 2010 @11:58PM (#1550191)
If you don't mind me asking what happened to it? Facebook now reports about 5,500 monthly users. I'm guessing you were acquired by SGN but I'm don't see why that would cause traffic to drop so much.
Re:Reddit costs $33k/month to operate? by teej on Tuesday July 27, 2010 @12:14AM (#1550230)
SGN switched their core focus from Facebook games to iPhone games in mid 2008. At that time, they droped support for all of their Facebook properties except (fluff)Friends. Games on the Facebook platform are doomed to rapid traffic loss without constant adaptation and viral tuning. Even with it, user retention is hard.

Facebook giveth and Facebook taketh away.

Re:Reddit costs $33k/month to operate? by apower on Tuesday July 27, 2010 @11:42AM (#1551561)
16M/24/60/60 = 185 pages/sec. 185/15 = 12 pages/sec/server. That's not much different than the Reddit number.
Re:Reddit costs $33k/month to operate? by CaanaCast on Tuesday July 27, 2010 @01:41AM (#1550345)
could you set up something similar not on EC2?
Re:Reddit costs $33k/month to operate? by masklinn on Tuesday July 27, 2010 @02:17AM (#1550413)
> I ran a web app with identical pageview numbers

That's cute, but was your web app as hard to cache as Reddit?

Scaling a fully static website is trivial, and the more dynamism you include, the harder it becomes.

There is barely anything static on a logged in Reddit user's page.

Re:Reddit costs $33k/month to operate? by rb2k_ on Tuesday July 27, 2010 @04:02AM (#1550579)
Isn't every single subreddit cachable? Only the starting page isn't 100%, but I'm sure that many users have the same combination of subreddits (e.g. the default ones). Would be interesting to see if they have certain "user classes". Something like:

USERCLASS = SHA1(SUBREDDIT1 XOR SUBREDDIT2 XOR SUBREDDIT3)

Re:Reddit costs $33k/month to operate? by masklinn on Tuesday July 27, 2010 @05:30AM (#1550719)
> Isn't every single subreddit cachable?

For logged in users, you're still going to need the voting status for the current user on every single submission as well as that submission's hidden status for the current user to decide whether or not a submission should be displayed in the listing.

I believe they almost never hit the DB directly, so these are probably recached immediately (or submitted to both the cache and the DB at the same time), but that still means quite a lot of traffic.

Re:Reddit costs $33k/month to operate? by jonknee on Tuesday July 27, 2010 @09:40AM (#1551138)
The cache is invalidated after every vote.