News for hackers. Stuff that matters.
Re:Reddit costs $33k/month to operate? by rufugee on Monday July 26, 2010 @09:06PM (#1549895)
Wow...this is hard to believe. Numbers like that make me wonder if the cost of going with ruby or python (in reddit's case) or insert-your-slower-dynamic-language-here outweigh the benefits in the long run. How can anyone bootstrap a company and scale to the size of Reddit without quickly running into a "must fund or sale" situation?

If it's truly a python issue, it should not be taken lightly by entrepreneurs. I hope it's not...I'm an avid user of both ruby and python, and want to believe they can both be used to create successful and maintainable (in terms of effort and cost) sites.

Re:Reddit costs $33k/month to operate? by emmett on Monday July 26, 2010 @09:54PM (#1549987)
14.3 million pageviews per day comes to 163 per second, on average. Reddit's traffic, like most sites of its kind, probably peaks during lunch hour during the workday and probably peaks at 5x that number. So we're looking at 815 pageviews per second. Given 80 servers total, we're looking at 10 pageviews per second per server.

Let's compare this with Facebook. In October 2009, Facebook announced they had 30k servers[0]. In September 2009, there was a rough estimate that Facebook served 200 billion pageviews per month[1]. That implies 73k pageviews per second, or 2 pageviews per second per server. Clearly the pageviews are a rough estimate, but even if facebook served 1 trillion pageviews per month they still wouldn't be beating reddit for efficiency.

I have a feeling if you run the numbers for any other highly dynamic site at scale, you'll find that amortized over every server in use, you won't get a lot better than 10 pageviews per second.

[0] http://www.datacenterknowledge.com/archives/2009/10/13/faceb... [1] http://www.businessinsider.com/googles-estimates-for-faceboo...

Re:Reddit costs $33k/month to operate? by axod on Tuesday July 27, 2010 @04:00AM (#1550577)
Comparing with facebook isn't a good idea IMHO. Facebook also have way too many servers. But they have a business model - they can afford it.

10 pageviews per second is pretty lame IMHO. Crazy crazy server costs. Amazon is crazy expensive! Why use them? Either Reddit need to drastically change the way they do things, or they deserve to die.

Re:Reddit costs $33k/month to operate? by moonpolysoft on Monday July 26, 2010 @10:02PM (#1550009)
That's terribly misleading, though. Facebook has huge clusters for performing offline business analytics. Therefore comparing them to Reddit, which presumably has a much smaller offline analytics backend, is not an apples to apples comparison in terms of efficiency.
Re:Reddit costs $33k/month to operate? by mrkurt on Monday July 26, 2010 @10:34PM (#1550070)
It's an educated guess, but I suspect comparing the entire infrastructure of a site like reddit with the entire infrastructure of a site like Facebook is pretty fair. Neither of them are using their entire server capacity to serve traffic. reddit's at a smaller scale than Facebook, obviously, but it's a pretty apt comparison.
Re:Reddit costs $33k/month to operate? by invisible on Monday July 26, 2010 @11:07PM (#1550116)
You think the functionality of Facebook compares equally to that of Reddit computational-wise? I mean, I know neither is the most complicated thing ever, but does Reddit even allow photos anywhere besides thumbnails? Does Reddit find things only specific to your account and show you a list of friends and their content you are allowed to see (or even something vaguely similarly complicated)? I also highly doubt that Facebook counts all of the ajax-goodness it delivers as pageviews. I could really go on for a long time (widgets, Facebook connect, Facebook Apps API...) about the huge differences that you glazed over since it's using PHP for the frontend.
Re:Reddit costs $33k/month to operate? by mrkurt on Tuesday July 27, 2010 @10:40AM (#1551320)
No, I think looking at Facebook's numbers, figuring roughly 2 pageviews/server, and then looking at reddits numbers (at roughly 10 pageviews/server) and saying "reddit's setup seems reasonable" makes it a good comparison. It's all just hand waving, but I don't mind using that comparison to put reddit's numbers in perspective.
Re:Reddit costs $33k/month to operate? by invisible on Tuesday July 27, 2010 @04:49PM (#1552677)
All of the math is just wacky in the post though - he's comparing peak pageviews/sec/server to Facebook's pageviews/sec/server for a month. Reddit's pageviews/sec/server is 2 using 163 pageviews/sec and 80 servers.

Facebook also doesn't count all webserver HITS as pageviews, whereas most of Reddit's do count.

Beyond that, pageviews for Facebook may eat more resources (CPU/RAM) than Reddit on average due to photo uploading and other misc. things on Facebook. This means a server is working harder at 2pageviews/sec/server for Facebook.

Re:Reddit costs $33k/month to operate? by masklinn on Tuesday July 27, 2010 @02:11AM (#1550399)
> but does Reddit even allow photos anywhere besides thumbnails?

How is that even relevant? Images cost in bandwidth, not in computing power.

> Does Reddit find things only specific to your account

Uh yes, on every single submission and comments. The admins clearly stated that what used the most resources was the voting system. As well as every single user (whether you marked them as friends) and the list of links itself, which is extracted from the user's own list of reddits.

There is barely anything on a logged in user's page which isn't at least in part specific to that user's account.

> I could really go on for a long time

No, I don't think so, you've been 0/everything so far, it's time to stop.

Re:Reddit costs $33k/month to operate? by blasdel on Tuesday July 27, 2010 @03:44AM (#1550554)
> How is that even relevant? Images cost in bandwidth, not in computing power.

Sounds like you need to do some STFU yourself, having clearly never come anywhere near this problem domain. Using standard tools to process and resize user-uploaded images can easily soak up all the CPU time and memory you could throw at it.

I recently spent a bunch of time rewriting a client's image processing pipeline fromt the usual O(n)-space ImageMagick crap that loads the full uncompressed image into memory a few times over to be O(1)-space doing streaming downsampling by exploiting the compression implementations of JPEG and PNG. It was more than worth it — even with the PNG implementation being a bit more CPU intensive now, it's a lot nicer having it operate in constant space without fear of swap and the OOM-killer.

Re:Reddit costs $33k/month to operate? by mdg on Tuesday July 27, 2010 @09:31AM (#1551110)
>IT COMPILES EFFICIENTLY
Re:Reddit costs $33k/month to operate? by burningout on Tuesday July 27, 2010 @06:03AM (#1550757)
Facebook's complexity is much bigger.

Your personalized news feed and what you can see depends on the settings of all your friends and those friends.

Reddit is just a simple forum site...

Re:Reddit costs $33k/month to operate? by axod on Tuesday July 27, 2010 @04:09AM (#1550592)
That's ridiculous.

Facebook have revenue, and profit. They can afford to buy more servers and have them sitting idle.

Reddit, should be trying to run on an optimal number of servers. Firstly, cheap hosting (Which is NOT amazon), and secondly less servers. 80 is just crazy.

Re:Reddit costs $33k/month to operate? by danudey on Tuesday July 27, 2010 @06:04AM (#1550759)
If they truly have to handle a massive amount of traffic with only four people, a service like Amazon is a pretty good deal. It makes it trivial to create, clone, archive, or upgrade servers, and you never have to worry about drive/CPU/motherboard/RAM failures bringing down your site at 3 AM.

As someone who's worked in a data centre before, I can say hardware problems only increase as you scale up. We had around 350 servers with an average of three drives each, and we probably went to the data centre to swap dead hard drives on average three times a week - SAS, SATA, or SCSI. Nevermind the time I spent testing motherboards, swapping CPUs out, etc.

Owning your own servers means dealing with maintenance and downtime. Virtualizing 10 servers on one physical server is great until that one physical server's RAM starts acting flaky, and then you have to take those 10 servers offline, or shift the VMs onto your other hardware.

Renting your servers typically means paying a monthly cost for the hardware, including RAM and disk, long after the fees have paid the DC back their costs.

For Reddit, being able to bring N extra nodes for X purpose (mysql, cassandra, web serving, etc.) with a few clicks in a few minutes is likely a huge draw for them. It means they can grow gradually (instead of having to shell out another $10-20k for a new server), and it means they can dynamically adjust to meet their needs. For example, if Monday is a busier day than Saturday by a significant stretch (and if their architecture allows it), they can bring more nodes online early Monday morning to handle the load. Take them offline until Thursday evening to handle the 'It's Friday, to hell with doing work' rush I'm sure they get, and so on.

Re:Reddit costs $33k/month to operate? by jnoller on Monday July 26, 2010 @09:21PM (#1549927)
Bad architecture or poor design crosses languages. It's entirely possible and done every single day, to emit a broken system using even the most optimal code.

That said; this looks, sounds and smells like database scaling issue more than anything. It is interesting to ponder the long-term costs of running a site like this on EC2 versus your own maintained server farm.

Re:Reddit costs $33k/month to operate? by steveklabnik on Tuesday July 27, 2010 @06:29AM (#1550799)
At least a few months ago, here's Reddit's architecture: http://blog.reddit.com/2010/03/and-fun-weekend-was-had-by-al...
Re:Reddit costs $33k/month to operate? by vinhboy on Tuesday July 27, 2010 @01:19AM (#1550310)
I am not familiar with the cost of running a high profile website, is the price tag for Reddit considered a lot, or little?
Re:Reddit costs $33k/month to operate? by axod on Tuesday July 27, 2010 @04:19AM (#1550621)
A very lot.
Re:Reddit costs $33k/month to operate? by joshu on Monday July 26, 2010 @09:08PM (#1549898)
Actually, I bet you it's a "we hit the database on every pageload" problem.
Re:Reddit costs $33k/month to operate? by emmett on Monday July 26, 2010 @09:39PM (#1549959)
I know the reddit guys, and I can tell you it's definitely not that. Hitting the DB is extremely rare - almost everything comes straight out of cache (either memcache, or memcachedb for caches which are more expensive to rebuild). Execution cost is dominated by the python app servers.
Re:Reddit costs $33k/month to operate? by blhack on Monday July 26, 2010 @09:54PM (#1549989)
The overwhelming majority of reddit's pageviews are users that are not logged in, which is very very cacheable.
Re:Reddit costs $33k/month to operate? by jedberg on Tuesday July 27, 2010 @04:13AM (#1550603)
About 20% of the users are logged in, and they represent about 50% of the page views.
Re:Reddit costs $33k/month to operate? by emmett on Monday July 26, 2010 @11:45PM (#1550173)
I am pretty sure this is false. While the overwhelming number of unique visitors to reddit are not logged in, those users are far less engaged than average. Logged in users make up a huge percentage of the pageviews because of their higher engagement level.
Re:Reddit costs $33k/month to operate? by jedberg on Tuesday July 27, 2010 @04:13AM (#1550600)
The database is rarely accessed in request. Almost all DB access comes from batch processes.
Re:Reddit costs $33k/month to operate? by giantfuzzypanda on Monday July 26, 2010 @09:20PM (#1549924)
Yep. That's why you shouldn't have pylons noobs running a site like reddit.
Re:Reddit costs $33k/month to operate? by tl on Monday July 26, 2010 @09:40PM (#1549964)
Here's an anecdote for you: our corporate intranet site gets a peak spike of a few thousand users every day, around lunch. (People browsing the web over lunch and the computers in-house all have the site as their fixed homepage.)

That web server is running separate from databases and other resources. It never sees load. We're using a static language, but even if ruby was 10x slower, it wouldn't matter.

On the other hand, change the contents of a stored procedure so that it doesn't line up with indexing properly, and page load goes from >0.1s to <4s.

Re:Reddit costs $33k/month to operate? by blueben on Monday July 26, 2010 @09:20PM (#1549925)
How did you get from "It costs this much" to "Dynamic languages are bad"?
Re:Reddit costs $33k/month to operate? by apower on Monday July 26, 2010 @09:33PM (#1549948)
Cause a server with these languages has high overhead in memory and CPU to serve requests thus having lower throughput per server and thus needing more servers to sustain certain site-wide SLA and thus causing more money overall.
Re:Reddit costs $33k/month to operate? by blueben on Monday July 26, 2010 @10:03PM (#1550010)
Do you have evidence that reddit (or any site using a dynamic language) would reduce overall operating costs by switching, or is this an assumption on your part? What about development costs? What about time to deploy new features? What about the cost to make platform revisions? Do you have evidence that a non-dynamic language would really be so much faster that it would require less hardware? Perhaps the problem is not the language, but the chosen architecture?

What I'm really trying to say here is... prove it.

Re:Reddit costs $33k/month to operate? by apower on Tuesday July 27, 2010 @01:03AM (#1550292)
I don't have evidence Reddit would reduce operating costs since they haven't switched yet, and I don't have access to their code/architecture/etc to decide one way or the other. However, based on my personal experience, static type languages are much faster than dynamic languages and use much less resources. They used substantial less servers to maintain the same SLA throughput. You don't have to take this advice and keep paying for the high cost of hardware. There's a belief that developer cost is much higher than hardware cost and thus it's justified. However, when scaling out, hardware cost is much higher than developer cost. Developer cost is a fixed sunken cost at initial development. Afterward it's just maintenance and can be scaled down, but the hardware operation cost is ongoing, increasing years after years.
Re:Reddit costs $33k/month to operate? by jedberg on Tuesday July 27, 2010 @04:15AM (#1550605)
> and I don't have access to their code/architecture/etc

Actually, you do! :) http://code.reddit.com

But to help you out, I'll tell you that a good chunk of the expensive loops are written in C.

Re:Reddit costs $33k/month to operate? by apower on Tuesday July 27, 2010 @11:47AM (#1551587)
Proof that static type languages are much faster than dynamic languages. Reddit has resorted to use C for speedup.
Re:Reddit costs $33k/month to operate? by blueben on Tuesday July 27, 2010 @01:09PM (#1551903)
Proof that in this single instance, Reddit developers decided this was a suitable optimization for their codebase.

This also calls into question the original assumption, that posters believed Reddit had far too many wasteful servers to handle their service, and that it was because they use a dynamic language.

Re:Reddit costs $33k/month to operate? by jedberg on Tuesday July 27, 2010 @02:16PM (#1552125)
Was there ever a question about that? I thought that was pretty much a fact. Of course static typed languages are faster.
Re:Reddit costs $33k/month to operate? by SkyMarshal on Tuesday July 27, 2010 @12:01AM (#1550196)
Difficult to prove, but an observation with precedent. 37 Signals is famous for justifying Ruby by saying that hardware is cheap, developer time is not, so use RoR and just throw servers at your app till it meets your performance needs. Hardware is cheap, but maybe not cheap enough to run a site like Reddit with intentionally scant ad revenue.
Re:Reddit costs $33k/month to operate? by axod on Tuesday July 27, 2010 @04:30AM (#1550642)
Hardware is cheap. Amazon hosting is not.
Re:Reddit costs $33k/month to operate? by apower on Tuesday July 27, 2010 @12:52AM (#1550284)
Do you have evidence that dynamic language have less overall development cost?
Re:Reddit costs $33k/month to operate? by roel_v on Tuesday July 27, 2010 @03:30AM (#1550531)
Because when all you have is a hammer, every problem looks like a nail...
Re:Reddit costs $33k/month to operate? by teej on Monday July 26, 2010 @09:58PM (#1549997)
I ran a web app with identical pageview numbers using Ruby on Rails and EC2. My server costs were 10x cheaper than what reddit is paying.

Scaling is caching and architecture, not writing your app in Java.

Re:Reddit costs $33k/month to operate? by cheald on Monday July 26, 2010 @10:35PM (#1550072)
It's a bold boast, but honestly, it doesn't mean much if the problem domain isn't similar. Reddit in particular runs into issues because so little of what they do is widely cachable. The site is heavily customized per user, and experiences an awful lot of cache churn. I don't disagree with your sentiment (design matters more than choice of language), but I could serve 16 million static pageviews/day off of a $20/month Linode. That doesn't mean that reddit could do the same.
Re:Reddit costs $33k/month to operate? by teej on Monday July 26, 2010 @10:50PM (#1550092)
Reddit has 8 million unique visitors, but only 300k subscribers to /r/pics, the top auto-subscribed subreddit. Generously assuming half of people manually unsubscribe, that's at most 7% of their user base that are logged in. I know full well about cache churn with logged in users, but that's a lot of lurkers.

I'm also not sure what makes a Facebook game's domain more "widely cachable". 100% of users are logged in. The vast majority of actions taken are changing state. Any app touchpoint is writing to the database. Page caching is nearly impossible.

It's a lot less different than you think.

Re:Reddit costs $33k/month to operate? by cheald on Monday July 26, 2010 @11:55PM (#1550186)
I can't claim to know the specifics of your application, but consider that a single upvote invalidates however many thousand copies of a given page. An action in a Facebook game might invalidate a page for 10 or 20 or 50 users. An upvote, (or a comment, or an edit, or an upvote on a comment) invalidates that page for upwards of 300k+ users. Combine that with the rate at which reddit's pages change, and you're talking an obscene amount of CPU time spent building fragments/pages to stick into your cache. Scope of visibility of a change means a lot.

You obviously know how to scale an app if you were pulling 16 million pageviews a day, and I don't intend to discount that at all. I just mean to point out that while fundamentally the same problem, reddit has to deal with a version of that particular problem that most applications don't begin to approach.

Re:Reddit costs $33k/month to operate? by parfe on Tuesday July 27, 2010 @11:12AM (#1551457)
>a single upvote invalidates however many thousand copies of a given page

I never understood this. Does reddit really need to spend the capital making sure I see a stranger's upvote the moment it occurs? A 60 second delay to refresh pages in batches seems perfectly reasonable. Perhaps with a client side script to mark my own upvotes so the system doesn't look like is losing my selections.

Re:Reddit costs $33k/month to operate? by axod on Tuesday July 27, 2010 @04:54AM (#1550677)
I don't know what the "writing your app in Java" bit is about... Care to elaborate?

Mibbit is written in Java (Custom written framework and server), and handles traffic just fine. Java is insanely efficient for network IO.

It's an apples to frogs comparison, but if you measure 'page views' then Mibbit does bajillions, on a handful of servers.

Re:Reddit costs $33k/month to operate? by awa on Monday July 26, 2010 @10:09PM (#1550019)
It would be nice if you elaborate a little.
Re:Reddit costs $33k/month to operate? by teej on Monday July 26, 2010 @10:32PM (#1550067)
I'm not sure what you want to hear, so I'll elaborate a lot.

Warbook was a Facebook application written in Ruby on Rails I ran by myself in late 2007 - 2008. It grew to over 16 million pageviews a day. At the time it was more pageviews than Twitter.

I scaled it using the following stack: Perlbal for load balancing, LightHTTPD for static assets, Mongrel for dynamic requests, Memcached for caching, and MySQL for relational data storage.

I used two medium instances for load balancing, one medium instance for asset hosting, 15 small instances for mongrel, one XL instance for memecached, and one XL instance for MySQL.

I used memcached as a "write-through" cache. Everything in cache was considered fresh. Every write of a cachable object would write to both MySQL and memcached. Every read of a cachable object would start with memcached first and failover to MySQL. This reduced reads on the database by 95%.

Total hosting costs were ~$2,000 a month.

Re:Reddit costs $33k/month to operate? by mdolon on Monday July 26, 2010 @11:58PM (#1550191)
If you don't mind me asking what happened to it? Facebook now reports about 5,500 monthly users. I'm guessing you were acquired by SGN but I'm don't see why that would cause traffic to drop so much.
Re:Reddit costs $33k/month to operate? by teej on Tuesday July 27, 2010 @12:14AM (#1550230)
SGN switched their core focus from Facebook games to iPhone games in mid 2008. At that time, they droped support for all of their Facebook properties except (fluff)Friends. Games on the Facebook platform are doomed to rapid traffic loss without constant adaptation and viral tuning. Even with it, user retention is hard.

Facebook giveth and Facebook taketh away.

Re:Reddit costs $33k/month to operate? by apower on Tuesday July 27, 2010 @11:42AM (#1551561)
16M/24/60/60 = 185 pages/sec. 185/15 = 12 pages/sec/server. That's not much different than the Reddit number.
Re:Reddit costs $33k/month to operate? by CaanaCast on Tuesday July 27, 2010 @01:41AM (#1550345)
could you set up something similar not on EC2?
Re:Reddit costs $33k/month to operate? by masklinn on Tuesday July 27, 2010 @02:17AM (#1550413)
> I ran a web app with identical pageview numbers

That's cute, but was your web app as hard to cache as Reddit?

Scaling a fully static website is trivial, and the more dynamism you include, the harder it becomes.

There is barely anything static on a logged in Reddit user's page.

Re:Reddit costs $33k/month to operate? by rb2k_ on Tuesday July 27, 2010 @04:02AM (#1550579)
Isn't every single subreddit cachable? Only the starting page isn't 100%, but I'm sure that many users have the same combination of subreddits (e.g. the default ones). Would be interesting to see if they have certain "user classes". Something like:

USERCLASS = SHA1(SUBREDDIT1 XOR SUBREDDIT2 XOR SUBREDDIT3)

Re:Reddit costs $33k/month to operate? by masklinn on Tuesday July 27, 2010 @05:30AM (#1550719)
> Isn't every single subreddit cachable?

For logged in users, you're still going to need the voting status for the current user on every single submission as well as that submission's hidden status for the current user to decide whether or not a submission should be displayed in the listing.

I believe they almost never hit the DB directly, so these are probably recached immediately (or submitted to both the cache and the DB at the same time), but that still means quite a lot of traffic.

Re:Reddit costs $33k/month to operate? by jonknee on Tuesday July 27, 2010 @09:40AM (#1551138)
The cache is invalidated after every vote.
Re:Reddit costs $33k/month to operate? by Devilboy on Monday July 26, 2010 @10:18PM (#1550038)
I run 2 Reddits on my calculator watch using C++ and CouchDB so clearly the technology is at fault here.