Cloud computing is cruise control for scaling, but you still have to steer.

How much advance warning did Reddit have that Obama was doing an AMA?

AskReddit thread about the stats: http://www.reddit.com/r/AskReddit/comments/z2fo7/admins_of_r...

Page views/uniques http://i.imgur.com/vPnEK.png

I'd be more interested in seeing how a site using a service like Google App Engine responds. In theory their Cloud Storage should just keep on scaling and scaling, rather than forcing the admin to spin up more instances, set up database replication, etc, etc.

Well, to be fair maybe they didn't have enough notice to spin up additional instances and get replication completed beforehand but I highly doubt that considering how high profile this AMA is. The US President doesn't probably drop in on people / or websites :P

Yeah GAE might work better but they're a bit out of reach mostly due to the technologies that you are required to use. Any migration at that scale is a seriously massive matter.

I guess the funny thing is that you can't plan to grow that big. Usually it just happens suddenly (facebook/reddit style) and then you've got to make some tough architectural calls.

It's funny how folks are like "That's a problem I'd like to have..". I'm not sure they're very happy with this problem at this very moment.. sigh

I don't know about websites, but the US President drops in on lots of places (restaurants for instance) as a matter of security. From what I understand with interviews, it's often not uncommon for the interviewer to get very little notice either at times.

AWS Elastic Beanstalk will do the same thing.

Beanstalk has nothing to do with the database, which is the point of contention. Spinning up more app servers only helps with serving signed in users.

You can configure beanstalk to scale your database out as well.

Scaling out database is nothing at all like spinning up stateless app servers.

Well there is no magical database solution where you can do that and I have to make decisions on what I can buy today not on what may be magically developed in the future.

In this respect, application partitioning with a deployable schema is not a bad solution for what I do.

AFAIK Reddit aren't using it. If anyone can clarify I'd love to hear more.

I agree that this is more of a problem with elastic cloud services than reddit engineers. I am sure that they anticipated the load and did what they could, but the traffic is surely immense

Stupid question: is the because reddits backend is in python?

I often hear that PHP + varnish is only limited by the size of the pipe. Can this scale on cloud platforms?

It's a stupid question because the particular dialect of assembler this was written in is completely irrelevant for performance considerations.

You've been brainwashed by the idiotic pop culture language wars that pretend a question like Ruby vs Python vs whatever has any meaning. These are debates over buzzwords, not reality.

All that matters is the database configuration and associated architecture. Not some buzzword drama.

Wow, news.ycombinator surely has become very offensive and attacking. So much that a person cannot ask questions about IT.

"become" ?

4 years ago, stupid comments were brutally discouraged. People with stupid things to say would be dogpiled by negative comments.

It is pop culture that thinks asking stupid questions is a virtue.

Most people are just too tactful to tell the truth.

The main issue with his post is he has no idea what he's talking about. He mentions languages neither interpreter is written in and talks about 'buzzwords', while all the products I mentioned are real things.

I'm not sure what you're talking about. Both Python, PHP and Varnish are (mostly) written in higher level languages.

Also performance has a massive effect on how and what language a site is written in.

>I'm not sure what you're talking about.

My comment is sarcastic and abstract. A literal interpretation will not suffice to understand my meaning.

If you want it literally: the language this is written in is irrelevant, because the performance differences between the various languages are negligible next to the performance differences that come about due to the precise design of the website.

A poorly engineered website in a "fast" language will still be slower than a well-engineered website in a "slow" language. The actual language is completely irrelevant next to the implementation details.

>Also performance has a massive effect on how and what language a site is written in.

Not really. Experienced engineers understand that language is basically irrelevant for a situation like this. Whether it is written in Python or something else is irrelevant information.

Additionally, newbies tend to focus on "language" and "language wars" and the comparison between languages not for any engineering or technical reason. Newbies do this because languages represent buzzwords around which various communities congregate. The Python vs. Ruby fight is as irrelevant as the Presbyterian vs. Baptist fight. It is the psychological appeal of "identity politics" that motivates people to engage in these meaningless debates about buzzwords.

No, it's probably due to the massive number of reads/writes pumping through their system. I'm pretty sure that their database isn't able to handle what's going on at the moment.

Also, if you were to use Varnish as a cache, that wouldn't help you much with write access because every time there is a write (a new post..) you'd have to invalidate the cache. In a high volume scenario such as this (tons of posts coming in..), the page is as good as dynamic, even if you DO cache it for a bit.

Varnish can only really help Reddit with users who aren't logged in since every page is customized. It'd still help in high volume scenarios since Varnish solves the thundering herd problem: only one request (at most) will hit the database for each invalidation.

Vanish can cache snippets of content. They refer to this as Edge Side Includes.

Reddit's customization is far more than snippets. I've used ESIs with Varnish extensively, and I don't think it would help much for authenticated Reddit users.

Thank you, this is the answer that I was looking for

Why ask it if you know it's stupid?

No, PHP+Varnish can not always scale. You're forgeting the most important important factor in this equation, the database. Varnish is simply for caching and you won't find any large site that doesn't have it. PHP is a scripting language and the runtime performance differences to other common web application languages (i.e. Python, Java) is negligible since you're using caching anyway. Here's what you should worry about if you want scaling: Your database, your data structures and your code quality.

reddit's backend could be written in assembly hand-tuned by God Himself, and having half the world hit it at once would still cause bottlenecks at the database.

The problem is that pretty much all of it is dynamically generated. Short of the sidebar & header there is nothing that is really static static. e.g. With all the comments it checks whether that person is on your friend list & colours those differently. Easy enough to do...but it doesn't scale well. They've got a blog explaining the issue(s) somewhere.

Looks like part of their infrastructure is still up. website is down, but I can connect on my mobile using baconreader app

I'm still able to access the AMA through Alien Blue on iPhone, so that part of their infrastructure must still be up.

I believe it was a surprise AMA, so no heads up to Reddit team.