We finally have an explanation for what happened. Our site was turned off by our hosts, who couldn't believe the traffic was legit. Maybe time to look for a different hosting company...

http://news.ycombinator.com/item?id=1283430

Considering what a big deal "reliability" is to most hosting companies (and customers), this is a shockingly stupid response.

Wow - and they didn't even contact you about it? I thought this had been a pretty high traffic, well known site for a while.

Was there a large jump in traffic, or are they just not willing or able to host a site of this size?

No, it was fairly gradual.

For a few days, I've had the problem where the threads page (e.g., http://news.ycombinator.com/threads?id=zck) takes a long time to load (> 30 seconds), and sometimes doesn't load at all, ending up with just a blank page -- no html is loaded.

This is a different problem from the one we're experiencing now. Threads takes a long time the first time because items are lazily loaded.

Why lazily load items yourself instead of use a database, like Postgres or even SQLite, that handles the paging in and out itself?

Because hacker news doesn't use a database, the site is file driven and runs entirely out of ram. Changes are made in memory and logged to disk and upon server restart, the files are lazily loaded on demand to prevent the need to load the entire db all at once. This might sound crazy but it's actually faster (once loaded) and easier to program than a relational database.

As hacker news was a fun project for Paul, I'd imagine he doesn't want to touch a relational database with a ten foot pole; I don't blame him. You can get a long way without ever using a real database, files and in memory hash tables work fine up to a pretty decent amount of traffic. It only becomes problematic when you need more than one server.

Interesting; I have noticed that once the threads page loads, it generally works fine the second time. I've been trying not to hammer on it too much to see when it works/doesn't work, though.

I guess it's one way to tell if the server's been restarted since I last was here. :)

This has been happening to me for weeks.

This has been happening to me since I signed up.

http://ycombinator.com is down, and it looks like Paul is hosting the static content of the site there...

Yes, that's the problem. We just switched over to serving the static stuff off this server while we investigate. The extra load may make things here a bit slower but the site seems usable.

Is the change to 25 front page stories a temporary one, to help performance?

Yes. I tweaked several variables to decrease load.

Does HN run on srv.arc and app.arc?

Yes. The code I released is identical to the files on the server. Or at least to a snapshot of them in the past.

did i win? http://twitter.com/mseiler1/status/12598697037 or was i too slow :)

HN is the single slowest website I use regularly and not just because the www server was cut off today. Typical page loads are 10+ seconds and if I want to view my threads page it usually times out before finishing on the first request (sometimes it's the same for the home page). It's quite absurd considering the audience.

Oh and it took three tries to post this comment, I kept getting timeouts.

Same here. Very slow for me, and when I try to load my comments or submissions pages, I often get a timeout and have to try 2-3 times.

I've always found it a bit ironic that a website where there's probably one of the highest concentration of good coders/web people was so slow and unreliable. But maybe that's just the nature of the beast; reddit is slow too, and they have a lot more resources.

Perhaps it's you, or latency to you, the sites always quite snappy and fast for me and I never see load times like that. Most pages load in 700-800 milliseconds for me and pretty much always have.

Other sites in the same data center are snappy and I've read enough other identical complaints to be sure that it's not just me.

Not all pages are slow, and not all the time (though pretty frequently).

But the "threads/comments/submissions" pages are almost always slow.

From what I've read it's because pg uses this as a playground for arc and is far from optimized. It would be great if he got some help from the community though, it is embarrassingly slow.

The same here. That was AFTER the fix that pg announced of changing his static content over to a new hosting service.

Pair.com decided to shut down our hosting, because we got 150k hits in 30 minutes. Not that much, really. Anyway, I switched to Slicehost.

are you seriously hosting news.yc on slicehost?

Not news, www. They're separate servers.

gmartin$ whois -h whois.arin.net 174.132.225.106

OrgName: ThePlanet.com Internet Services, Inc. OrgID: TPCM Address: 315 Capitol Address: Suite 205 City: Houston StateProv: TX PostalCode: 77002 Country: US

It seems to be an on-and-off thing. The comments and submissions links on profiles are the worst offenders.

Yep, that's a problem for me too. Images and hyperlinks aren't rendering properly. Checked HN in Chrome, Firefox and Safari to make sure, and cleared cache.

I've had a 15-30 second delay after clicking onto the comments thread for a few days now.

A possibly related issue is that I occasionally find myself logged in as someone else when I load the front page (ie some other user's name/karma are displayed in the top right). This goes away after refreshing or loading any other link (so I can't post as someone else, for example).

Plus I get that "unknown or expired link" error sometimes when trying to post (including when trying to post this message). This seems to occur when clicking on a thread from the front page loads an out-of-date version of the thread, and can be fixed by force-refreshing the thread page.

Slow on Threads page, images didn't load recently. Images fine now (but not Threads)

However... The 'GREEN Certified Site' image now loads after about 6 months as a dead image!

Same here—sometimes when the rest of HN seems to be working normally, I end up with timeouts on Threads.

I have been having a lot of stupid caching behavior, too (I think I have a transparent cache between me and yc, and there doesn't seem to be an Expires header. Some article pages show me as logged out, or when I try to reply, I get deleted/expired link message.

It's worth noting that Chromium was still giving me bad images/css even after the outage until I moved to a completely new tab. (Even with trying ctrl+F5, though I'm unsure that Chromium binds that to 'reload + reload cache' like IE did/does.)

not slow... but images are not showing up for me.

Its good to me now.

Yes, no images.

seemed to be temporary...

yes!