Re:Ask PG: Polite ways to scrape HN?
by jrockway on Thursday September 23, 2010 @11:13PM
(#1722501)
I think the issue is that news.arc doesn't use a distributable storage system. It is as far from "shared nothing" as you get.
[ Parent
| on HN ]
Hm, interesting.
You can scale anything, though. For example, a pair of reverse proxies in front of the single "share nothing" app server could reduce the load on that one machine. I'm sure the collective brilliance of the HN readership could come up with solutions.
Simple: buy a bigger server. It's amazing how powerful one machine can be these days.
Assuming nothing has changed, it's running on a "3.0 GHz Core whatever, 12 GB RAM, 64-bit FreeBSD 7.1."
Upgraded on 4/19/09 from a 2.4 GHz Pentium 4, 4 GB RAM, 32-bit FreeBSD 5.3.
rtm's comment: http://news.ycombinator.com/item?id=516122
Yeah, well, this is a middle of the road server these days. I have a 16-core server with 64G of RAM as a dev server at work. They get even more powerful than this, too.
Not saying pg should spend any money on this, though. A redesign of news.arc will stretch the hardware a lot further.