News for hackers. Stuff that matters.
Re:Ask PG: Polite ways to scrape HN? by pg on Thursday September 23, 2010 @07:33PM (#1721997)
You can crawl so long as you respect robots.txt and don't retrieve more than a couple pages a minute. (Conservative, I know, but we're serving 800k pages a day off one server, using an app written in a slow language.)
Re:Ask PG: Polite ways to scrape HN? by portman on Thursday September 23, 2010 @11:10PM (#1722497)
Paul, have you considered asking the community to volunteer resources?

You've mentioned several times that the site is on one server and that performance is an issue at times. I would donate time/money to make those problems go away, and I suspect others would as well.

Proposal: Post a new thread, "somebody build me a HN server farm", and include the software and hardware prerequisites for an Arc webserver. I bet within 48 hours you would have a 3-4 servers and a load balancer at your disposal.

Re:Ask PG: Polite ways to scrape HN? by jrockway on Thursday September 23, 2010 @11:13PM (#1722501)
I think the issue is that news.arc doesn't use a distributable storage system. It is as far from "shared nothing" as you get.
Re:Ask PG: Polite ways to scrape HN? by portman on Thursday September 23, 2010 @11:27PM (#1722548)
Hm, interesting.

You can scale anything, though. For example, a pair of reverse proxies in front of the single "share nothing" app server could reduce the load on that one machine. I'm sure the collective brilliance of the HN readership could come up with solutions.

Re:Ask PG: Polite ways to scrape HN? by staunch on Friday September 24, 2010 @12:04AM (#1722619)
Simple: buy a bigger server. It's amazing how powerful one machine can be these days.
Re:Ask PG: Polite ways to scrape HN? by grinich on Friday September 24, 2010 @04:52PM (#1725380)
Assuming nothing has changed, it's running on a "3.0 GHz Core whatever, 12 GB RAM, 64-bit FreeBSD 7.1."

Upgraded on 4/19/09 from a 2.4 GHz Pentium 4, 4 GB RAM, 32-bit FreeBSD 5.3.

rtm's comment: http://news.ycombinator.com/item?id=516122

Re:Ask PG: Polite ways to scrape HN? by jrockway on Friday September 24, 2010 @08:46PM (#1725960)
Yeah, well, this is a middle of the road server these days. I have a 16-core server with 64G of RAM as a dev server at work. They get even more powerful than this, too.

Not saying pg should spend any money on this, though. A redesign of news.arc will stretch the hardware a lot further.

Re:Ask PG: Polite ways to scrape HN? by siculars on Thursday September 23, 2010 @09:58PM (#1722304)
Thank you!
Re:Ask PG: Polite ways to scrape HN? by grails4life on Friday September 24, 2010 @05:11PM (#1725446)
Are you calling Arc Lisp slow?