News for hackers. Stuff that matters.
Re:Ask PG: Are there IP blocks? by pg on Friday November 04, 2011 @12:21PM (#3196616)
Yes, we block IPs that seem to be crawlers ignoring robots.txt. We've always blocked abusive IPs, but I tightened up the blocking a few weeks ago. A lot of people were crawling HN, most of them unnecessarily because they were doing things they could have done more efficiently through HNSearch's API.

Some users may remember that the site had gotten really slow a few weeks ago. One of the reasons it's faster now is that we cracked down on crawlers.

Re:Ask PG: Are there IP blocks? by halostatue on Friday November 04, 2011 @12:43PM (#3196715)
That makes sense. I suspect that the "cause" in this case was the chrome extension that I had been using (I don't remember the name offhand, but it's the one that shows changes in HN between visits). I usually left the tab open but only visited a couple of times a day, so it may have been doing something like that.
Re:Ask PG: Are there IP blocks? by ronnier on Friday November 04, 2011 @01:21PM (#3196883)
I think it's time I shut down http://ihackernews.com. It's getting impossible to keep the IP address from being blocked. Really unfortunate, because a lot of people like to use the site.
Re:Ask PG: Are there IP blocks? by pg on Friday November 04, 2011 @02:18PM (#3197131)
If you tell me the ip addr I can whitelist it. [email protected]
Re:Ask PG: Are there IP blocks? by ronnier on Friday November 04, 2011 @02:30PM (#3197173)
Wow, thanks! Email sent.
Re:Ask PG: Are there IP blocks? by Timothee on Friday November 04, 2011 @01:51PM (#3197023)
I had to stop using it because of that. And I started to think of re-writing it as a self-hosted solution to hopefully avoid IP bans.

I don't know if you'd consider releasing it for anyone to install on their own server, and if pg would be fine with that solution. (assuming that each page load would dynamically scrape a single page on ycombinator.com)

edit: of course, it might still have its issues if a lot of people were to host it with the same host…

Another solution would be for pg to add a mobile stylesheet. That's honestly the only reason why I would need it.

Re:Ask PG: Are there IP blocks? by polyfractal on Friday November 04, 2011 @01:35PM (#3196960)
Anyone know if this what the WP7 HN App uses? I've noticed that the app no longer works which is a shame, I used it all the time.
Re:Ask PG: Are there IP blocks? by wx77 on Friday November 04, 2011 @02:01PM (#3197061)
Appears so: https://github.com/jpf/hacker-news-wp7/blob/master/README

If HN could give back json instead of html I'm guessing a lot of the crawling could be mitigated to client side. Kind of like appending .json to everything on reddit