Any good api to scrape HN other than this?

Re:Any good api to scrape HN other than this? by carbocation on Monday December 23, 2013 @02:59AM (#6953529)

The robots.txt from news.ycombinator.com reads as follows:

    User-Agent: * 
    Disallow: /x?
    Disallow: /vote?
    Disallow: /reply?
    Disallow: /submitted?
    Disallow: /submitlink?
    Disallow: /threads?
    Crawl-delay: 30

So nominally you should feel free to set up a scraper that crawls one non-disallowed resource every 30 seconds.

[ Parent | on HN ]

Re:Any good api to scrape HN other than this? by wslh on Monday December 23, 2013 @06:33AM (#6953862)

I always asked myself where the HN's data portability policy is? When I back up my submissions and comments I am forced to break the rules.

It would be good to have a way to download ALL your stuff. Ask PG?

[ Parent | on HN ]

Re:Any good api to scrape HN other than this? by rrpadhy on Monday December 23, 2013 @09:32AM (#6954387)

I see that submitted and threads are also not allowed.

What is a safe limit to crawl this data, if I have to absolutely need that data? 30 mins between users? 1 hour between users?

[ Parent | on HN ]

Re:Any good api to scrape HN other than this? by kaushikfrnd on Monday December 23, 2013 @01:25PM (#6955682)

i am following hnsearch api for long and found that they crawl user submission url and user details urls every 2-4 hours or so .

[ Parent | on HN ]

Re:Any good api to scrape HN other than this? by nashequilibrium on Monday December 23, 2013 @06:21AM (#6953848)

rp = robotparser.RobotFileParser()

rp.set_url("https://news.ycombinator.com/news/robots.txt")

rp.read()

# Reads the robots.txt

rp.can_fetch("*", 'https://news.ycombinator.com/news')

>>>> True

[ Parent | on HN ]

Re:Any good api to scrape HN other than this? by kaushikfrnd on Monday December 23, 2013 @07:03AM (#6953919)

cool

[ Parent | on HN ]

Re:Any good api to scrape HN other than this? by t0 on Monday December 23, 2013 @05:26AM (#6953773)

But /x? is for the next page.

[ Parent | on HN ]

Re:Any good api to scrape HN other than this? by pedrocr on Monday December 23, 2013 @07:31AM (#6953982)

So apparently you can get two pages of ranking, using / and /news2.

[ Parent | on HN ]

Re:Any good api to scrape HN other than this? by randomdata on Monday December 23, 2013 @08:24AM (#6954119)

Depends on the intent. If it is user-initiated (like say a mobile formatted version of the site), it wouldn't have to be obey the robots.txt, since it is not a crawler, just another web browser.

[ Parent | on HN ]

Re:Any good api to scrape HN other than this? by kaushikfrnd on Monday December 23, 2013 @05:31AM (#6953780)

well i am trying to get user submissions also so may be i have to violate the robots.txt

[ Parent | on HN ]