News for hackers. Stuff that matters.

Hacker News search

I'm starting to do 'Introduction to CS' from Udacity (http://www.udacity.com/overview/Course/cs101/CourseRev/apr2012). I'm planning to make a Hacker news search app through the course. What do you think, is that a good idea, or not? I know there is something similar; this is my plan: the search will be executed in the domains, referred by the hacker news stories. Thank you
( 13 comments | 10 points | on HN )
Re:Hacker News search by slash-dot on Sunday December 09, 2012 @01:32PM (#4895477)
Keep in mind that if you crawl this site your ip will be banned. At least mine was when I was playing around with a web crawler i built.
Re:Hacker News search by dbaupp on Sunday December 09, 2012 @01:42PM (#4895520)
There is an unofficial API: http://www.hnsearch.com/api (Provided by the very search engine referred to in the OP, haha!)
Re:Hacker News search by pyre on Sunday December 09, 2012 @01:49PM (#4895551)
Unfortunately there is no API for getting access to personal information on HN (i.e. comments I have made, or stories I've upvoted). You're relegated to scraping if you want that information.
Re:Hacker News search by unholygoat on Monday December 17, 2012 @06:36AM (#4931875)
now there is :

here is how you can pull a specific username's submissions and you can add filters: http://api.thriftdb.com/api.hnsearch.com/items/_search?filte...

And then here is how you can pull the comments for a specific thread/discussion/id: http://api.thriftdb.com/api.hnsearch.com/items/_search?filte...

you can now grab a lot of data.. including they enlarged the site's rss feed in hopes of slowing a few of the scrapers..

there are a few items missing, but they added a lot: http://www.hnsearch.com/api

btw that includes a user bio now, as well as things you've upvoted... etc.. its all just done via filters..

the also boosted the rss feed to help slow down the strapers

Re:Hacker News search by beatgammit on Sunday December 09, 2012 @01:59PM (#4895611)
You can always get around this by throttling your web crawler. It will take a much longer time, but at least you'll be able to read HN in the meantime.
Re:Hacker News search by alphast0rm on Sunday December 09, 2012 @03:37PM (#4896026)
The tricky thing when doing this is knowing what rate to stop at without getting permanently banned. I built an Android Market crawler two summers ago, and luckily Google only temp bans (from my experience), so that might be an easier project without any risk.
Re:Hacker News search by quesera on Sunday December 09, 2012 @09:59PM (#4897254)
Respecting robots.txt is probably the best plan.
Re:Hacker News search by ig1 on Sunday December 09, 2012 @05:56PM (#4896537)
Use disposable IPs.
Re:Hacker News search by ymn_ayk on Sunday December 09, 2012 @03:06PM (#4895904)
Why will you be banned? Do you know about the reason?
Re:Hacker News search by jgeralnik on Sunday December 09, 2012 @03:50PM (#4896085)
To prevent people from (unintentionally) DDoSing the site.
Re:Hacker News search by unholygoat on Monday December 17, 2012 @06:40AM (#4931887)
there's a bunch simliar but i don't think any have updated with the newly added API additions...go for it! http://www.hnsearch.com/api

here are most of the other apps still up http://news.ycombinator.com/item?id=2672826

Re:Hacker News search by rikacomet on Sunday December 09, 2012 @01:51PM (#4895560)
speaking of this, there should be one. And a "http://Archive as well.

only a handful of topics make it to the top everyday, but the majority of content is high quality stuff, which if applied at a proper place would help many. What say?

Re:Hacker News search by arikrak on Sunday December 09, 2012 @04:50PM (#4896277)
I think a practical way to implement simple version would be through Google CSE, but you would have more control if you roll your own search.