News for hackers. Stuff that matters.
Re:HackerNews API: What if HN does not have API? Make API on... by fizx on Saturday June 02, 2012 @07:16PM (#4059496)
Hah! tectonic and I applied to YC with almost exactly this in 2009?!

We went as far as building a browser-based IDE-like environment for generating these, and a language called parsley for expressing the scrapes. If you're interested in this, you could check out some of our related open source libraries:

Edit: I just open-sourced the scraping wiki site we created here: https://github.com/fizx/parselets_com

http://selectorgadget.com

https://github.com/fizx/parsley

https://github.com/fizx/parsley-ruby

https://github.com/fizx/pyparsley

https://github.com/fizx/csvget

    > cat hn.let
    {
      "headlines":[{
        "title": ".title a",
        "link": ".title a @href",
        "comments": "match(.subtext a:nth-child(3), '\\d+')",
        "user": ".subtext a:nth-child(2)",
        "score": "match(.subtext span, '\\d+')",
        "time": "match(.subtext, '\\d+\\s+\\w+\\s+ago')"
      }]
    }
    > csvget --directory-prefix=./data  -A "/x" -w 5 --parselet=hn.let http://news.ycombinator.com/
    > head data/headlines.csv
    comments,title,time,link,score,user
    4,Simpson's paradox: why mistrust seemingly simple statistics,2 hours ago,http://en.wikipedia.org/wiki/Simpson%27s_paradox,41,waldrews
    67,America's unjust sex laws,2 hours ago,http://www.economist.com/opinion/displaystory.cfm?story_id=14165460,59,MikeCapone
    23,Buy somebody lunch,3 hours ago,http://www.whattofix.com/blog/archives/2009/08/buy-somebody-lu.php,58,DanielBMarkham
Re:HackerNews API: What if HN does not have API? Make API on... by sc00ter on Saturday June 02, 2012 @07:39PM (#4059550)
Oh! Selector gadget is wonderful, and I still use it to this day. It's so easy to forget that there are real people behind the tools we use - so fizx and tectonic, I thank you!
Re:HackerNews API: What if HN does not have API? Make API on... by christiangenco on Saturday June 02, 2012 @11:34PM (#4059982)
Seconded - selector gadget is one of the best scraping tools around. You've saved me many a frustrated evenings.
Re:HackerNews API: What if HN does not have API? Make API on... by tectonic on Sunday June 03, 2012 @06:05PM (#4061873)
Thanks for the kind words! Maybe this attention will push me to update the code and make a new screencast.
Re:HackerNews API: What if HN does not have API? Make API on... by tluyben2 on Sunday June 03, 2012 @06:23AM (#4060474)
Selectorgadget is great; without checking out your creations in real life, did you consider taping selectorgadget to a proxy so you can scrape sites and store the paths you found in one go? That would massively enhance the process imho :) Maybe Apify can do that, but I hope they put that in github as well; i'm not a great fan of closed source/cloud development tools.
Re:HackerNews API: What if HN does not have API? Make API on... by sathish316 on Saturday June 02, 2012 @07:52PM (#4059578)
Do you mind if i use selector gadget in APIfy? I use firebug and firefinder to get selectors. But i can't expect everyone to have the same tools.
Re:HackerNews API: What if HN does not have API? Make API on... by tectonic on Sunday June 03, 2012 @06:03PM (#4061869)
Please do use selectorgadget! And if you'd like to push anything back, or chat about parsing in general, send me a message. I have a more advanced branch of SG that can generate better selectors, but I haven't pushed it out yet. It's all in the repo.
Re:HackerNews API: What if HN does not have API? Make API on... by fizx on Saturday June 02, 2012 @10:39PM (#4059866)
Everything mentioned is MIT-licensed, and like many OSS authors, we would love any ideas or code in there to be widely used in virtually any capacity.

Also, if you're in or near SF, I'd be happy to get coffee sometime.

Re:HackerNews API: What if HN does not have API? Make API on... by sathish316 on Tuesday June 12, 2012 @10:51AM (#4100708)
Thanks for suggesting SelectorGadget. I ended up building an in-browser firebug+firefinder clone to make the app even more easier to use. Please check out Preview webpage for any API
Re:HackerNews API: What if HN does not have API? Make API on... by tluyben2 on Sunday June 03, 2012 @06:21AM (#4060470)
I was going to suggest here that you do. And opensource your creation on github so we can all improve it.