Re:HackerNews API: What if HN does not have API? Make API on...
by sathish316 on Sunday June 03, 2012 @05:18AM
(#4060405)
If html content is loaded using ajax, you have to give the URL of the Ajax request html. See http://apify.heroku.com/resources/4fc2a234ae684d0001000008/e... for example.
It won't work for #! URLs. Twitter has a nice streaming API for search if that's what you're looking for.
Re:HackerNews API: What if HN does not have API? Make API on...
by gildas on Sunday June 03, 2012 @05:35AM
(#4060427)
* Ajax responses are often JSON content. How do I apply a CSS/XPath selector on it?
* I guess #! URLs could be transformed into _escaped_fragment_ URLs [1]
* I know twitter has an API. It was just an example. Maybe this example [2] would be more relevant (content could also be fetched with an _escaped_fragment_ URL).
[1] https://developers.google.com/webmasters/ajax-crawling/docs/...
[2] http://apify.heroku.com/resources/4fcb2f7ba06a160001000044
Re:HackerNews API: What if HN does not have API? Make API on...
by 6ren on Sunday June 03, 2012 @06:10AM
(#4060462)
You need to specify a unique index for the attribute - you specified it as "tweet", but not how to scrape it. The error messages are not informative at this stage; it might be something else too.
I tried to fix your e.g. but couldn't get it to work (I tried //span@data-time (xpath) - what is the unique index of a tweet?)
"API call failed: Internal Server Error"
[1] http://apify.heroku.com/resources/4fcb23c5a06a160001000014