Re:Who needs to scrape millions of pages, or monitor them?
by volokoumphetico on Monday September 09, 2013 @07:08PM
(#6356908)
very cool, is it doing a depth first blind crawl of any domain you throw at it?
[ Parent
| on HN ]
Re:Who needs to scrape millions of pages, or monitor them?
by calufa on Monday September 09, 2013 @10:11PM
(#6357662)
It will basically go through all the links it finds, that can be millions of links. You can also tell it to ignore certain links using regular expressions or via tales java apis.
Re:Who needs to scrape millions of pages, or monitor them?
by calufa on Monday September 09, 2013 @10:43PM
(#6357814)
Here is a sample code of a 1 depth scrape:
https://github.com/calufa/tales-templates/blob/master/core/s...
This is the api call to start the scraper on twitter:
http://localhost:8080/start?process=tales.scrapers.LoopScrap... -template tales.templates.FirstDepthTemplate -threads 2 -namespace com_twitter -baseURL twitter.com