Re:Yahoo open sources Anthelion web crawler for parsing stru...
by loopbit on Wednesday December 16, 2015 @07:29AM
(#10743808)
Just a couple of minor clarifications:
- Athelion is a plugin for Apache Nutch, which is the web crawler part and has been open source for a long time.
- As far as I can tell, Athelion parses structured data (microformats, microdata, RDFa...) but that's not the most interesting bit. The online classifier of pages and scoring of new links discovered looks like the real important piece.
- Actually, the other two parts of the plugin are modifications of existing Nutch plugins.
All in all, I can't wait to have some time to see it working.
- Athelion is a plugin for Apache Nutch, which is the web crawler part and has been open source for a long time.
- As far as I can tell, Athelion parses structured data (microformats, microdata, RDFa...) but that's not the most interesting bit. The online classifier of pages and scoring of new links discovered looks like the real important piece.
- Actually, the other two parts of the plugin are modifications of existing Nutch plugins.
All in all, I can't wait to have some time to see it working.