Ask YC: Any ideas about intelligent crawlers :)

Re:Ask YC: Any ideas about intelligent crawlers :) by ivan on Tuesday January 08, 2008 @03:16PM (#96090)

And still one thing .. if you want to ask the site owner for permissions, why not ask them to produce some specific xml file for you?

[ Parent | on HN ]

Re:Ask YC: Any ideas about intelligent crawlers :) by akkartik on Tuesday January 08, 2008 @03:17PM (#96091)

Because granting permission is easy. Why would they go to more effort than that for random people?

[ Parent | on HN ]

Re:Ask YC: Any ideas about intelligent crawlers :) by ivan on Tuesday January 08, 2008 @03:20PM (#96095)

Why thousands of job sites produce custom xml output for simplyhired or indeed ??

[ Parent | on HN ]

Re:Ask YC: Any ideas about intelligent crawlers :) by imsteve on Wednesday January 09, 2008 @09:05AM (#96339)

They like buzzwords?

[ Parent | on HN ]

Re:Ask YC: Any ideas about intelligent crawlers :) by marketer on Tuesday January 08, 2008 @04:20PM (#96122)

I think this is the best way to go. There's no reason that you should be scraping HTML from sites, when there might be a nice xml feed available. For instance, pricegrabber will only index your site if they have prices in XML.

[ Parent | on HN ]