News for hackers. Stuff that matters.
Re:Ask HN: Best tool for technical web crawler data (entire ... by veggies_4us on Tuesday February 13, 2018 @06:03PM (#16371719)
they don't advertise it well, but nerdydata's custom reports had some excellent results... way more than their regular search website shows.

They were also able to customize our search and extract specific data from the page. pretty advanced stuff.

Re:Ask HN: Best tool for technical web crawler data (entire ... by ghawkescs on Tuesday February 13, 2018 @07:06PM (#16372155)
That sounds promising, can they run a custom report across their entire index? I'm trying to get a count of competing products to gauge market size and interest.
Re:Ask HN: Best tool for technical web crawler data (entire ... by veggies_4us on Wednesday February 14, 2018 @09:37AM (#16375966)
They gave us the option of either running a report on their entire index, OR if we had a list of domains to search explicitly (which we didn't have).

Our use case was different, though, we wanted to find websites using a javascript library, and then run a javascript function on all of those sites to extract account information and specific html patterns.

But yes, their larger custom reports index did the trick, would recommend.

Re:Ask HN: Best tool for technical web crawler data (entire ... by ghawkescs on Wednesday February 14, 2018 @11:12AM (#16376839)
Great to know, thank you. Was the pricing reasonable?