It probably won't solve the problems you're working on, but I could imagine quite a lot of interesting text analysis cases.
If I get back into blogging, I would be really happy to have my posts structured and indexed in whatever standard way made them easier bots to use.
That being said, I find this highly interesting, if it works like that. We are working on a peer-to-peer database that lets you query a semantic database, popularized mostly by public web data, but with strong guarantees of accurate and timely data, and this could be a great way to write more robust linked-data converters.
I think this is where the web is headed - where common users gain the ability to perform tasks that currently only developers or technical experts can do.
Every crawler task requires different paging methods, different xpath patterns, etc that it makes things more complicated to generalize it.
What if they did? Would you buy it then? What could they possibly offer you before you'd be willing to use their product?
1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted from some obscure xpath)
2. Second, most people are interested in daily snapshots of a page. Like the daily prices of items in an ecommerce store. Not a static, one time snapshot.
I strongly believe crawling is something that can rarely be productized. the needs are so different for every use case. And even if you were to provide a product that would make crawling obsolete, I would still never use it. Because I dont trust you crawled the data correctly. And clean, accurate data is everything