Crawlers and anti-crawler wasted too much resource, data shared in open database is a great idea. BTW, if someone can maintain all data listed here, that will be really helpful. https://github.com/awesomedata/awesome-public-datasets