Ask HN: Need some tips for web crawling
Hello fellow hackers,I am trying to create a tool for myself that can crawl a few websites that I usually go on to compare the price of the same item. They have no APIs.A couple questions:1. is this LEGAL? 2. if I am crawling, what is the best way to approach this? does each website's crawling mechanism have to be manually written since they are unique or is there some strategy for scale if i...