Re:Why scraping and ecommerce are a perfect fit
by dmritard96 on Thursday November 20, 2014 @12:47AM
(#8634582)
hm, I didn't mean infrastructure, I mean, did you buy proxy nodes from someone like sslprivateproxy and slap HAProxy in front...
Most ecommerce sites wise up to bots crawling them and have a robots.txt that suggests you might not want to...
We have our placing orders infrastructure on AWS, and whole in-house cloud dedicated to product crawling built on top of Digital Ocean.