Re:Why scraping and ecommerce are a perfect fit
by dmritard96 on Wednesday November 12, 2014 @04:59PM
(#8598531)
How many proxy nodes do you have?
[ Parent
| on HN ]
Re:Why scraping and ecommerce are a perfect fit
by sradu on Wednesday November 12, 2014 @05:04PM
(#8598558)
Paraphrasing newrelic "it takes a village to count our proxy nodes".
We have our placing orders infrastructure on AWS, and whole in-house cloud dedicated to product crawling built on top of Digital Ocean.
Re:Why scraping and ecommerce are a perfect fit
by dmritard96 on Thursday November 20, 2014 @12:47AM
(#8634582)
hm, I didn't mean infrastructure, I mean, did you buy proxy nodes from someone like sslprivateproxy and slap HAProxy in front...
Most ecommerce sites wise up to bots crawling them and have a robots.txt that suggests you might not want to...