Firefox removes core product support for RSS/Atom feeds

Re:Firefox removes core product support for RSS/Atom feeds by kbenson on Friday October 12, 2018 @06:53PM (#18205290)

AWS is far from a cheap provider. DigitalOcean (as well as many others) have 1GB instances with 25GB disks for $5/mo. Some allow additional IPs for a few dollars. Deploy a bunch of minimal instances that all they do is run a socks proxy and are firewalled off from everyone except your source IP address, and use them as the configured proxy for your actual scanning machine(s). For chrome, you can configure a proxy on the command line. Most quality http libraries in most languages support proxies (otherwise I wouldn't consider them quality libraries...).

For a business (which what I'm thinking in terms of), $500/mo for 100 IP addresses which you can trivially cycle by destroying and creating new virts is well within cost. For an individual, I assume you wouldn't be crawling 24/7, so bring up as many as needed for short bursts. At $0.007/hr, you could bring up 100 from a snapshot/template, use them for a few hours to crawl whatever you need, and it will cost you just over $2.[1] For longer term but smaller needs, just spin up 5-10 for $25-$50/mo cost.

1: 100 * $0.007/hr * 3 hours = $2.10

[ Parent | on HN ]

Re:Firefox removes core product support for RSS/Atom feeds by jjeaff on Saturday October 13, 2018 @12:01AM (#18206365)

You also have to factor in the data transfer which will add up quickly. Aws also charges an extra fee for cycling through IP addresses. I think maybe 25 makes it under the free tier, but more than that and it will start to add up as well.

DO data is cheaper, but not free, and I'm not sure if they charge for changing IP addresses a lot.

Not to mention that in many cases, admins start blocking while up ranges. And due to scraping, lots of sites block popular ip ranges like that of AWS. There are actually forums where people keep updated ranges for the different cloud platforms.

[ Parent | on HN ]