if your IPs are in a contiguous /25 , they can all be blocked with a single firewall rule. And detection is also easy. It only helps with rate limits.
To avoid a block, you need a list of good socks5 proxies
> if your IPs are in a contiguous /25
They're not.
> To avoid a block, you need a list of good socks5 proxies
No; furthermore, that leaks your data to a third party and introduces significant latency.
With 128GB of RAM I can only load small amounts into memory for targeted analysis. Processing the rest of the data requires loading it directly from storage. To improve I/O performance I parallelize the transfer across link aggregated ethernet interfaces. Naturally that would cause disk reads to become a bottleneck; to take proper advantage of the network transfer speeds I hold all data in RAID 0 with 7200 RPM drives.