Re:How to scrape anything on the web and not get caught
by pdkl95 on Sunday April 22, 2018 @07:29AM
(#16895712)
I assume simple BFS and DFS traversal behavior shows up brightly in access logs, making detection more likely. Does it help to use Random First Search[1]? Or is it better to attempt emulating human actors (requires much more develop effort)?
[1] https://bl.ocks.org/mbostock/11161648