Show HN: I have written a cloud native dark web crawler in Go

Re:Show HN: I have written a cloud native dark web crawler i... by visox on Wednesday January 06, 2021 @03:25PM (#25661917)

nice, what is actually needed to crawl onion sites ?

i am not a go dev, would try it with scala/java

[ Parent | on HN ]

Re:Show HN: I have written a cloud native dark web crawler i... by creekorful on Wednesday January 06, 2021 @05:49PM (#25663779)

Just a tor proxy (socks) instance, and a HTTP client configured to use the proxy. Other than that nothing more than what a web crawler should have.

[ Parent | on HN ]

Re:Show HN: I have written a cloud native dark web crawler i... by visox on Thursday January 07, 2021 @03:31AM (#25668662)

cool i look into that some time.

[ Parent | on HN ]