Show HN: I have written a cloud native dark web crawler in Go

Posted by creekorful on Wednesday January 06, 2021 @12:33PM
from the show-hn dept.

( 7 comments | 7 points | on HN )

Re:Show HN: I have written a cloud native dark web crawler i... by lrossi on Wednesday January 06, 2021 @12:37PM (#25660044)

Can you share some examples of what it finds? As long as it is safe to post that.

[ Parent | on HN ]

Re:Show HN: I have written a cloud native dark web crawler i... by creekorful on Wednesday January 06, 2021 @01:09PM (#25660403)

I have an instance running for about 5 days+ with 1,300,000 docs. It's mostly huge amount of porns, some weird forums & lot of marketplaces.

There's still some good finds like a 'sysadmin' forum with huge amount of good posts & tips.

[ Parent | on HN ]

Re:Show HN: I have written a cloud native dark web crawler i... by creekorful on Wednesday January 06, 2021 @01:10PM (#25660421)

I'll certainly publish an index of interesting (legal) hostnames on my GitHub account.

[ Parent | on HN ]

Re:Show HN: I have written a cloud native dark web crawler i... by creekorful on Wednesday January 06, 2021 @12:33PM (#25659986)

Release 1.0.0 will be out very soon. Check it out guys!

[ Parent | on HN ]

Re:Show HN: I have written a cloud native dark web crawler i... by visox on Wednesday January 06, 2021 @03:25PM (#25661917)

nice, what is actually needed to crawl onion sites ?

i am not a go dev, would try it with scala/java

[ Parent | on HN ]

Re:Show HN: I have written a cloud native dark web crawler i... by creekorful on Wednesday January 06, 2021 @05:49PM (#25663779)

Just a tor proxy (socks) instance, and a HTTP client configured to use the proxy. Other than that nothing more than what a web crawler should have.

[ Parent | on HN ]

Re:Show HN: I have written a cloud native dark web crawler i... by visox on Thursday January 07, 2021 @03:31AM (#25668662)

cool i look into that some time.

[ Parent | on HN ]