News for hackers. Stuff that matters.

Ask HN: How do I compile a list of censored websites in my country

Hello

I live in Pakistan and my government censors websites that contain "immoral" content. Sometimes random websites get caught in the censors, e.g pastebin.com

I wanted to build a portal that kept check of which websites are blocked, but I'm unsure about how to go about this:

1. I could download the zonefiles for all the popular tlds and test them, but this will exclude unpopular tlds

2. I could crawl the web but this will turn into a more expensive side project than I have the funds for.

3... any suggestions?

( 8 comments | 10 points | on HN )
Re:Ask HN: How do I compile a list of censored websites in m... by loopbacker on Sunday January 15, 2017 @09:03AM (#13403983)
Most dns servers don't allow zone transfers, so I don't believe you'll be able to enumerate domains like this.

In general enumerating domains is not really possible (you can't just dump all North Korean domains for example, and people have tried).

I'd start by figuring out how they are blocking sites. Probing to see which IPs are blocked would be much easier. There are various tools to lookup which sites are hosted on a given IP address (using crawled data).

One way to do this would be to try connecting from outside Pakistan (e.g. EC2 server) and from inside and seeing if a IP is blocked.

Scanning all 4 billion IPs surprisingly doesn't take too long on a modern internet connection.

Re:Ask HN: How do I compile a list of censored websites in m... by Raed667 on Sunday January 15, 2017 @09:26AM (#13404088)
My two cents: Don't do it on your own. Contact NGOs that work on the censorship issues, EFF, AccessNow, FLD, etc.. They have been working on these issue for a while and probably can help you with their resources.
Re:Ask HN: How do I compile a list of censored websites in m... by amingilani on Sunday January 15, 2017 @06:27PM (#13407099)
I might after I build the site :)
Re:Ask HN: How do I compile a list of censored websites in m... by jjoe on Sunday January 15, 2017 @01:14PM (#13405399)
I think you should crowd source it. Build a simple two-page site where people can submit censored websites and another where you display them. But your very own website could end up on the list.
Re:Ask HN: How do I compile a list of censored websites in m... by amingilani on Sunday January 15, 2017 @06:27PM (#13407097)
That is a wonderful idea!
Re:Ask HN: How do I compile a list of censored websites in m... by AdamTheAnlayst on Sunday January 15, 2017 @02:10PM (#13405728)
I own https://www.DomainDetect.io, we pull new domains from Zone files/3rd party sources daily. ~200,000-400,000 domains per day to alert on new phishing sites for businesses. So we do something similar. I would start with something like https://czds.icann.org/en attempting to resolve them asyncronously in your country, then work your way through the new daily additions. Use an ElasticSearch/Cassandra backend for speed to query that volume of information.

Your problem will come from the countries/TLD's that dont play ball with their zone files. These are trickier to obtain.

Re:Ask HN: How do I compile a list of censored websites in m... by miguelrochefort on Sunday January 15, 2017 @02:49PM (#13405982)
Yours will soon be on that list.
Re:Ask HN: How do I compile a list of censored websites in m... by amingilani on Sunday January 15, 2017 @06:26PM (#13407093)
I'd like to see that happen. I don't take kindly to being censored arbitrarily and will fight tooth and nail to the full extent of the law :)