If you can find or make a list of websites you want to scan, you can script it. The biggest problem is doing it in a way that doesn't bring you to the attention of those doing the blocking.

1. Where can I find a list of all domain names, top 1000, top 100000?

Alexa's "top 1,000,000" list (~10.2 MB download) is at http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

2. Is it possible to conclusively determine censorship from headers only or do I have to load the whole page and compare HTML code with a sample? Bandwidth is very expensive here.

It depends on the method used to block you from visiting a website.

If DNS-based blocking is used, you can use very small DNS lookups to identify whether or not a website is blocked — all of the hostnames of blocked websites will probably resolve to the same IP address. (You can check this with "nslookup www.website.com" in Windows or "host www.website.com" on Linux, OS X, etc.) If this method works, it's probably the best way — DNS requests are less likely to be logged than HTTP requests, and DNS requests and responses are small.

If the blocking uses a transparent proxy instead of forged DNS records, you could use HTTP HEAD requests and match against the "Server" header in the reply:

    Server: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d mod_wsgi/3.2 mod_perl/1.29 PHP/4.4.1

I think this is a HUGE issue that should not be taken lightly. A guy scanning certain websites from Iran IS going to attract some attention no matter how benign his motives. It just won't be taken lightly. That attention can land you on lists you don't want to be on.

I'm not saying that I don't sympathize with his/her situation... I just think that certain actions can be viewed by people with a security mindset as hostile. Indeed it may only increase the number of sites being blocked. As well as, SEVERELY restricting his/her ability to travel without being arrested. And if you attract enough of the right attention... you may find that being arrested is the least of your worries.

And all of this doesn't even take into account what Iranian authorities may do from their end.

Advice like this, given on a public forum via easily identifiable pseudonyms, should be taken with a BIG grain of salt.

You just don't go port scanning and probing willy nilly in the US. That's DOUBLY true if you are port scanning and probing sites that the US government has blocked... AND you are doing it from inside Iran.

You're just BEGGING for Homeland Security to take a closer look at you. It's very foolish.

You may know your Government... but I know mine. I can tell you that an Iranian probing sites whose access from Iran is blocked by the American government for security reasons... that's not bright. Authorities here will not take kindly to it.

گر حکم شود که مست گیرند

در شهر هر آنکه هست گیرند

Sorry, I couldn't help citing this particular piece of Persian poetry. Trust me, it's relevant.

I would not recommend you to do a mass censorship scan from your own IP. It's a given that one or more of the top 100,000 sites triggers some kind of flag, apart from the fact that such activity itself may mark you as a person of interest.

Another thing to consider is that the government can likely link your YCombinator account to you because there are few YCombinator users in Iran, and from that subset, only a small number (maybe 1) matches your posting timestamps. I'm of course assuming that they keep such traffic logs. Syria's surveillance system did/does.

Something similar from China, but with a different hotlist.

If Github gets blocked we should get something on, If only blocked governments took this issue seriously and had these essential services covered, but I guess something as amazing as github takes real starters and not some lame government founded dev group.

Anyway, just saying, we blocked people should hang around more often.

Alexa http://www.alexa.com/topsites could provide you with data which is for the "top 500".

Edit: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

Edit: Never mind, you already answered it in another comment.

http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

Facebook is censored in Iran, yet it is very popular. If you want to roughly estimate how many people circumvent censorship on a daily basis, just find out how many Iranians actively use Facebook. I guess it would be possible to find a number with Graph Search.

In Iran with the censorship, could the same thing be happening? Someone created it, the government sees some uses for it, an organization was tasked with enforcing it but since there are a lot of ways to circumvent it and everybody knows how, that might show this organization does not actively care enough to update it's filter. That is, it's there because someone someday had the stupid idea of creating a big firewall, but there is less and less support to actually make it real and effective?

Would I be right in assuming this much?

When internet became popular, the government monopoly got threatened. Since then they did everything in their power to restrict the use of internet. First they passed regulations that forced private ISPs to buy their bandwidth through a government organization and deployed a censorship software (rumored to be Chinese) on all of it, so blocking is national not per ISP. The they passed a law restricting home users to 128 kbps (yes, kilobits per second not even kilobytes). Then they criminalized providing anti-censorship solutions (but not using them, although it is debated). In some occasions (like after 2009 election), they make the internet so slow it is virtually impossible to use. HTTPS traffic is always slower than HTTP and occasionally completely blocked.

It is all about maintaining power to control the narrative. As I said, it hasn't worked as well as they have expected. Now they are building something called National Internet. They say they don't plan to block access to the internet but I am not so hopeful.

It has been a decade-long battle between government and freedom of information and speech. Most people who are affected aren't dissidents, but simply people who want to update their Facebook status.

It is sad because we are a fairly developed country. There is no war or famine, our healthcare system is good, we have powerful industries, good universities, big cities with good public transportation and interstate highways, etc.

The problem is that over the years since the Islamic Revolution our nation has become more liberal in general while the government remains rigidly conservative. It will be a long answer to describe where we are and how we got here, but I think this short comment is enough to answer your question.

Pretty sure most coursera vids and materials are ripped and avail via torrent too. If not could wholesale rip the site and mirror it free on yandex cloud. Russia laughs at US petty sanctions

I, personally, was always very skeptical about the VPN services sold in the wild. Who knew what machine you were routing your data through was not controlled by the government themselves?

Why?

I don't think it is probable that Oracle, Xilinx or Google change their policies. Yet it is in the developer's power to decide where to host a project.

I like to ask everyone who has a project hosted on Google Code or is thinking about using it for future projects to use another service if possible.

That is, unless you're Cuban. In that case you can go to the US and they'll welcome you to Miami with a red carpet and fool you with what freedom and cars and big houses and laptops you'll be able to have.

Google in particular has been one of the hardliners when it comes to restricting access to users with Iranian IPs. There are other big US companies (like Microsoft, for instance) that are much less active in banning IPs.

Hell, I have trouble paying for stuff in the EU with a US card...

Anyway, VPN providers are NOT your friend for privacy and escaping surveillance/censorship. They actually cooperate with the authorities!

Hidemyass and Astrill used to openly state they will provide any and all details of your identity and activities if the authorities request it!

There's no need for honeypots when all these companies give up your identity with a single call from the police.

They also discriminate against automated tools and robots for whatever reason, as if paying a human to do the work is better :-)