OONI Explorer: Censorship and other Network Anomalies Around the World

by agrabeli | March 23, 2016

Today the Open Observatory of Network Interference (OONI) team is pleased to announce the public beta release of OONI Explorer: a global map of more than 8.5 million network measurements which have been collected across 91 countries around the world over the last 3 years.

OONI is based on 15 free software tests which are designed to measure the following:

  • Blocking of websites
  • Detection of systems responsible for censorship, surveillance and manipulation
  • Reachability of Tor, proxies, VPNs, and sensitive domains

These tests have been run across 398 different vantage points by volunteers around the world since 2012. The OONI Explorer announced today provides a location to interact and - dare we say - explore all of the collected measurements.

Key Findings

Some of the highlights in the data:

1. Confirmed cases of censorship in 9 countries

Multiple HTTP request tests were run around the world and based on our heuristics, we were able to detect block pages in 9 countries: Iran, Saudi Arabia, Turkey, Greece, China, Russia, India, Indonesia and Sudan.

Blocked websites include media, gambling and over-the-counter money exchanges. In Greece, for example, all of the tested ISPs employed DNS hijacking to block such websites, with the exception of Vodafone that also used Deep Packet Inspection. OONI tests in Turkey illustrate that 62 websites were blocked, including piratebay.com, livescore.com and 4shared.com, possibly under Law No. 5651 on the ‘Regulation of Publications on the Internet and Suppression of Crimes Committed by means of Such Publication’. Notably, 362 blocked websites were detected as blocked in Iran and 50 in Saudi Arabia, including arabtimes.com, mossad.gov.il and anonym.to, a URL shortening service with privacy properties.

Some of our tests for domains were focused on specific websites which were rumored or reported to be blocked. In January 2015, for example, the Government of India ordered the blocking of 32 websites under Section 69A of the Information Technology Act, 2000, and under the Information Technology (Procedures and Safeguards for Blocking of Access of Information by Public) Rules, 2009. Following these reports, OONI tests run on those websites were able to confirm that 23 of those websites were in fact blocked in the network that was tested, including websites such as pastebin.com, dailymotion.com and archive.org.

Leading up to the 2016 general elections in Uganda, OONI volunteers ran HTTP request tests in response to reports that Facebook and Twitter were being blocked. We did not detect block pages, but we did detect general network anomalies which indicate that it's likely the case that Ugandan ISPs were blocking some requests, but not others. It is also possible that Facebook and Twitter were only blocked in specific networks, and not countrywide.

2. Network anomalies in 71 countries

Out of the 91 countries with reported data, network anomalies were detected in 71 of them.

“Network anomalies” and “network interferences” are broad terms that we use to describe symptoms of censorship through the manipulation of internet traffic. These anomalies can take many forms, including connectivity failures, timeouts and unusual slowness, or unexpected error messages.

Not all HTTP request tests allow us to conclusively know that interference has occurred, because not all interference looks like a clear block page. Sometimes, censorship is hidden as connection failures instead. To gain confidence in detecting this type of interference, we can look at repeated failures to websites that are known to be operating normally. In Cuba, for example, it is interesting to see that while no block pages were detected, HTTP requests to cubafreepress.org failed multiple times.

Symptoms of traffic manipulation were detected in multiple countries around the world through HTTP invalid request line and HTTP header field manipulation tests, which look for middle boxes: network equipment that intercept and sometimes alter the traffic passing through them. Multiple HTTP invalid request line tests run in Vietnam from 2013 to 2015 triggered errors and indicate that middle boxes were regularly observing the traffic in the country. Similarly, many HTTP invalid request line tests in Pakistan and elsewhere indicate the presence of software which is capable of traffic manipulation.

3. Blue Coat, Squid and Privoxy detected in 11 countries

Transparent HTTP proxies can be used inside of small and large networks for various purposes: to intercept the web traffic of users, to implement caching or to speed up requests for commonly visited websites.

Through OONI tests we detected 3 different types of proxy technology: Blue Coat, Squid and Privoxy. Blue Coat Systems is a US security and networking solutions provider which has been called out for selling network appliances capable of filtering, censorship, and surveillance to governments with poor human rights records. Its presence, along with Squid and Privoxy, has been reported in the networks of 11 countries: USA, Canada, Portugal, Spain, Italy, the Netherlands, Switzerland, Moldova, Iraq, Myanmar and Uganda. It remains unclear though whether such middle boxes were actually used for online censorship, surveillance and traffic manipulation, or if they were merely used for caching purposes.

Furthermore, not all the detected instances of proxy technologies are necessarily deployed country-wide or even on an ISP level, but in some cases they might simply be running inside of the local network of the OONI user. It is interesting to note that the use of Blue Coat was first detected in Myanmar in 2012, but when another measurement was run from the same network in 2014 it was no longer detectable in the same way. This can either mean that it was removed or that it is no longer detectable.

Contribute to OONI Explorer

OONI Explorer was made possible by the growing community of volunteers around the world who have contributed to the project. You can contribute too by:

Happy OONI exploring!

Comments

Please note that the comment area below has been archived.

March 23, 2016

Permalink

poor human rights=USA, Canada, Portugal, Spain, Italy, the Netherlands, Switzerland, Moldova, Iraq, Myanmar and Uganda.
strange test , worst are not here !

Hi, it's not necessary that the presence of proxy technology is indicative of human rights abuses. There are many legitimate reasons for using proxy technology, such as caching or load balancing.

Through the data we collected we able to show that certain proxy technologies are being used in specific networks of specific countries.

However we are unable to show how or why these technologies are being used as answers to such questions go beyond the types of data that OONI is currently designed to collect.

March 24, 2016

Permalink

>detect block pages in 9 countries
Really only 9? I know that The Pirate Bay and other pirate websites are blocked in most of countries under the rule ow copyright. Don't you think that blocking of these websites is also censorship?

Glad you brought this up. You're right - it's likely the case that censorship is not only occurring in 9 countries.

If you manually look at the measurements you can evaluate whether censorship is happening or not in other countries as well. Our current heuristic is more conservative, in the sense that it only takes into account when censorship is happening due to a block page. So far, we have only been able to detect block pages in 9 countries, but will be expanding the heuristics to take into account also other forms of blocking and more block-pages as we get more measurements.

March 24, 2016

In reply to art

Permalink

You are only testing for a list of websites that might be blocked in countries like China, Iran, Saudi Arabia, Russia pornography, gay rights, sex education, human rights etc. but I am sure that western countries block websites that are not on the list most are probably blocked for legitimate reasons child pornography but in the minds of Muslim fundamentalists and Russian orthodox blocking regular pornography, blasphemy, gay rights etc is also legitimate.

March 24, 2016

Permalink

Recently I use TOR and it's suck
TorBundle has error.
Everytime I create blog in wordpress.com the result annoying
Wordpress site said this:
An Active token must be used create a new site for user
While on Wix, everytime I want to create blog or whatever it just do nothing

Now I have no problem when I use
Blackbelt privacy or even I use torrified system over Backbox. I have no problem at all only TorBundle.

If you are using the TOR browser, try going into
add-ons/ Tor Button Manager /Prefrences

Uncheck the box: Restrict third party cookies and other tracking data.

Reload Wordpress

March 25, 2016

Permalink

ITALIANO
Una domanda che vorrei inserire è
un captcha può servire a chiunque per tracciare l'identità e quindi i movimenti di un utente anche su rete tor?
-----------
english
One question that I would insert it
a captcha can serve to anyone to trace the identity and therefore the movement of a user even on the network tor?

a captcha is a secure tool identifying a user as a legitimate user but it is a misuse and a nasty behavior when it is is blocking an open website or a part of the net.
do not fill up the captcha , you are trolling from the bad guys provocation.

March 25, 2016

Permalink

It seems not possible to display OONI Explorer's measurements using Tor Browser...

March 25, 2016

Permalink

Why do you only consider the blocking of certain sites as censorship and not others? What about revenge-porn which is now being blocked by every major government? and hate-speech? And child porn? Its a clear violation of free speech that people arent allowed to access media of kids performing sex acts but they can access media of violence against children, and stupid given their argument against the former is the latter.

As far as im concerned all internet is censored which makes your observations moot.

March 25, 2016

Permalink

Can the Tor Project not do something to get the evil Clownfare off our backs? Even if it meant Clownfare going out of business - so be it! :)

March 25, 2016

Permalink

You do such good work!

I'd be interested in learning which countries make it a crime to even make these sorts of measurements for the purpose of enumerating state censorship.

I often get things like reset connections when browsing sensitive things without a proxy (in Germany). Other forms of blocking include mandated self-censorship (Youtube in Germany, and previously Demonoid), and free "anti-virus" products that block Torproject (also Germany). I run into blocked content just about every day and I am surprised that OONI hasn't detected it yet.

March 26, 2016

Permalink

For the italian test. The ASN where squid has been detect is not present. Could you fix it?

Thx

March 29, 2016

Permalink

TOR!

March 30, 2016

Permalink

You can get list of blocked Russian websites on antizapret.info with information why they were blocked and by which agency.
ФСКН usually blocks sites about psychoactive substances, роспотребнадзор blocks sites about suicide and child porn (including rendered one), МВД blocks sites about extremism.

Court (суд) usually blocks websites about activities either prohibited or discouraged, such as prostitutes, gambling (it is prohibited in russia if not in special areas), avoiding conscription, tampering with electricity and water counters, medicines e-shops, anonymity and anti-censorship means (it's illegal, but they block), cryptocurrency, piracy, etc.
I don't know the law allowing such blocks (except piracy). If the site is blocked by court order, you can see the identifier of lawsuit.

BTW, antizapret has API, you can get the list of prohibited websites and check them with your tool.

April 04, 2016

Permalink

At least two crucial points which were not explained in the post:

o US, UK, DE, IT companies sell censorship and dragnet surveillance technology to the worst governments in the world (Blue Coat just one example, and not the worst)

o not just intentional censorship but also human prejudices and misconceptions built into the algorithms which make most of the decisions (censor or not):

https://www.washingtonpost.com/news/the-intersect/wp/2016/03/29/how-wer…
How we’re unwittingly letting robots censor the Web
Caitlin Dewey
29 Mar 2016

For example, consider how BBC treats all Tor users as "foreign".

April 07, 2016

Permalink

Speaking of censorship, let's all do what we can to help CN and RU readers read all about the Panama Papers!

http://www.theguardian.com/world/2016/apr/07/china-ramps-up-panama-pape… China ramps up Panama Papers censorship after leaders' relatives named
Websites ordered to purge all reports related to documents following publication of political elite’s offshore secrets
Tom Phillips
7 Apr 2016

> Chinese censors have stepped up their censorship of websites, ordering all content related to the Panama Papers to be scrubbed as new revelations emerged of how relatives of some of the country’s top leaders had used secretive offshore companies to store their wealth. Documents from the leaked Mossack Fonseca database showed the relations of three of the seven members of the Communist party’s elite ruling council, the politburo standing committee, had companies that were clients of the offshore law firm. They included relatives of Chinese president Xi Jinping. A Communist party censorship directive instructed news organisations to purge all reports, blogs, bulletin boards and comments relating to this week’s highly sensitive revelations.

What are President Xi and his cronies frightened of?

This, perhaps?

https://en.wikipedia.org/wiki/Tiananmen_Square

One of the most significant implications of the CN ruling elite moving so much money out of CN is that they foresee the collapse of the Communist Party. I certainly hope so, because the Communist Party--- and the Democratic and Republican parties in the US--- are explicitly refusing to seriously address the rapidly approaching disasters of climate change, the pervasive replacement of human employees by robotic workers, inadequate government response to environmental disasters, global financial instability, and more.

But if all that is too scary to contemplate, here's a bit of juicy scandal (murder, money, luxury beyond imagining--- it's the Chinese "Dallas"!):

http://www.theguardian.com/news/2016/apr/06/panama-papers-french-villa-…
he Panama Papers and the French villa at the heart of a Chinese scandal
Panama Papers detail the efforts made to conceal the true owner of the property – the wife of disgraced politician Bo Xilai
Juliette Garside and David Pegg
7 Apr 2016

> A villa linked to a British businessman’s murder in an extraordinary Chinese political scandal secretly changed hands just two weeks after his death. The luxurious €7m (£6m) property was at the heart of the tussle that led to Neil Heywood’s killing and triggered the downfall of the flamboyant politician Bo Xilai, jailed for life for corruption and power abuses. Bo’s wife, Gu Kailai, is serving a life sentence for poisoning Heywood in November 2011.

Then there's a world leader who is so paranoid that he can't move his arms when he walks:

http://www.theguardian.com/news/2016/apr/07/putin-dismisses-panama-pape…
Putin dismisses Panama Papers as an attempt to destabilise Russia
Russian president notes his name does not feature in files, which he says offer no proof of corruption
Alec Luhn in Moscow and Luke Harding in London
7 Apr 2016

> Vladimir Putin has offered his first response to the revelations in the Panama Papers, describing their publication as an attempt to destabilise Russia and saying there was no proof of corruption in the dossiers.

This somewhat dubious contention--- it's not like people in the UK aren't talking about the shenanigans of the Cameron/Astor plutocrats--- has won support among some US people who support Donald Trump. Putin neglected to point out that a considerable portion of the scandal surrounding Prime Minister Cameron's very wealthy family's tax shelters involves shell games in the Bahamas. And the NSA has for years recorded every phone call made in the Bahamas. So they knew, and they certainly didn't tell the New York Times--- a disgraced paper whose editors are hopping mad because they were not included among the 100 papers around the world which have been given access to the Panama Papers.

It's not just the RU and CH financial political elite who are ducking for cover behind censorship. The US political leadership is also very very frightened. What are they scared of? When we obtain a similar leak from the big US law firms which set up DE and NV companies, the same explosive implication will emerge: the US elite foresees the collapse of the Democratic and Republican parties which have protected their tax shelters for so long, as the political fortunes of Sanders and Warren continue to rise:

https://www.salon.com/2016/04/06/lessons_of_the_panama_papers_yes_the_r…
Lessons of the Panama Papers: Yes, the rich are different from us — they stole our money
A vast web of lies and corruption protecting the super-rich is revealed -- and it's only the tip of an evil iceberg
Andrew O'Hehir
6 Apr 2016

> ...
> What have we learned so far from the Panama Papers, the largest volume of leaked documents in history, which have begun to peel the lid off a vast web of global greed, deception and iniquity among the highest level of the moneyed classes? For starters, they should serve to remind us how different the very rich are from the rest of us. Yes, it starts with the fact that they have more money, but it doesn’t end there. How did they get all that money, and what are they doing with it? Why do they have so much more money than the rest of us — unimaginably more, and on an unprecedented scale? Why do they seem so perpetually unsatisfied with their wealth, and so desperate to nurture it, shield it and multiply it?

Because they are selfish?

David Cameron is a perfect example: he enjoys running the government, he just doesn't want to help pay for it.

It seems to be an enduring truism that poor people are more generous than rich people. Rich people can avoid taxes which poor people can't duck but that's not what I mean. Poor people also give more voluntarily, per capita. Some speculate about a law of Nature holding that the more you have, the more frightened you are of losing it all. Others say that the simplest explanation is that super-rich people are nasty because you can't get super-rich without being nasty to others. Possibly there is some truth in all of these conjectures.

But the US elite is worried about more than the collapse of the two party system. They foresee the collapse of the global economy and violent political upheaval inside the US, and they are preparing to flee. But they have the same problem as privacy advocates: there's really nowhere to flee too.

One of the really big stories which the US press is completely ignoring (so is the US political elite) is a rapidly approaching "phase transition" in the human condition, in which the default condition for adults changes from "employed" to "unemployable" (and if the rich are avoiding paying taxes, how will government obtain revenue to provide even the most essential services, like food banks for the starving?):

http://www.theguardian.com/commentisfree/2016/apr/07/robots-replacing-j…
Are robots going to steal your job? Probably
People ridiculed the Luddites for opposing technological change that ultimately created new work. Today’s economic indicators don’t offer that hope
Moshe Y Vardi
7 Apr 2016

> If you put water on the stove and heat it up, it will at first just get hotter and hotter. You may then conclude that heating water results only in hotter water. But at some point everything changes – the water starts to boil, turning from hot liquid into steam. Physicists call this a “phase transition”. Automation, driven by technological progress, has been increasing inexorably for the past several decades. Two schools of economic thinking have for many years been engaged in a debate about the potential effects of automation on jobs, employment and human activity: will new technology spawn mass unemployment, as the robots take jobs away from humans? Or will the jobs robots take over release or unveil – or even create – demand for new human jobs?

Interestingly enough, the CN government *does* mention this issue, but takes the irresponsible position that it is entirely up to Chinese workers to figure out how to deal with the loss, not just of their job, but of their lifetime employability.

It's all of them against all of us.