A few days ago, we published a blog post exposing the use of Deep Packet Inspection (DPI) to filter all Internet traffic in Ethiopia, including connections to the Tor network. We concluded that they are doing some sort of TLS fingerprinting, but had not been able to figure out exactly what they are fingerprinting on. Since then, we have managed to determine exactly how Ethiopia blocks Tor and we have developed a workaround. We will publish a full technical analysis very soon.
The long-term solution for Tor users in Ethiopia is to use the Obfsproxy Tor Browser Bundle. The bundles are, unfortunately, not up to date at the moment, but this is something we are working on (see #5937 for details). In the meantime, try using one of the following three bridges:
If the bridges are not working, or you have questions, send an email to firstname.lastname@example.org.
The Ethiopian Telecommunication Corporation, which happens to be the sole telecommunication service provider in Ethiopia, has deployed or begun testing Deep Packet Inspection (DPI) of all Internet traffic. We have previously analyzed the same kind of censorship in China, Iran, and Kazakhstan.
Reports show that Tor stopped working a week ago -- even with bridges configured. Websites such as https://gmail.com/, https://facebook.com/, https://twitter.com/, and even https://torproject.org/ continue to work. The graphs below show the effects of this deployment of censorship based on Deep Packet Inspection:
An analysis of data collected by a volunteer shows that they are doing some sort of TLS fingerprinting. The TLS server hello, which is sent by the Tor bridge after the TLS client hello, never reaches the client. We don't know exactly what they are fingerprinting on, but our guess is that it is either the client hello or the server hello. An illustration can be found in this network flow diagram.
Thanks to Philipp Winter and George Kadianakis for helping me analyze the data. If you have more information about the censorship in Ethiopia, please email email@example.com.
We're pleased to announce that the Tor Project and Tails are hosting students this year as part of Google Summer of Code. Out of the 26 applications to us we were able to take on six fantastic students:
Ravi Padmala - Introduction
Project: Stem Improvements and Arm port
Mentor: Damian / Sathyanarayanan
Feroze Naina - Introduction
Project: Implementing Hidden Service Configuration
Mentor: Tomás / Sebastian
Michele Orrù - Introduction
Project: Anonymous Python Application Framework
Mentor: Arturo / George
Julien Voisin - Introduction
Project: Tails Server
Mentor: intrigeri, anonym
Project: Pluggable Transports in Python
Mentor: George / Nick
Mentor: Zack Weinberg / Steven / Roger
Projects officially begin on May 21st. We're thrilled to have them with us, and have our fingers crossed that they'll stay afterward to become core developers.
Many thanks to Google for having the program again this year! -Damian
In December 2011 we were aware of Kazakhstan increasing Internet censorship in response to some unrest and protests in Zhanaozen in the west. The censorship was then deployed around the country, in many cases with the full support of the populace. The initial invesitgation showed simple IP address blocking coupled with basic dns censorship. Tor continued to work without incident until this week.
JSC KazTransCom, AS35104, has deployed or begun testing deep packet inspection (dpi) of all Internet traffic. They specifically target SSL-based protocols for blocking. This includes Tor, IPsec, and PPTP-based technologies, as well as some SSL-based VPNs. Business and private users of these technologies are equally affected.
An example of the censorship, as recorded by volunteers in country, can be found in this network flow diagram. Kazakhstan is identifying and blocking the SSL client key exchange during the setup of an SSL connection. This graph shows the effects of this deployment of censorship based on dpi.
Luckily, due to our recent experience with Iran we have an answer for people: use obfsproxy. Obfsproxy continues to work in Kazakhstan, as well as Iran. In fact, it works in any country where dpi is used to censor citizens' access to the Internet.
Thank you to the volunteers for spending their Valentine's Day collecting and analyzing data.
On Feb 9, Iran started to filter SSL connections on much of their network. Since the Tor protocol uses SSL, that means Tor stopped working too — even Tor with bridges, since bridges use SSL too.
We've been quietly developing Obfsproxy, a new tool to make it easier to change how Tor traffic looks on the network. In late 2011 Iran moved into the #2 position in global Tor user count, and several important political events are scheduled in Iran this month and next. This situation seemed like a good time to test our new tool while also helping improve Internet freedom around the world.
We started with a "Tor Obfsproxy Browser Bundle" with two test obfsproxy bridges in it, to verify that it worked in-country. Then we got over 300 volunteers running more obfsproxy bridges (even with our complex build instructions!), and picked fourteen fast stable trustworthy obfsproxy bridges for an updated bundle which we put out the morning of Feb 11. We spent the weekend fixing usability, stability, and scalability bugs, and put out another bundle on Feb 13 with new versions of Vidalia, Tor, and Obfsproxy.
Thousands of people in Iran successfully used the Obfsproxy Bundle over the weekend:
We did some spot-checking and it seems that the new addresses on Feb 14 are mostly different from the new addresses on Feb 13; but I would guess these are mostly returning users with dynamic IP addresses, rather than actually fresh users. More importantly, these people will be thinking about Obfsproxy next time the filter cracks down — and based on current events, that next time won't be far off. Finally, even though it looks like SSL and Tor are back, I expect Iran will keep throttling SSL traffic as they've been doing for months, so the Obfsproxy bundle will still be more fun to use in Iran than the normal Tor bundles.
How does it work?
Deep Packet Inspection (DPI) algorithms classify Internet traffic by protocol. That is, they look at a given traffic flow and decide whether it's http, ssl, bittorrent, vpn, etc. Governments like Iran, China, and Syria increasingly use these tools (and they often purchase them from Western corporations, but that's a different story) to implement their country-wide censorship, either by looking for a given protocol and outright blocking it, or by more subtle mechanisms like squeezing down the bandwidth available to a given protocol to discourage its use.
Obfsproxy's role is to make it easy for Tor's traffic flows to look like whatever we like. This way Tor can focus on security and anonymity, and Obfsproxy can focus on appearance. The first thing we decided to try looking like was nothing at all: the "obfs2" module adds an encryption wrapper around Tor's traffic, using a handshake that has no recognizable byte patterns.
It won't work perfectly. For example, the traffic flows will still have recognizable timing, volume, and packet size characteristics; a good entropy test would show that the handshake looks way more random than most handshakes; and the censors could always press the "only allow protocols my DPI box recognizes" panic button. Each step in this arms race aims to force the censor to a) put more development time and DPI resources into examining flows, and b) risk more false positives, that is, risk blocking innocent users that they didn't realize they'd be blocking.
This particular new obfuscating layer isn't the most important feature of Obfsproxy. The best part is that makes it easy to design, deploy, and test other obfuscating layers without messing with the rest of Tor. There's a lot of research into trying to make traffic flows look like other protocols, so for example we could rewrite the Tor flows as valid http that the DPI engine considers harmless. That problem is harder than it sounds — and it sounds hard. But by making a separate component that only has to worry about how the traffic looks, other researchers can try out different strategies without needing to learn so much about the rest of Tor. This approach will also let us easily plug in other transports like Telex, and it will also let other circumvention projects reuse Obfsproxy so they don't have to reinvent our wheels.
One of the choices we faced was how widely and loudly to mention the bundle. While we think it would be hard and/or risky for attackers to block the Obfsproxy protocol, the bundle included 14 preconfigured bridge addresses, and censors could just plug those addresses into their blacklists. We started the weekend telling people to only tell their close friends, but on Sunday we opted for a broader publicity push inside the activist community for two reasons. First, the new Vidalia release (0.2.17) lets users configure their own obfsproxy bridge addresses, so if the preconfigured addresses get blocked the user can just put in new ones. Second, it became clearer that the blocking would let up in a few days once the immediate political pressure was over, and we decided it was more important to get the word out about Obfsproxy in general so these users will know about it next time.
I should point out that I don't think they were targeting Tor here. They were targeting popular websites that use SSL, like Gmail and Facebook. Tor was collateral damage because we opted to make Tor traffic look like SSL. That said, we should not forget that we are on their radar: they targeted Tor by DPI in September 2011, and the Diginotar breakin obtained a fake SSL cert for torproject.org.
The next choice we face is: what other communities should we tell? The bundle works great in China too, where their aggressive censorship has been a real hassle for us the past year or so. Some other countries in Asia appear to be newly using DPI to recognize Tor traffic (more on that in an upcoming blog post). We have more development work to do before we can keep up with the China arms race, including teaching obfsproxy bridges to automatically report their addresses to us and teaching our bridgedb service to give them out, and we need to answer research questions around getting more bridges, giving them out in smarter ways, learning when they get blocked, and making it hard for censors to find them. We also need to spread the word carefully, since the arms race is as much about not drawing the attention of the censors as it is about the technology. But the Obfsproxy Bundle works out of the box right now in every censoring country we know of, so we shouldn't be too quiet about it.
And finally, thanks go to George Kadianakis for taking Obfsproxy on as his Google Summer of Code 2011 Project; to Nick Mathewson for mentoring him and getting the Obfsproxy architecture going in the right direction; to Sebastian Hahn for spending all weekend with me fixing bugs and making packages; and to Karsten Loesing, Erinn Clark, Robert Ransom, Runa Sandvik, Nick, George, and the broader Tor community for stepping up over the weekend to help us take it from "early prototype" to "deployed software in use by 5000+ people" in 72 hours.
Greetings! My name is Tim Wilde, Software Engineer at Team Cymru, Inc., and a big fan of the Tor Project and everything that they do. We've helped out the Tor Project with a few investigations into probing/blocking of Tor by oppressive regimes, and the guys asked me to write this one up for the Tor Blog, so here I am! Note: any opinions expressed here are mine, nor those of Team Cymru or the Tor Project.
In October 2011, ticket #4185 was filed in the Tor bug tracker by a user in China who found that their connections to US-based Tor bridge relays were being regularly cut off after a very short period of time. At the time we performed some basic experimentation and discovered that Chinese IPs (presumably at the behest of the Great Firewall of China, or GFW) would reach out to the US-based bridge and connect to it shortly after the Tor user in China connected, and, if successful, shortly thereafter the connection would be blocked by the GFW. There wasn't time for a detailed investigation and analysis at the time, but that kernel eventually grew into the investigation detailed below. We were, however, able to determine that limiting connections to the bridge relay to only the single IP expected to be its client would, in fact, block the probes and allow the connection to remain open for an extended period (>48 hours in our testing).
Between 05 DEC and 09 DEC 2011, we undertook a detailed and methodical investigation into probing and blocking of Tor connections originating within China. Unfortunately for our analysis, it appears that the GFW's active blocking of connections to Tor bridges had stopped, but we were still able to gather valuable data about the probing performed by the GFW, which previously led directly and verifiably to blocking.
To this end we discovered two types of probing. First, "garbage binary" probes, containing nothing more than arbitrary (but sometimes repeated in later probes) binary data, were experienced by the non-China side of any connection that originated from China to TCP port 443 (HTTPS) in which an SSL negotiation was performed. This probe was performed in near-real-time after the connection was established, implying near-line-rate deep packet inspection (DPI) capabilities. TCP/443 connections not performing an SSL handshake, such as using the obfsproxy obfs2 protocol or a plain-text protocol, did not provoke probing. The purpose of these probes is unknown, and further investigation is difficult to justify when it seems relatively clear that these probes are not aimed at Tor.
The second type of probe, on the other hand, is aimed quite directly at Tor. When a Tor client within China connected to a US-based bridge relay, we consistently found that at the next round 15 minute interval (HH:00, HH:15, HH:30, HH:45), the bridge relay would receive a probe from hosts within China that not only established a TCP connection, but performed an SSL negotiation, an SSL renegotiation, and then spoke the Tor protocol sufficiently to build a one-hop circuit and send a BEGIN_DIR cell. No matter what TCP port the bridge was listening on, once a Tor client from China connected, within 3 minutes of the next 15 minute interval we saw a series of probes including at least one connection speaking the Tor protocol.
The good news is, we were able to isolate which characteristic of the Tor handshake the GFW was using to decide whether or not to initiate these probes. By making a simple change to the list of supported SSL ciphers in the "hello" packet sent by the Tor client within China, we were able to prevent the probes from taking place. This has been documented and is being discussed in ticket #4744. This differs slightly from the method used by Iran to block Tor in September 2011, using the client-side of the SSL negotiation as its trigger rather than the server-side. It is likely, however, that technology capable of targeting either side of the connection to this degree could also target the other side, so it remains important to consider both the server and client sides of the Tor connection when attempting to blend in with normal, benign traffic.
The bad news, however, is that this is happening at all. This probe again implies sophisticated near-line-rate DPI technology, coupled with a system that is aimed directly at Tor, using code that actually speaks the Tor protocol. Clearly there is a target painted firmly on Tor, and it is quite likely that the Chinese will continue to adapt their censorship technology as the Tor Project adapts to them.
There is light at the end of the tunnel, though. A number of ideas have been put forward about new protocols, handshakes, and extensions to the Tor protocol that could be used to combat this type of censorship technology in a more long-term-sustainable way. Proposal 190 provides for password authorization for bridge relays. obfsproxy provides an extra layer on top of a Tor connection that makes it look like generic binary data. The Tor v3 link protocol/handshake, currently available and in testing in the 0.2.3.x-alpha series of Tor releases, eliminates SSL renegotiation in Tor session establishment, removing one of Tor's current "sore thumbs" that stick out from normal HTTPS SSL traffic.
You're welcome to check out my full report on this investigation for more detail than I could provide in the blog post here. Thanks very much to everyone who assisted with the investigation and the report! It was fun to investigate and report on this, and I hope to have the opportunity to help out with similar adventures in the future.
The July 2011 Progress Report is at the bottom of this post and at https://blog.torproject.org/files/2011-July-Monthly-Report.pdf.
Highlights include continued progress on protocol obfuscating proxy, a new bridge guard design, outreach, scalability improvements, orbot updates, and a number of translation updates.
Update 2011-08-15: based on feedback, created a plaintext version of the pdf. It doesn't contain the images obviously, but does contain all of the content. Generated the text file via pandoc. The text file is here, https://blog.torproject.org/files/2011-July-Monthly-Report.txt
The April 2011 Progress Report is attached to this post and available at https://blog.torproject.org/files/2011-April-Monthly-Report.pdf.
Highlights include releases for tor, vidalia, arm, and libevent. Updates to pluggable transports, obfsproxy, torbutton, many translation and core architecture updates.
We're pleased to announce that the Tor Project, Tails, and Guardian are hosting students this year as part of Google Summer of Code. Out of the 31 applications to us we were able to take on six fantastic students:
Julien Voisin - Introduction
Project: Metadata Anonymisation Toolkit
Mentor: Mike Perry
George Kadianakis - Introduction
Project: Pluggable transports
Mentor: Nick Mathewson
Sathya Gunasekaran - Introduction
Project: Orbot + ORLib
Mentor: Nathan Freitas
Kamran Khan - Introduction
Project: GTK+ Frontend and Client Mode Improvements for arm
Mentor: Damian Johnson
Max - Introduction
Project: Custom GDM3 startup menu, aka. tails-greeter
Project: Blocking-resistant Transport Evaluation Framework
Mentor: Thomas Benjamin
Projects officially begin on May 23rd. We're thrilled to have them with us, and have our fingers crossed that they'll stay afterward to become core developers.
Many thanks to Google for having the program again this year! -Damian
I was asked to give a five minute speech to open a panel in front of a number of policy makers and advisors in Washington, DC in the past few weeks. The discussion was under Chatham House Rule. A number of people have encouraged me to publish the speech notes as a blog post, it is as follows.
Here I am, a technologist in a room full of policy people. I'll stick
to what I know and try not to put anyone to sleep in the next five
Technology is agnostic, who uses and how they use it matters. Roads,
cars, phones, email, websites are all technologies used for good and bad.
In the 1930s, the feds and police warned of mass chaos if the interstate
highway system was built in the US. The ability for criminals to quickly
transit between cities was of grave concern. Crime would spread faster,
further, and this would hasten the breakdown of the very fabric of the
American society, community. Time has shown the benefits vastly outweighed
the costs. This same principle has shown to be true of the internet
and computer technology. Sure, we may have new kinds of crime with botnets,
zombies, phishing, but do we really? Lying, impersonation, and tricking
someone into doing your work are the same crimes they have been for the
past few millenia. It's just that the substrate that is used, has changed.
What are some of the largest companies in the world? GE, IBM, Apple,
Microsoft, Google. What one should or should not do is policy and law,
what one can actually do or not do is technology.
Circumvention, anonymity, and privacy tools used in a free world can be
a minor annoyance, i.e. wikileaks used wikis, ssl, email, and yes, tor,
but in the end, it's an annoyance. We don't have people in the streets
rioting trying to overthrow our govt. Wikipedia uses the same technology
in wikis, ssl, and email. Everyone loves Wikipedia and considers it a net positive.
The same circumvention, anonymity, and privacy tools are deadly to
repressive regimes. The free flow of information and education are of
great concern to a regime trying to control the horizontal and vertical
of every day life. The tactics a regime can use are legal, technical,
and physical. The regime can switch between tactics, generally
depending upon what's economical and most effective.
Roughly 1 billion people are online in some way. Berkman did a study
that found roughly 2% of that billion know what a proxy is, or even that
technology exists to circumvent internet censorship. 98% of the world
accepts that facebook, google, cnn, and the bbc, are blocked and doesn't
try to find ways around it. This doesn't even broach the topic of online
privacy relating to commercial entities nor law enforcement and
intelligence agencies trying to learn the who, what, where, and how of your Internet activities.
Arguing about which proxy technology should get all of the funding and
attention is silly. The budgets and adversaries vastly outweigh the
funding and research into proxies. It's not a zero-sum game, and
the different technologies take very different approaches to success;
freegate/ultrasurf, vpns, psiphon, and web proxies play a game of cat
and mouse with ip addresses and sometimes encryption; tor uses the
strategy of R&D and protecting ones anonymity and privacy first, the
secondary effects of this are well-suited to circumvention too. Tor,
freegate, psiphon, and vpns sum up to roughly $50m in funding from the US govt
of the past few years. Only a very small fraction of that total has made it to actual technology. Compare that to the billions spent on snakeoil
black box technology by the DoD and intelligence agencies preparing for
a cyberwar arms race, much like the nuclear arms race, to deter other
nations from attacking us.
I talked to a member of a terrorist organization in Vietnam. He's been
stalked, harassed, and had everything confiscated multiple times by his
government. You know his organization as Deutsche Welle. He's a
reporter. He had no idea how his plans, documents, and contacts were
being discovered and used against him. His ability to understand the
differences between Tor, JAP, and Freegate was like asking which tires
are best for gravel, snow, or tarmac. The question he didn't even know
to ask is, "What are safe and secure computing and online practices?"
to use my analogy, "what car do I want for those tires? the answer is
a rally car." I spent 4 hours going over how the internet works,
how to think about adversaries online, what is ssl, what it means, what
are phishing, viruses, botnets, and state-sponsored malware. By the
end of the 4th hour, he understood how tor is different than a simple
vpn or proxy server, and when to use tor and when it isn't needed. 3.5h
of that discussion was basic operational, computer, and online security
and safe practices.
So where does this leave us? It leaves us with a mix of education,
technology, and many, many unanswered questions. This is a young field
overall. As the censorship providers and technologies get better, so
will those circumvention technologies. Educating users about internet
safety, risks, and making the tools vastly easier and safer to use
should be a goal.
Tor published a "10 things to think about circumvention tools" paper to
try to distill what we've learned over the past 10 years of doing this.
In a few of these areas, tor is not the best choice, for now.
What about technology? Isn't it going to save us all? Currently,
freegate/ultrasurf, vpns, and web proxies are looking for money to fund
their growing infrastructure costs. The more users you have, the more
servers, more bandwidth, and more costs you incur. Its a quick way to
spend lots of money and get lots of users. However becoming the ISP for
the world gets very expensive, very quickly as you scale up to hundreds
of millions of users. Look at the infrastructures of google, facebook,
yahoo, and microsoft to see the challenges that lie ahead for these tools.
Tor and "distributed tools" look to improve the research and development
and rely on the scaling of users to both provide the circumvention and
grow to become a self-sustaining ISP to the world. We have only begun
to see the power and effects of these technologies with bittorrent, jap,
skype, freenet, i2p, and tor.