ioerror's blog

TorBirdy: 0.1.2 - Our third beta release!

TorBirdy 0.1.2 is out! All users are encouraged to upgrade as soon as possible, especially if you are using Thunderbird 24.

Notable changes in this release include:

0.1.2, 04 Nov 2013

  • New options:
    • restore default TorBirdy settings
    • toggle checking of new messages automatically for all accounts
  • The minimum version of Thunderbird we now support is 10.0 (closes #9569)
  • `--throw-keyids' is now disabled by default (closes #9648)
  • We are no longer forcing Thunderbird updates (closes #8341)
  • Add support for Thunderbird 24 (Gecko 17+) (closes #9673)
  • Enhanced support for Thunderbird chat
  • We have a new TorBirdy logo. Thanks to Nima Fatemi!
  • Improved documentation:
  • Add new translations and updated existing ones
    • Please see the Transifex page for more information and credits

We offer two ways to install TorBirdy -- either by visiting our website (sig) or by visiting the Mozilla Add-ons page for TorBirdy. Note that there may be a delay -- which can range from a few hours to days -- before the extension is reviewed by Mozilla and updated on the Add-ons page.

As a general anonymity and security note: we are still working on two known anonymity issues with Mozilla. Please make sure that you read the Before Using TorBirdy and Known TorBirdy Issues sections on the wiki before using TorBirdy.

We had love help with translations, programming or anything that you think will improve TorBirdy!

TorBirdy: our first beta release!

Today we are happy to release our first beta of TorBirdy. It has been in development since April of last year and was released internally on the tor-talk mailing list. We think we've had just over five thousand users testing it in the last year. We have polished it and we've made great progress.

What is TorBirdy?

TorBirdy is a Torbutton like extension for Thunderbird, Icedove and related Mozilla mail clients. It may also work with other non-web browser Mozilla programs such as Sunbird. We've also added support for JonDo, Whonix, Tails; if that means something to you, let us know how it works!

We offer two ways to install TorBirdy - either by visiting our website (sig) or by visiting the Mozilla AddOn page for TorBirdy (xpi available here).

As a general Anonymity and security note: We're still working on two known anonymity issues with Mozilla. When our improvements to Thunderbird are accepted, it will be anonymity ready out of the box and we'll do a proper full release.

We'd love help with translations, programming or anything that you think will improve TorBirdy!

Thanks to all of our TorBirdy users and contributors - Sukhbir and I would especially like to tagnaq and Karsten N!

Ultrasurf: the definitive review

In the summer of 2011, I spent a few months learning how to effectively reverse engineer Windows software. I'm still learning and while I have a lifetime of learning to do on the topic, I chose to audit Ultrasurf as a challenge. This research was performed as a labor of love and it was funded work. My interest in reverse engineering Ultrasurf comes entirely because I have seen people promoting it without also offering evidence that it is safe. Additionally, a few people had asked me what I thought of the software and in order to form an opinion, I decided to dig deeper.

Ultrasurf is software produced by the UltraReach company for censorship circumvention, privacy, security and anonymity. Unfortunately for them, I found their claims to be overstated and I found a number of serious problems with Ultrasurf.

My report is available for download from the following link:

Most of my research was done while traveling in Brazil, Canada, Germany, and very small amount of it was performed in the US. Additionally, a number of interesting data points in my research paper came from interception devices in Syria. As of early April 2012, an independent tester confirmed many of my findings from China; the versions of Ultrasurf tested did directly connect to blocked addresses and did not in-fact work at all. Newer versions appear to have different, not yet blocked, addresses baked into the program.

I believe that coordinated disclosure is reasonable in most cases and I ensured that Ultrasurf was notified long before the publication of this blog post. I had a face to face meeting in early December of 2011 to discuss my findings with the lead developer of Ultrasurf and to give them time to fix the problems that I discovered. Ultrasurf updated their website to change a number of their security, privacy and anonymity claims; they did not actually remove all of the bogus claims, merely the most egregious statements. Our meeting was overall quite positive and in fact led me to write notes that may become a second paper.

However, for various reasons, I've had to sit silently on this report for nearly four full months after our December meeting. I believe it is important to ensure that the issues discovered and discussed in my paper are resolved and that users are not kept in harm's way. I have serious concerns about ongoing security issues for the users of Ultrasurf and that is my primary reason for wishing to perform and release this research for all to see.

Here's the abstract of the paper:
Ultrasurf is a proxy-based program promoted for Internet censorship circumvention. This report gives a technical analysis of the Ultrasurf software and network. We present the results of reverse engineering the Ultrasurf client program, give an in-depth study of the known Ultrasurf network, especially those portions that interface in some way with the client or the Internet, and discuss network signatures that would allow an adversary to detect its use on a network. We cover client bootstrapping methods, censorship and censorship resistance, anonymity, user tagging by Ultrasurf and other parties, cryptographic internals and other previously unknown or undiscovered details about the Ultrasurf client and the Ultrasurf network. We find that it is possible to monitor and block the use of Ultrasurf using commercial off-the-shelf software. In particular, BlueCoat sells software and hardware solutions with such capabilities that have been deployed in Syria and other countries.

The vulnerabilities presented in this paper are not merely theoretical in nature; they may present life-threatening danger in hostile situations. We recommend against the use of Ultrasurf for anonymity, security, privacy and Internet censorship circumvention.

The main substance of the paper takes the time to refute nearly all of the claims that UltraReach makes on their website about their software Ultrasurf:
This paper addresses the following claims by UltraReach and other Ultrasurf advocates about the Ultrasurf client and Ultrasurf network:

  1. “Ultrasurf enables users to browse any website freely” — refuted in Section 3.1
  2. “employs a decoying mechanism to thwart any tracing effort of its communication with its infrastructure.” — refuted in Section 5.13
  3. “Protect your privacy online with anonymous surfing and browsing. Ultrasurf hides your IP address, clears
    browsing history, cookies, and more.” — refuted in Section 6.2 and Section 6.3.

  4. “change IP addresses a million times an hour” — refuted in Section 6.1
  5. “Untraceable” — refuted in Section 6.10
  6. “Unblockable: Client uses wide array of discovery mechanisms to find an available proxy server and, when necessary, to switch/hop to avoid tracking/blocking” — refuted in Section 6.8
  7. “Invisible: Leaves no traces on the user’s computer, and its traffic is indistinguishable from normal access to HTTPS sites” — refuted in Section 5.12
  8. “Anonymous: No registration is requires [sic], and no personally identifying information collected” — refuted in Section 6.10
  9. “Tamperproof: Using privately-signed SSL certificates which dont depend on external, potentially compromised CAs (thus preempting MITM attacks), Ultrasurf proactively detects attempts by censors to reverse-engineer, sabotage, or otherwise interfere in the secure operation of the tool” — refuted in Section 5.8.

We conclude that each of these claims is false, incorrect, or misleading.

The issues involved in the writing, discussion and publication of this report are the stuff of movies. It has taken ages to publish this report and attempts at coordinated disclosure have been time consuming, largely fruitless and extremely frustrating. While some of the issues I have identified have been fixed, to the best of my knowledge the most important issues, such as a lack of forward secrecy, remain serious outstanding security issues. Ultrasurf often boasts of their decade long fight against censorship and while I respect the spirit of their efforts, I have a hard time respecting the technical implementation. I'm afraid that they've not had forward secrecy in their cryptographic protocol for that entire decade. Ultrasurf also often boasts of being untraceable when in fact they admitted to logging and disclosing user identifying logs to law enforcement when the data was requested. These kinds of security failures, both social and technical, are simply negligent and it means that users have been and are likely still in harm's way.

I firmly believe that Ultrasurf must publish their full technical specifications, peer review their designs of both obfuscation and cryptography, open their source code for the world to review and they must absolutely discontinue all data retention without exception.

I hope you'll enjoy the research presented in the paper and that it will help everyone to move towards building a more secure set of options for users.

UltraReach/Ultrasurf have released a response document and a response page that confirms a number of my claims, side steps a large swath of them and then attacks me, Tor and others for the report. They specifically claim that what is true in my paper is for older versions of Ultrasurf. They do not disclose which versions or when the fixes were released. This is a typical vendor tactic considering that they pressured me not to release the report until they felt they were given enough time to fix the issues involved. They also believe that I claim that Ultrasurf was broken but at no time did I ever claim it was broken; rather, I said it has problems. The claims they made and make do not live up to the implementation of policies or technical capabilities. This I think is quite reasonable because their claims were, frankly, entirely unreasonable.

I put a great deal of time and effort into disclosing these report findings to Ultrasurf - both what would be considered responsible and coordinated - it's too bad that they've decided to ignore most of the findings and to attack me over the undefendable issues.

Another Update: Collin Anderson has written up his view of the disclosure process. He is an independently involved third party that attempted to mediate our disclosure, solutions and a reasonable time frame for all parties involved.

University of Washington Open Hackfest

We're having an open hackfest at the University of Washington on Feb 22nd and 23rd; we may hold an additional open hackfest day on Friday, Feb 24th if we feel the demand. This meeting is largely possible due to the support of the UW Security and Privacy Research Lab.

This hackfest coincides with our Winter Developer summit and many Tor developers will be in attendance. As I write Tor developers have already started their travel to Seattle and many will stick around for the following week.

We'd love to welcome everyone interested in attending. We'd especially like people to feel welcome to discuss ideas or proposals, who want to know what's happening in the world of censorship resistance, anonymity, privacy and related topics. Most of all if you're prepared to write software, we're planning to do quite a lot of that next week as well.

So please, come join us on the UW campus for our winter open hack fest! Times and locations are available on our wiki page about the open hack fest.

A tale of new censors - Vodafone UK, T-Mobile UK, O2 UK, and T-Mobile USA

The right to read is a fictional story but it warns of a future that has already started to arrive; it paints a picture where information is controlled with a heavy hand and simply reading, let alone speaking is an extremely dangerous activity. In the words of William Gibson, "The future is already here — it's just not very evenly distributed". Restrictions on the right to read though the Internet perfectly match this observation. A lot should be said about perceptions of censorship, and it is often thought that places like Syria or Iran are unique. Generally, people in the West hold that those countries obviously censor as is consistent with facts of life in a supposedly non-free country. This probably holds a lot of truth but it absolutely fails to address the core of the issue — these countries and those networks are not unique.

In fact, we find uncensored networks to almost be an abnormal state. The so-called free countries in the West often shape and tamper with network traffic. They often also log data and even collaborate with governments. Generally, people don't see evidence of this and as a result, they often perceive that their Internet connections aren't monitored or censored. These days are quickly coming to an end and while it sounds like hyperbole, here are examples in the United Kingdom and in the United States of America.

Recently it has come to our attention that our primary website is filtered by Vodafone in the UK, by 3 ( in the UK, by O2 in the UK, and by T-Mobile in the UK and the USA. It used to be the case that we only saw filtering and censorship events in places like Egypt, Syria, or Iran and now we're going to explore what those attacks look like in the context of the UK and the USA.

When a visitor uses a pre-paid account on the T-Mobile USA network and attempts to visit, they are redirected to a block page. This is enabled by default without user's affirmative consent and only savvy privileged users may even attempt to disable this censorship. There is an informational page about the T-Mobile censorship system and it explains that this censorship may be disabled. We've heard reports that attempts to disable the censorship are not always successful and this certainly doesn't bode well for an easy and censorship-free Internet experience.

The T-Mobile USA network censorship appears to be simple to bypass: it appears to only trigger when a client sends Host: on TCP port 80 and visitors that use HTTPS will probably not notice or be obviously impacted by their censorship.

This kind of censorship raises all kinds of interesting questions. I suspect it raises US legal and social questions as well. The Tor Project is a registered 501c3 non-profit corporation in the state of Massachusetts, and the block was experienced in California. Does this count as interfering with interstate commerce? What duty of care does T-Mobile USA have when it relies on systems or infrastructure funded by the public? What duty of care do they have as a common carrier?

Similarly, when a user on the UK Vodafone network visits they are greeted by a block page as well. You can visit this block page without directly using their networks. Detecting their filters is straightforward and we see tampering at the sixth hop.

Here is a tcptraceroute to TCP port 80 of from an Ubuntu machine connected to the Internet via Vodafone UK:

Tracing the path to ( on TCP port 80 (www), 30 hops max
1 2.379 ms 1.011 ms 1.313 ms
2 90.998 ms 133.672 ms 95.963 ms
3 78.865 ms 91.722 ms 91.415 ms
4 * * *
5 88.502 ms 73.259 ms 80.765 ms
6 ( [open] 77.927 ms 152.599 ms 96.399 ms

Here is a normal traceroute to from an Ubuntu machine connected to the internet via Vodafone UK:

traceroute to (, 30 hops max, 60 byte packets
1 ( 9.669 ms 9.583 ms 9.460 ms
2 ( 98.084 ms 98.046 ms 98.224 ms
3 ( 98.760 ms 109.326 ms 109.261 ms
4 host203.msm.che.vodafone ( 109.087 ms 127.554 ms 127.426 ms
5 * * *
6 * * *
7 * * *
8 * * *
9 ( 180.920 ms 180.692 ms 180.652 ms
10 ( 180.659 ms 180.473 ms *
11 ( 260.480 ms * ( 152.107 ms
12 ( 152.265 ms 152.099 ms 151.808 ms
13 ( 151.453 ms 151.124 ms ( 151.129 ms
14 ( 157.978 ms ( 119.699 ms 129.820 ms
15 ( 129.999 ms 136.314 ms 136.338 ms
16 ( 136.033 ms 135.826 ms 135.666 ms
17 ( 151.282 ms 118.185 ms 114.603 ms

We've additionally found that pre-paid T-Mobile UK accounts also experience censorship that is similar to T-Mobile USA. Detection of their filter is possible with some of the techniques that I've demonstrated, and it is quite trivial to see that TCP port 80 and 443 are treated in a special way.

Here is a tcptraceroute to TCP port 80 of from an Ubuntu machine connected to the Internet via T-Mobile UK:

Tracing the path to ( on TCP port 80 (www), 30 hops max
1 * * *
2 305.721 ms 429.908 ms 449.875 ms
3 480.031 ms 339.890 ms 429.951 ms
4 480.447 ms 449.365 ms 439.979 ms
5 ( [open] 459.935 ms 659.964 ms 449.849 ms

Here is a tcptraceroute to TCP port 443 of from an Ubuntu machine connected to the Internet via T-Mobile UK:

Tracing the path to ( on TCP port 443 (https), 30 hops max
1 * * *
2 357.474 ms 360.016 ms 389.772 ms
3 490.136 ms 409.878 ms 359.945 ms
4 469.956 ms 489.883 ms 389.868 ms
5 ( 410.024 ms 420.494 ms 399.888 ms
6 389.470 ms 429.923 ms 339.861 ms
7 430.002 ms 349.850 ms 450.012 ms
8 339.900 ms 389.836 ms 390.031 ms
9 369.851 ms * 924.522 ms
10 420.035 ms 379.878 ms 409.968 ms
11 ( 469.942 ms 480.002 ms 499.940 ms
12 ( 399.851 ms 379.892 ms 379.929 ms
13 ( 419.899 ms 479.926 ms 449.923 ms
14 ( 389.925 ms 449.789 ms 549.993 ms
15 ( [open] 419.869 ms 469.997 ms 479.839 ms

Compare with a normal traceroute to from an Ubuntu machine connected to the Internet via T-Mobile UK:

traceroute to (, 30 hops max, 60 byte packets
1 * * *
2 ( 99.671 ms 99.856 ms 159.584 ms
3 ( 179.672 ms 190.046 ms 159.760 ms
4 ( 190.250 ms 179.356 ms 90.611 ms
5 ( 90.565 ms 110.275 ms 90.508 ms
6 ( 110.476 ms 110.449 ms 110.391 ms
7 ( 70.022 ms 70.062 ms 60.303 ms
8 ( 60.322 ms 69.380 ms 69.383 ms
9 * * *
10 ( 59.798 ms 60.535 ms 179.659 ms
11 ( 240.999 ms 221.715 ms 221.191 ms
12 ( 230.570 ms 229.966 ms 210.814 ms
13 ( 210.575 ms 200.446 ms 199.453 ms
14 ( 169.521 ms 148.181 ms 168.037 ms
15 ( 248.264 ms 229.474 ms 249.066 ms
16 ( 249.289 ms 249.234 ms 259.448 ms

In the examples above we see that T-Mobile UK treats TCP port 80 in a special manner and effectively stops users from reaching our web site. This is an attack against users who attempt to connect to our infrastructure. This attack, while primitive, demonstrates an active and malicious action on the part of the above named Internet providers.

We've additionally seen reports of the UK O2 network blocking connections to in exactly the same way that Vodafone UK blocks access. The O2 filter has been covered in the popular media in the recent past and we're sad to hear that they've decided to include Tor's website in their race to the bottom.

In all the above cases we do not see DNS tampering but rather outright Man-In-The-Middle attacks against connections to our web server. These censorship systems do not currently implement a Man-In-The-Middle attack against the SSL services offered by our web server. It is not much of a stretch of the imagination to think that such an action may be a future plan; we've seen it elsewhere.

Current users of the Tor network are not impacted by this filtering, but these networks are attempting to deny new users the ability to start using Tor without extensive efforts. You can view their filter page without using their service; the exact block page is also available externally. It appears that it is possible for users to disable this censorship by providing a credit card as a proof of age. This is not exactly a privacy-friendly tactic. The O2 Twitter account contacted me and said they were willing to review their censorship policy for but they did not offer to remove the censorship entirely.

This trend of providing partially censored Internet in what we all think of as free countries is alarming. Are we supposed to look the other way because the mobile Internet isn't the same as the "real" Internet? Should we worry that Vodafone's capabilities and behavior here remind us of what they did in Egypt last year? It would seem that the war over network neutrality is far from won.

(Investigation and research thanks go to Andrew Lewman, Steven Murdoch and Runa Sandvik of the Tor Project, SiNA of RedTeam LLC, Jim Killock, Lee Maguire, Peter Bradwell of the Open Rights Group and their project and Richard Clayton from the University of Cambridge.)

DigiNotar Damage Disclosure

About an hour ago I was contacted by the Dutch Government with more details about the DigiNotar Debacle. It seems that they're doing a great job keeping on top of things and doing the job that DigiNotar should've done in July. They sent a spreadsheet with a list of 531 entries on the currently known bad DigiNotar related certificates.

The list isn't pretty and I've decided that in the interest of defenders everywhere without special connections, I'm going to disclose it. The people that I have spoken with in the Dutch Government agree with this course of action.

This disclosure will absolutely not help any attacker as it does not contain the raw certificates; it is merely metadata about the certificates that were issued. It includes who we should not trust in the future going forward and it shows what is missing at the moment. This is an incomplete list because DigiNotar's audit trail is incomplete.

This is the list of CA roots that should probably never be trusted again:

DigiNotar Cyber CA
DigiNotar Extended Validation CA
DigiNotar Public CA 2025
DigiNotar Public CA - G2
Koninklijke Notariele Beroepsorganisatie CA
Stichting TTP Infos CA

The most egregious certs issued were for *.*.com and *.*.org while certificates for Windows Update and certificates for other hosts are of limited harm by comparison. The attackers also issued certificates in the names of other certificate authorities such as "VeriSign Root CA" and "Thawte Root CA" as we witnessed with ComodoGate, although we cannot determine whether they succeeded in creating any intermediate CA certs. That's really saying something about the amount of damage a single compromised CA might inflict with poor security practices and regular internet luck.

Of particular note is this certificate:
CN=*,SN=PK000229200006593,OU=Sare Toro Ham Mishkanam,L=Tehran,O=Hameye Ramzaro Mishkanam,C=IR

The text here appears to be be an entry like any other but it is infact a calling card from a Farsi speaker. is not a valid domain as of this writing.

Thanks to an anonymous Farsi speaker, I now understand that the above certificate is actually a comment to anyone who bothers to read between the lines:
"RamzShekaneBozorg" is "great cracker"
"Hameyeh Ramzaro Mishkanam" translates to "I will crack all encryption"
"Sare Toro Ham Mishkanam" translates to "i hate/break your head"

Without any further delay, I've uploaded the original spreadsheet and a CSV text file for people who don't trust spreadsheets. The information contained in both files should be the same. Hopefully this information will help people to mitigate certain harm from the DigiNotar Debacle.

The DigiNotar Debacle, and what you should do about it

Recently it has come to the attention of, well, nearly the entire world that the Dutch Certificate Authority DigiNotar incorrectly issued certificates to a malicious party or parties. Even more recently, it's come to light that they were apparently compromised months ago or perhaps even in May of 2009 if not earlier.

This is pretty unfortunate, since correctly issuing certificates is exactly the function that a certificate authority (CA) is supposed to perform. By comparison, ComodoGate looks fairly minor.

This incident doesn't affect the functionality of Tor clients or the Tor Network itself, since Tor doesn't use the flawed CA system. The Tor network uses a much simpler and flatter trust design that protects us from many of these CA issues. Further, Tor's distributed-trust design limits the damage from compromise of any given network component.

But the incident does affect users that are attempting to reach The Tor Project's infrastructure: with one of these bogus certificates, an attacker could convince your browser that you were talking to The Tor Project website, when really you were talking to the attacker.

We have taken direct action in an attempt to stop this kind of attack in the future with two major browser vendors and we hope to integrate a fix with all other willing browsers. Please contact us if you ship a browser and you'd like to help your users to be proactively secure when visiting our sites. TorBrowser users should upgrade to our latest release and verify signatures for all downloaded files. All Tor Browser Bundles have been updated to Firefox 6 with a patch to stop trusting the offending CA, and users are encouraged to upgrade. Below, we describe what we found out, what we're doing about it, and what you should do to keep yourself safe.

The attack

In the last seventy-two hours we were working to find positive confirmation that The Tor Project was one of the targeted groups. It was originally disclosed that at least one certificate was issued for '*' and that it was being used to actively Man-In-The-Middle SSL and TLS connections. Quite quickly we found a similar pattern to the ComodoGate fiasco. It appears likely that the Mozilla Addons site, Yahoo, Facebook, Twitter, and a few other major players were targeted. We do not have an authoritative list but I personally believe those targets to be accurate; time will tell. Additionally, we heard rumors that we had graduated to the big leagues and we had also heard that DigiNotar had reached out to the major browser vendors. We did not receive any proactive contact from DigiNotar as a browser vendor and this worried us greatly when compounded with the rumor of being one of the targets as well.

We ship a rather specific and special browser and it appeared that all of our sites are specifically in the attacker's target list. Having received no contact from DigiNotar, we reached out to DigiNotar by email and by telephone.

I spoke on the telephone with a rather nice but obviously overworked DigiNotar point of contact who will remain anonymous. He was guarded and careful in what he said but was clearly sympathetic to the severity of the matter at hand. It seemed quite clear that he repeated similar information to other impacted callers:

"What I can say is the following " ... "Any fraudulent certificate that has been requested has been revoked. Any serial numbers that we know about have been revoked. All serial numbers have been communicated to the major browsers vendors." ... "Any certificate that we know of is revoked by OCSP server."

We emailed quite a bit back and forth after the phone call. A few hours later that same point of contact from DigiNotar sent a list of all of the certificates in a spreadsheet. It appears that the attackers requested twelve certificates, and each certificate was for '*'. The first batch of six certificates was issued on July 18th and the second batch of six certificates was issued on July 20th. According to the spreadsheet, the first six of the certificates expired on August 17th, 2011 and second batch of six certificates expired on August 19th, 2011. According to the information disclosed by DigiNotar the certificates in question should all have expired. The contact at DigiNotar stressed that there was no confirmation about the attacker(s) receiving the certificates. I have no reason to believe that these certificates would have any more trouble reaching the requesting party than the Google certificate used in the wild.

This is the current list of serial numbers for all twelve Tor Project certificates as disclosed to us by DigiNotar:


DigiNotar has not provided us with a copy of any of the certificates that they issued. We are not sure that they have copies nor if they are willing to disclose any copies they may or may not have. This point is extremely disconcerting as the CRL/OCSP revocation process is essentially worthless. Mere serial numbers are simply not enough in some cases — especially when a full list of all likely compromised serial numbers has not been disclosed as happens to currently be the case.

To the best of our knowledge and by analyzing the CRLs for DigiNotar, we do not believe that any of the fraudulently issued '*' certificates have been revoked at the time of this writing. It may be the case that they are simply not in the business of revoking certificates after they have expired. There is no evidence to support revocation during the time that these certificates were perfectly valid.

I believe that you can clearly see the MITM attack in action around the tenth hop of this traceroute thanks to an anonymous person in Iran:

1 3 ms 14 ms 2 ms
2 67 ms 67 ms 65 ms 91.99.***.*** [91.99.***.***]
3 65 ms 67 ms 93 ms
4 67 ms 72 ms 66 ms
5 66 ms 64 ms 64 ms
############### [ MORE Nodes ] #################
6 451 ms 195 ms 154 ms
7 626 ms 231 ms 88 ms
8 93 ms 91 ms 96 ms
9 88 ms 94 ms 120 ms
################### [ MORE ] ###################
10 88 ms 88 ms 88 ms ####DIfferent IP (

#### [ OUT OF IRAN ] ####
11 340 ms * * []

To quote someone I respect greatly: "That's not dodgy at all!"

Early statements By DigiNotar translated by someone and mentioned by a friendly Dutch man lead us to believe that DigiNotar and their parent company are in damage-control mode. It would be unsurprising to hear that the Dutch Government is similarly in the dark about the scope of the compromise, as it appears DigiNotar does not control a canonical list for all certificates issued. While some Browser vendors have received a list, I do not have confidence that this list actually contains all malicious certificates that have been issued: rather it appears to be a subset that did not even include the Google certificate that was being used in the wild. We hope that DigiNotar will fully disclose whatever information they have and explain what information they honestly lack.

The defense

Modern versions of Chrome (13) were able to prevent MITM attacks against most, if not all of the Google sites where they had certificate pinning and where HSTS was enabled. Google has also announced the attacks and updated information about it. Additionally, they have distrusted DigiNotar in Chrome.

We've sent a request to Google that they enable HSTS and pin certificates for some of the critical Tor sites and that patch is pending. Google has been very good about all of this and I can't thank them enough for their help.

As it stands, Chrome appears to have shipped a fix that distrusts DigiNotar and it appears to treat hundreds of certificates as if they are specifically known to be malicious or hostile. Mozilla and others have shipped a fix as well. Sadly, it appears that the Dutch Government asked various browser vendors to create an exemption for certain trust chains as some kind of compromise. However, we were not party to any of the discussions, and we don't understand the core concerns for such a compromise. We're not willing to take a leap of faith for a Certificate Authority that did not contact us when they first noticed this problem. Right now, if we found a DigiNotar-issued certificate certifying that water was wet, we wouldn't believe it without checking for ourselves. Twice.

We have proactively given DigiNotar an "Internet Death Sentence" in the Tor Browser. The direct impact of removing DigiNotar should be on the order of around seven hundred certificates according to some cursory queries run against the EFF's SSL Observatory. I believe that the number of certificates revoked is nearing parity with the number of possibly legitimate still-valid certificates issued by DigiNotar. That's a sad state of affairs.

We do not currently have evidence of any tampering with Tor downloads, but we're looking. If an attacker can successfully perform a MITM attack, there is nothing to prevent them from giving you a bogus package instead of the software you were actually looking for — if you're not checking package signatures, there's no easy way to tell good software from bad.

What you should do

First of all, upgrade your browser(s). See this blog post announcing the new Tor Browser Bundles with Firefox 6.

Note that verifying the signatures on Tor packages prevents attackers like this from causing you to install a possibly backdoored version of Tor. You should always verify the signature of any software you download. We encourage you to learn more about secure signature verification.

If you have downloaded copies of any Tor software in the past few months, and you did it over any network that you don't trust, please help us check to see whether there was any attempt to alter them. We don't expect you'll find anything, but if you do, we really want to know about it. In any case, it will be good practice for checking signatures.

If you have any information about certificates that you believe to be false, please do send us the certificates and we'll take a look.

The Certificate Authority system as it stands today is a house of cards and we're witnessing in public what many have known for years in private. The entire system is soaked in petrol and waiting for a light. There are some new directions for trust in the works such as Convergence and various ways to do DNSSEC authenticated HTTPS as well as other hacks. Still, nothing is set in stone or standardized and this is why we need to remain vigilant. We're hoping to detect these kinds of attacks in the future with our distributed SSL Observatory and we hope that you'll join us.

I'd like to end on a positive note and quote a personal hero and friend, Matt Blaze: "A decade ago, I observed that commercial certificate authorities protect you from anyone from whom they are unwilling to take money. That turns out to be wrong; they don't even do that much."

Detecting Certificate Authority compromises and web browser collusion

Thanks to Ian Gallagher, Seth Schoen, Jesse Burns, Chris Palmer, and other anonymous birds for their invaluable feedback on this writeup.

The Tor Project has long understood that the certification authority (CA) model of trust on the internet is susceptible to various methods of compromise. Without strong anonymity, the ability to perform targeted attacks with the blessing of a CA key is serious. In the past, I’ve worked on attacks relating to SSL/TLS trust models and for quite some time, I’ve hunted for evidence of non-academic CA compromise in the wild.

I’ve also looked for special kinds of cooperation between CAs and browsers. Proof of collusion will give us facts. It will also give us a real understanding of the faith placed in the strength of the underlying systems.

Does certificate revocation really work? No, it does not. How much faith does a vendor actually put into revocation, when verifiable evidence of malice is detected or known? Not much, and that’s the subject of this writing.

Last week, a smoking gun came into sight: A Certification Authority appeared to be compromised in some capacity, and the attacker issued themselves valid HTTPS certificates for high-value web sites. With these certificates, the attacker could impersonate the identities of the victim web sites or other related systems, probably undetectably for the majority of users on the internet.

I watch the Chromium and Mozilla Firefox projects carefully, because they are so important to the internet infrastructure. On the evening of 16 March, I noticed a very interesting code change to Chromium: revision 78478, Thu Mar 17 00:48:21 2011 UTC.

In this revision, the developers added X509Certificate::IsBlacklisted, which returns true if a HTTPS certificate has one of these particular serial numbers:

  • 047ecbe9fca55f7bd09eae36e10cae1e
  • d8f35f4eb7872b2dab0692e315382fb0
  • b0b7133ed096f9b56fae91c874bd3ac0
  • 9239d5348f40d1695a745470e1f23f43
  • d7558fdaf5f1105bb213282b707729a3
  • f5c86af36162f13a64f54f6dc9587c06

A comment marks the first as "Not a real certificate. For testing only." but we don’t know if this means the other certificates are or are not also for testing.

With just these serial numbers, we are not able to learn much about the certificates that Chromium now blocks. To get more information, I started the crlwatch project. Nearly every certificate contains a reference to a Certificate Revocation List (CRL). A CRL is a list of certificates that the CA has revoked for whatever reason. In theory, this means that an attacker is unable to tamper with the certificate to prevent revocation as the browser will check the CRL it finds in a certificate. In practice the attacker simply needs to tamper with the network - this is something they’re already able to do if they are performing a SSL/TLS Machine-In-The-Middle attack. Even if an attacker has a certificate, they generally are unable to modify the certificate without breaking the digital signature issued by the CA. That CA signature is what gives the certificate value to an attacker and tampering takes the attacker back to square zero. So while investigating these serials, we clearly lack the CRL distribution point in the Chrome source. However, the project that I announced on March 17th, crlwatch, was specifically written to assist in finding who issued, and potentially revoked the serial numbers in question. By matching the serial numbers found in the source for Chrome with the serial numbers of revoked certificates, we’re able to link specific serials to specific CA issuers. The more serial numbers we match in revocation lists, the higher our probability of having found the CA that issued the certificates.

About twelve hours (Thursday, March 17, 2011 | 13:00) after the above patch was pushed into source control - Google announced an important Chrome Update that involved HTTPS certificate issues.

This also is mostly uninteresting until we notice that this is not isolated to Google. Mozilla pushed out two patches of interest:

The complete changeset is semi-informative. Mozilla references a private bug in that fix that Mozilla will hopefully disclose. Similar to Chromium, the Mozilla patches create a list of certificate serial numbers that will be treated as invalid. However, the serial numbers from the Mozilla patches are different:

  • 009239d5348f40d1695a745470e1f23f43
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 72032105c50c08573d8ea5304efee8b0
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00d7558fdaf5f1105bb213282b707729a3
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 00f5c86af36162f13a64f54f6dc9587c06
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71

Thus, both Mozilla and Google shipped similar patches to their code at roughly the same time. The two browsers now have partially overlapping certificate blocklists. Here is the union of the two lists:

  • 009239d5348f40d1695a745470e1f23f43
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00d7558fdaf5f1105bb213282b707729a3
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00f5c86af36162f13a64f54f6dc9587c06
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71
  • 72032105c50c08573d8ea5304efee8b0
  • 9239d5348f40d1695a745470e1f23f43
  • b0b7133ed096f9b56fae91c874bd3ac0
  • d7558fdaf5f1105bb213282b707729a3
  • d8f35f4eb7872b2dab0692e315382fb0
  • f5c86af36162f13a64f54f6dc9587c06

Why do the browsers have these blocklists, and why don’t they have the same blocklists?

This returns me to the reason for creating crlwatch last week - I wanted to find the someones who knowingly revoked the above listed special certificates. Anyone looking from the same starting point as I did, obviously lacks the leaf certificates in question and as a result, I had to look in a rather round about manner. Thanks to the EFF’s SSL Observatory, I was able to populate the base list for crlwatch. Armed with a nearly canonical list of all CRLs, I fetched them over Tor and parsed the CRL data into human readable text. The goal was to search for the above serial numbers and to find something linkable.

This is the result from searching the crlwatch data:

Looking for 009239d5348f40d1695a745470e1f23f43 in parsed CRLs...
Looking for 00b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...
Looking for 00d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...
Looking for 00d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...
Looking for 00e9028b9578e415dc1a710a2b88154447 in parsed CRLs...
Looking for 00f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...
Looking for 047ecbe9fca55f7bd09eae36e10cae1e in parsed CRLs...
Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E
Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E
Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E
Looking for 392a434f0e07df1f8aa305de34e0c229 in parsed CRLs...
Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Looking for 3e75ced46b693021218830ae86a82a71 in parsed CRLs...
Match! Serial Number: 3E75CED46B693021218830AE86A82A71
Match! Serial Number: 3E75CED46B693021218830AE86A82A71
Match! Serial Number: 3E75CED46B693021218830AE86A82A71
Looking for 72032105c50c08573d8ea5304efee8b0 in parsed CRLs...
Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06

Huzzah! It appears that we've found a few matches!

Here are the three matching files in human readable format:

Matching entries in the list look like this:

Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Revocation Date: Mar 15 20:15:38 2011 GMT

An interesting note is that this date is a bit earlier than the above patches. The CA knew to revoke it on March 15th and the above patches were worked into software a few days later. If the attacker was targeting specific users, the damage to those users may have already been inflicted.

All three of the CRLs in question belong to the same CA:

issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST

This appears to be a reseller or something similar for the Comodo CA company when the trust chain for USERTRUST is inspected:

CN = COMODO High Assurance Secure Server CA

We appear to have no initial matches for the following Mozilla specific serials from the data that I gathered during the initial crlwatch data population run:

  • 009239d5348f40d1695a745470e1f23f43
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00d7558fdaf5f1105bb213282b707729a3
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00f5c86af36162f13a64f54f6dc9587c06

Those serial numbers appear to not match, right? Nope. Mozilla appears
to deal with certificate serial numbers in a slightly different manner - Chrome does the same internally but Mozilla exposes a weird quirk of certificate encoding directly in the source. The human readable data does not contain this quirk. Thus if you remove the prefix of ‘00’ from those serial numbers and search for the sixteen byte values, we find what we'd expect:

Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Looking for e9028b9578e415dc1a710a2b88154447 in parsed CRLs...
Match! Serial Number: E9028B9578E415DC1A710A2B88154447
Match! Serial Number: E9028B9578E415DC1A710A2B88154447
Match! Serial Number: E9028B9578E415DC1A710A2B88154447
Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06

Here's sample from one of those CRLs:

Serial Number: D7558FDAF5F1105BB213282B707729A3
Revocation Date: Mar 15 20:15:26 2011 GMT

Ironically, after all of this work, the Mozilla patch also leaks the CA name and confirmed my suspicions without question.

In the end, when the lists are merged, we find eleven certificates with two certificates probably acting as testing certificates for the two vendors involved:

  • 077a59bcd53459601ca6907267a6dd1c
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71
  • 72032105c50c08573d8ea5304efee8b0
  • 9239d5348f40d1695a745470e1f23f43
  • b0b7133ed096f9b56fae91c874bd3ac0
  • d7558fdaf5f1105bb213282b707729a3
  • d8f35f4eb7872b2dab0692e315382fb0
  • e9028b9578e415dc1a710a2b88154447
  • f5c86af36162f13a64f54f6dc9587c06

This is evidence of a rather serious event and one that cannot be ignored. If I had to make a bet, I'd wager that an attacker was able to issue high value certificates, probably by compromising USERTRUST in some manner, this was discovered sometime before the revocation date, each certificate was revoked, the vendors notified, the patches were written, and binary builds kicked off - end users are probably still updating and thus many people are vulnerable to the failure that is the CRL and OCSP method for revocation. Even after users update, I'm guessing they may be unequally protected. Mozilla and other browsers should force OCSP verification by default as part of their next release and remove CAs that are unable to handle this requirement. Users of Mozilla Firefox that are concerned about this issue should enable security.OCSP.require in the about:config dialog. The surveillance concerns of enabling OCSP are serious - a CA learns what sites you’re visiting. However, they are nullified by the fact that OCSP checking is enabled by default on Firefox at least; it simply doesn’t provide any security gains for the end user because when it fails, it fails open!

I contacted both Google and Mozilla (bug #643056) for comment after discovering the above data. Mozilla expressed some concern about disclosure and I offered to embargo this document until Tuesday March 22nd, the launch day for Firefox 4. They agreed and I kept this under my hat. After discussions between Comodo and Microsoft, passed on to me by Mozilla, the embargo was to be extended until Wednesday, March 23rd. This extension was ostensibly to ensure that Microsoft would be able to ship their Internet Explorer mitigation pack. After further discussion, Mozilla pushed their blog post about the issue and I now consider the embargo lifted. Google has already shipped a fix to users. Install the latest Firefox to get a patch, if you haven't already. A Tor Browser update is in the works and will be available soon.

Mozilla offered some additional information and disclosed that was one of the certificates acquired by the attacker. In total, nine certificates were acquired. Seven were uniquely named. Two of the certificates were re-issued for a previously issued host name. One certificate was issued for "global trustee" rather than a valid host name. With testing certificates in the list, we have a good accounting of the certificates found in the source code of each browser. Google clarified their discrepancy with the list, acknowledged the duplicate serial mistake and issued subsequent patches. Saving for test hosts, the lists are now identical.

If I had to guess at sites, I'd speculate that Facebook, Skype, Google, Microsoft, Mozilla, and others are worthy of targeting. Comodo should disclose this information and clear up this speculation with very clear information about who was targeted.

Both vendors expressed that the CA in question had done something quite remarkable by disclosing this compromise. The incentives may not be in the favor of the CA for disclosure. Many CAs may fall to similar attacks and simply refuse to disclose. Hopefully crlwatch will provide us with meaningful data regarding revocation events. The EFF and The Tor Project are working on solutions for detecting anomalies in certificates found in the wild. Still, some CAs may simply be unaware of compromises or unwilling to revoke for fear of detection.

Are all other browsers deploying similar countermeasures? Thanks to the free software nature of Firefox and Chrome, we have an answer for at least two projects. I wish that we could say the same for the rest of the browser world. One may assume that the CA in question did their best to contact all impacted vendors and targets.

Comodo has not yet revealed the extent of the compromise to the public - were their signing keys in a hardware security module? How many certificates in total were ever issued by this specific signing key? Wouldn't it be best to remove the specific signing keys from all trust roots to be extra careful given the stakes? Who exactly did they deem important enough for disclosure? The Tor Project, which ships the Tor Browser Bundle was not notified. Clearly some groups are being left out of the loop and this is where even a single attack really causes the entire CA trust model to fall apart.

Comodo should release the full certificates to the internet as well as all of the details relating to the attack. Mozilla and other browsers should open their bug reports, explain their process and lay out a path forward where we won't have to repeat this entire process again.

There is some suspicion that this action was taken by a state level adversary and there are some specific states that have been named. I’ll leave further speculation about which nation states may be involved to others. The mere fact that the web’s system of trust relies on an all or nothing property should be enough of a cause for alarm. It’s obvious that this has been and will continue to be exploited.

The impact on other cryptographic systems, such as S/MIME signatures and other cryptographic systems secured by CAs is entirely un-discussed. Quite seriously, when a CA is compromised, it will impact a great deal more than the web; users of email systems (SMTP, IMAP, POP,etc), Jabber servers, and any other SSL/TLS enabled systems are all at risk. Blocking specific serial numbers or relying on flawed, provably broken methods of revocation will simply not cut it anymore. When the actual protection mechanisms are not enforced, there is little hope of end users being protected.

This should serve as a wake up call to the internet. We need to research, build, and share new methods for ensuring trust, identity, authenticity, and confidentiality on the internet. Proposals such as DANE, CAA, HASTLS, and Monkeysphere are steps in the right direction but they face an uphill battle from entrenched economic interests.

Certification Authorities may continue to provide a piece of the puzzle but it’s high time we ensure that they’re not the alpha and the omega, anymore.

Comodo has issued a statement confirming everything that I've said and more. They believe that this was a targeted attack by a state level actor and they have named Iran as the country they suspect. Mozilla has now opened the bug reports about the issue to the public. Microsoft has now disclosed their report as well.

In the details of their statement we have a confirmation that they have the ability to monitor and thus surveille people who wish to know if certificates are valid.

Comodo also clearly demonstrate a mis-understanding - they believe that checks for revocation are proof positive that certificates are being used. They need to read and understand why this is not true.

The browsers have dropped the ball and they have chosen to fail open in nearly every single case; an attacker who is able to MITM SSL/TLS will also MITM the OCSP/CRL requests. Moxie's sslstrip demonstrated that an attacker would do this automatically and his software has done this for OCSP in public since 2009. Mozilla did not fix this issue at the time and they have once again punted on the issue. An even lower tech attack is possible and it's why revocation does not work: By returning a HTTP 500 error, the browser will the continue on as if revocation checks showed the certificate to be perfectly fine.

The browsers chose a user privacy invasive stance without the user protecting security properties. They did this because they claim that CAs are unable to provide working OCSP/CRL systems for request handling. This is a fair claim if true but it must not stand any longer. If the CA cannot provide even a basic level of revocation, it's clearly irresponsible to ship that CA root in a browser. Browsers should give insecure CA keys an Internet Death Sentence rather than expose the users of the browsers to known problems.

It's probably the case that Mozilla and other browsers should write a secure, caching OCSP server for use when a CA has a failure. It should probably be run by a neutral third party such as the EFF with a strong user privacy stance. This would only serve as a temporary fix and until Browsers get their act together, users are doomed anyway.

OCSP stapling does not fix this issue. The browsers treat revocation errors as soft errors and a MITM is deadly for revocation. The browsers believe they have to treat them as soft errors because the CAs are failing to do their job properly and are almost entirely unaccountable. The browsers are failing users by refusing to hold CAs to account. If OCSP and CRL failures mean the internet doesn't work, we need to create alternatives and not simply sweep these issues under the rug for later analysis. Browsers should hard fail on certificate revocation errors.

Comodo has further failed by:

  • Failing to produce further information about those certificates
  • Selective disclosure to "principal browsers and domain owners"
  • Failing to disclose what sub-CA/intermediate root actually did the signing
  • Believing that the attacker must control DNS for these attacks to succeed
  • Waiting eight days to disclose evidence of a specific targeted attack

I believe that the browsers, such as Mozilla, are doing the best that they can in some ways but the lack of immediate full disclosure is a major failure.

Syndicate content Syndicate content