Detecting Certificate Authority compromises and web browser collusion

Thanks to Ian Gallagher, Seth Schoen, Jesse Burns, Chris Palmer, and other anonymous birds for their invaluable feedback on this writeup.

The Tor Project has long understood that the certification authority (CA) model of trust on the internet is susceptible to various methods of compromise. Without strong anonymity, the ability to perform targeted attacks with the blessing of a CA key is serious. In the past, I’ve worked on attacks relating to SSL/TLS trust models and for quite some time, I’ve hunted for evidence of non-academic CA compromise in the wild.

I’ve also looked for special kinds of cooperation between CAs and browsers. Proof of collusion will give us facts. It will also give us a real understanding of the faith placed in the strength of the underlying systems.

Does certificate revocation really work? No, it does not. How much faith does a vendor actually put into revocation, when verifiable evidence of malice is detected or known? Not much, and that’s the subject of this writing.

Last week, a smoking gun came into sight: A Certification Authority appeared to be compromised in some capacity, and the attacker issued themselves valid HTTPS certificates for high-value web sites. With these certificates, the attacker could impersonate the identities of the victim web sites or other related systems, probably undetectably for the majority of users on the internet.

I watch the Chromium and Mozilla Firefox projects carefully, because they are so important to the internet infrastructure. On the evening of 16 March, I noticed a very interesting code change to Chromium: revision 78478, Thu Mar 17 00:48:21 2011 UTC.

In this revision, the developers added X509Certificate::IsBlacklisted, which returns true if a HTTPS certificate has one of these particular serial numbers:

  • 047ecbe9fca55f7bd09eae36e10cae1e
  • d8f35f4eb7872b2dab0692e315382fb0
  • b0b7133ed096f9b56fae91c874bd3ac0
  • 9239d5348f40d1695a745470e1f23f43
  • d7558fdaf5f1105bb213282b707729a3
  • f5c86af36162f13a64f54f6dc9587c06

A comment marks the first as "Not a real certificate. For testing only." but we don’t know if this means the other certificates are or are not also for testing.

With just these serial numbers, we are not able to learn much about the certificates that Chromium now blocks. To get more information, I started the crlwatch project. Nearly every certificate contains a reference to a Certificate Revocation List (CRL). A CRL is a list of certificates that the CA has revoked for whatever reason. In theory, this means that an attacker is unable to tamper with the certificate to prevent revocation as the browser will check the CRL it finds in a certificate. In practice the attacker simply needs to tamper with the network - this is something they’re already able to do if they are performing a SSL/TLS Machine-In-The-Middle attack. Even if an attacker has a certificate, they generally are unable to modify the certificate without breaking the digital signature issued by the CA. That CA signature is what gives the certificate value to an attacker and tampering takes the attacker back to square zero. So while investigating these serials, we clearly lack the CRL distribution point in the Chrome source. However, the project that I announced on March 17th, crlwatch, was specifically written to assist in finding who issued, and potentially revoked the serial numbers in question. By matching the serial numbers found in the source for Chrome with the serial numbers of revoked certificates, we’re able to link specific serials to specific CA issuers. The more serial numbers we match in revocation lists, the higher our probability of having found the CA that issued the certificates.

About twelve hours (Thursday, March 17, 2011 | 13:00) after the above patch was pushed into source control - Google announced an important Chrome Update that involved HTTPS certificate issues.

This also is mostly uninteresting until we notice that this is not isolated to Google. Mozilla pushed out two patches of interest:

The complete changeset is semi-informative. Mozilla references a private bug in that fix that Mozilla will hopefully disclose. Similar to Chromium, the Mozilla patches create a list of certificate serial numbers that will be treated as invalid. However, the serial numbers from the Mozilla patches are different:

  • 009239d5348f40d1695a745470e1f23f43
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 72032105c50c08573d8ea5304efee8b0
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00d7558fdaf5f1105bb213282b707729a3
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 00f5c86af36162f13a64f54f6dc9587c06
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71

Thus, both Mozilla and Google shipped similar patches to their code at roughly the same time. The two browsers now have partially overlapping certificate blocklists. Here is the union of the two lists:

  • 009239d5348f40d1695a745470e1f23f43
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00d7558fdaf5f1105bb213282b707729a3
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00f5c86af36162f13a64f54f6dc9587c06
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71
  • 72032105c50c08573d8ea5304efee8b0
  • 9239d5348f40d1695a745470e1f23f43
  • b0b7133ed096f9b56fae91c874bd3ac0
  • d7558fdaf5f1105bb213282b707729a3
  • d8f35f4eb7872b2dab0692e315382fb0
  • f5c86af36162f13a64f54f6dc9587c06

Why do the browsers have these blocklists, and why don’t they have the same blocklists?

This returns me to the reason for creating crlwatch last week - I wanted to find the someones who knowingly revoked the above listed special certificates. Anyone looking from the same starting point as I did, obviously lacks the leaf certificates in question and as a result, I had to look in a rather round about manner. Thanks to the EFF’s SSL Observatory, I was able to populate the base list for crlwatch. Armed with a nearly canonical list of all CRLs, I fetched them over Tor and parsed the CRL data into human readable text. The goal was to search for the above serial numbers and to find something linkable.

This is the result from searching the crlwatch data:

  1. <br />
  2. Looking for 009239d5348f40d1695a745470e1f23f43 in parsed CRLs...<br />
  3. Looking for 00b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...<br />
  4. Looking for 00d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...<br />
  5. Looking for 00d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...<br />
  6. Looking for 00e9028b9578e415dc1a710a2b88154447 in parsed CRLs...<br />
  7. Looking for 00f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...<br />
  8. Looking for 047ecbe9fca55f7bd09eae36e10cae1e in parsed CRLs...<br />
  9. Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E<br />
  10. Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E<br />
  11. Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E<br />
  12. Looking for 392a434f0e07df1f8aa305de34e0c229 in parsed CRLs...<br />
  13. Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229<br />
  14. Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229<br />
  15. Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229<br />
  16. Looking for 3e75ced46b693021218830ae86a82a71 in parsed CRLs...<br />
  17. Match! Serial Number: 3E75CED46B693021218830AE86A82A71<br />
  18. Match! Serial Number: 3E75CED46B693021218830AE86A82A71<br />
  19. Match! Serial Number: 3E75CED46B693021218830AE86A82A71<br />
  20. Looking for 72032105c50c08573d8ea5304efee8b0 in parsed CRLs...<br />
  21. Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...<br />
  22. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  23. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  24. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  25. Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...<br />
  26. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  27. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  28. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  29. Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...<br />
  30. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  31. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  32. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  33. Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...<br />
  34. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  35. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  36. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  37. Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...<br />
  38. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />
  39. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />
  40. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />

Huzzah! It appears that we've found a few matches!

Here are the three matching files in human readable format:

Matching entries in the list look like this:

  1. <br />
  2. Serial Number: 392A434F0E07DF1F8AA305DE34E0C229<br />
  3. Revocation Date: Mar 15 20:15:38 2011 GMT<br />

An interesting note is that this date is a bit earlier than the above patches. The CA knew to revoke it on March 15th and the above patches were worked into software a few days later. If the attacker was targeting specific users, the damage to those users may have already been inflicted.

All three of the CRLs in question belong to the same CA:

  1. <br />
  2. issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST<br />
  3. Network/OU=<a href="
  4. [/geshifilter-code" rel="nofollow"><br />
  5. [/geshifilter-code</a>]</p>
  6. <p>This appears to be a reseller or something similar for the <a href="" rel="nofollow">Comodo CA company</a> when the trust chain for USERTRUST is inspected:</p>
  7. <p>[geshifilter-code]<br />
  8. CN = COMODO High Assurance Secure Server CA<br />

We appear to have no initial matches for the following Mozilla specific serials from the data that I gathered during the initial crlwatch data population run:

  • 009239d5348f40d1695a745470e1f23f43
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00d7558fdaf5f1105bb213282b707729a3
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00f5c86af36162f13a64f54f6dc9587c06

Those serial numbers appear to not match, right? Nope. Mozilla appears
to deal with certificate serial numbers in a slightly different manner - Chrome does the same internally but Mozilla exposes a weird quirk of certificate encoding directly in the source. The human readable data does not contain this quirk. Thus if you remove the prefix of ‘00’ from those serial numbers and search for the sixteen byte values, we find what we'd expect:

  1. <br />
  2. Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...<br />
  3. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  4. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  5. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  6. Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...<br />
  7. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  8. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  9. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  10. Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...<br />
  11. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  12. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  13. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  14. Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...<br />
  15. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  16. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  17. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  18. Looking for e9028b9578e415dc1a710a2b88154447 in parsed CRLs...<br />
  19. Match! Serial Number: E9028B9578E415DC1A710A2B88154447<br />
  20. Match! Serial Number: E9028B9578E415DC1A710A2B88154447<br />
  21. Match! Serial Number: E9028B9578E415DC1A710A2B88154447<br />
  22. Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...<br />
  23. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />
  24. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />
  25. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />

Here's sample from one of those CRLs:

  1. <br />
  2. Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  3. Revocation Date: Mar 15 20:15:26 2011 GMT<br />

Ironically, after all of this work, the Mozilla patch also leaks the CA name and confirmed my suspicions without question.

In the end, when the lists are merged, we find eleven certificates with two certificates probably acting as testing certificates for the two vendors involved:

  • 077a59bcd53459601ca6907267a6dd1c
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71
  • 72032105c50c08573d8ea5304efee8b0
  • 9239d5348f40d1695a745470e1f23f43
  • b0b7133ed096f9b56fae91c874bd3ac0
  • d7558fdaf5f1105bb213282b707729a3
  • d8f35f4eb7872b2dab0692e315382fb0
  • e9028b9578e415dc1a710a2b88154447
  • f5c86af36162f13a64f54f6dc9587c06

This is evidence of a rather serious event and one that cannot be ignored. If I had to make a bet, I'd wager that an attacker was able to issue high value certificates, probably by compromising USERTRUST in some manner, this was discovered sometime before the revocation date, each certificate was revoked, the vendors notified, the patches were written, and binary builds kicked off - end users are probably still updating and thus many people are vulnerable to the failure that is the CRL and OCSP method for revocation. Even after users update, I'm guessing they may be unequally protected. Mozilla and other browsers should force OCSP verification by default as part of their next release and remove CAs that are unable to handle this requirement. Users of Mozilla Firefox that are concerned about this issue should enable security.OCSP.require in the about:config dialog. The surveillance concerns of enabling OCSP are serious - a CA learns what sites you’re visiting. However, they are nullified by the fact that OCSP checking is enabled by default on Firefox at least; it simply doesn’t provide any security gains for the end user because when it fails, it fails open!

I contacted both Google and Mozilla (bug #643056) for comment after discovering the above data. Mozilla expressed some concern about disclosure and I offered to embargo this document until Tuesday March 22nd, the launch day for Firefox 4. They agreed and I kept this under my hat. After discussions between Comodo and Microsoft, passed on to me by Mozilla, the embargo was to be extended until Wednesday, March 23rd. This extension was ostensibly to ensure that Microsoft would be able to ship their Internet Explorer mitigation pack. After further discussion, Mozilla pushed their blog post about the issue and I now consider the embargo lifted. Google has already shipped a fix to users. Install the latest Firefox to get a patch, if you haven't already. A Tor Browser update is in the works and will be available soon.

Mozilla offered some additional information and disclosed that was one of the certificates acquired by the attacker. In total, nine certificates were acquired. Seven were uniquely named. Two of the certificates were re-issued for a previously issued host name. One certificate was issued for "global trustee" rather than a valid host name. With testing certificates in the list, we have a good accounting of the certificates found in the source code of each browser. Google clarified their discrepancy with the list, acknowledged the duplicate serial mistake and issued subsequent patches. Saving for test hosts, the lists are now identical.

If I had to guess at sites, I'd speculate that Facebook, Skype, Google, Microsoft, Mozilla, and others are worthy of targeting. Comodo should disclose this information and clear up this speculation with very clear information about who was targeted.

Both vendors expressed that the CA in question had done something quite remarkable by disclosing this compromise. The incentives may not be in the favor of the CA for disclosure. Many CAs may fall to similar attacks and simply refuse to disclose. Hopefully crlwatch will provide us with meaningful data regarding revocation events. The EFF and The Tor Project are working on solutions for detecting anomalies in certificates found in the wild. Still, some CAs may simply be unaware of compromises or unwilling to revoke for fear of detection.

Are all other browsers deploying similar countermeasures? Thanks to the free software nature of Firefox and Chrome, we have an answer for at least two projects. I wish that we could say the same for the rest of the browser world. One may assume that the CA in question did their best to contact all impacted vendors and targets.

Comodo has not yet revealed the extent of the compromise to the public - were their signing keys in a hardware security module? How many certificates in total were ever issued by this specific signing key? Wouldn't it be best to remove the specific signing keys from all trust roots to be extra careful given the stakes? Who exactly did they deem important enough for disclosure? The Tor Project, which ships the Tor Browser Bundle was not notified. Clearly some groups are being left out of the loop and this is where even a single attack really causes the entire CA trust model to fall apart.

Comodo should release the full certificates to the internet as well as all of the details relating to the attack. Mozilla and other browsers should open their bug reports, explain their process and lay out a path forward where we won't have to repeat this entire process again.

There is some suspicion that this action was taken by a state level adversary and there are some specific states that have been named. I’ll leave further speculation about which nation states may be involved to others. The mere fact that the web’s system of trust relies on an all or nothing property should be enough of a cause for alarm. It’s obvious that this has been and will continue to be exploited.

The impact on other cryptographic systems, such as S/MIME signatures and other cryptographic systems secured by CAs is entirely un-discussed. Quite seriously, when a CA is compromised, it will impact a great deal more than the web; users of email systems (SMTP, IMAP, POP,etc), Jabber servers, and any other SSL/TLS enabled systems are all at risk. Blocking specific serial numbers or relying on flawed, provably broken methods of revocation will simply not cut it anymore. When the actual protection mechanisms are not enforced, there is little hope of end users being protected.

This should serve as a wake up call to the internet. We need to research, build, and share new methods for ensuring trust, identity, authenticity, and confidentiality on the internet. Proposals such as DANE, CAA, HASTLS, and Monkeysphere are steps in the right direction but they face an uphill battle from entrenched economic interests.

Certification Authorities may continue to provide a piece of the puzzle but it’s high time we ensure that they’re not the alpha and the omega, anymore.

Comodo has issued a statement confirming everything that I've said and more. They believe that this was a targeted attack by a state level actor and they have named Iran as the country they suspect. Mozilla has now opened the bug reports about the issue to the public. Microsoft has now disclosed their report as well.

In the details of their statement we have a confirmation that they have the ability to monitor and thus surveille people who wish to know if certificates are valid.

Comodo also clearly demonstrate a mis-understanding - they believe that checks for revocation are proof positive that certificates are being used. They need to read and understand why this is not true.

The browsers have dropped the ball and they have chosen to fail open in nearly every single case; an attacker who is able to MITM SSL/TLS will also MITM the OCSP/CRL requests. Moxie's sslstrip demonstrated that an attacker would do this automatically and his software has done this for OCSP in public since 2009. Mozilla did not fix this issue at the time and they have once again punted on the issue. An even lower tech attack is possible and it's why revocation does not work: By returning a HTTP 500 error, the browser will the continue on as if revocation checks showed the certificate to be perfectly fine.

The browsers chose a user privacy invasive stance without the user protecting security properties. They did this because they claim that CAs are unable to provide working OCSP/CRL systems for request handling. This is a fair claim if true but it must not stand any longer. If the CA cannot provide even a basic level of revocation, it's clearly irresponsible to ship that CA root in a browser. Browsers should give insecure CA keys an Internet Death Sentence rather than expose the users of the browsers to known problems.

It's probably the case that Mozilla and other browsers should write a secure, caching OCSP server for use when a CA has a failure. It should probably be run by a neutral third party such as the EFF with a strong user privacy stance. This would only serve as a temporary fix and until Browsers get their act together, users are doomed anyway.

OCSP stapling does not fix this issue. The browsers treat revocation errors as soft errors and a MITM is deadly for revocation. The browsers believe they have to treat them as soft errors because the CAs are failing to do their job properly and are almost entirely unaccountable. The browsers are failing users by refusing to hold CAs to account. If OCSP and CRL failures mean the internet doesn't work, we need to create alternatives and not simply sweep these issues under the rug for later analysis. Browsers should hard fail on certificate revocation errors.

Comodo has further failed by:

  • Failing to produce further information about those certificates
  • Selective disclosure to "principal browsers and domain owners"
  • Failing to disclose what sub-CA/intermediate root actually did the signing
  • Believing that the attacker must control DNS for these attacks to succeed
  • Waiting eight days to disclose evidence of a specific targeted attack

I believe that the browsers, such as Mozilla, are doing the best that they can in some ways but the lack of immediate full disclosure is a major failure.


October 20, 2011

In reply to by Anonymous (not verified)




March 22, 2011


The verification process for buying a certificate is a joke with most cert vendors. All they care about is the email address registered with the domain registrar. Hijack the domain and you can order yourself all the certs you want for the domain. Next, ISVs and so-called software consultants are constantly implementing code in orgs where certificates are explicitly trusted. Most commercial software developers treat certs as a liability. PKI is totally bankrupt as a trust service.

Why do you think it's even important to validate the email address. An SSL cert is (or at least was) used for the sole purpose of making a site secure. It was never intended to assert the site owners identity, or their trustworthiness. The latter has only come to market with the launch of extended validation services.

This is not really true. Back in the "good old days" in 1994, EV-type stuff was the NORM. I got an SSL cert for Netscape Server from Verisign and it took about 2 weeks to get all the verifications in order. The truth is that CAs got greedy and loosened their processes. More than half of the sites out there get their certs issued in a matter of can set up a reseller account and get certs issued on your clients' behalf with essentially NO verification.

The system relies on honesty, and someone was dishonest. I would imagine there was no exploitation here other than someone trying to get a cert for a high-value domain and succeeding. It was like brute force through multiple vendors.

Encryption is useless without authentication: what is the point of keeping things secret so only your peer can read it if you don't know who the peer is?

I think a lot of people would take issue with your statement that TLS is not meant to assert identity. This is why WebAppSec folk argue that a login page should be on an TLS supported page as opposed to only POSTing to an TLS supported page.

TLS is for making a connection to a site secure, yes. That means you have to know you're talking to the site, and not some system in between. That in turn implies that you have to know that the cert belongs to the site, which implies validation. Drop that, and https is no better than http.

CAs have used the "we must verify who you are" argument for justifying the cost. The crypto is free, process is the only possible justification. Apparently there was only voluntary enforcement, no regulation. We had a race to the bottom and it's too late to go back.

You'd do well to research a little more before you post. CA's not only have to validate your request for a certificate - which, in my experience, is strict to the point of nitpickery, at least among the major CA's - but they also have to host a server farm to process OCSP and CRL requests, process requests to revoke certificates, and have to adhere to a large bundle of rules and regulations. In the US I believe they use FIPS, in Europe it's Directive 99/93/EC which has been worked out into ETSI TS 101 456 and ETSI TS 102 042. These specify requirements of constant internal audits and annual external audits bij ISO-certified independent bodies. Also, CA's must keep records for years in case of a legal dispute where the information is required as evidence. All this isn't free and go a good way to explaining the prices of certificates.

This, in combination with the recent compromise of some of RSA's more important systems and one other event I'm not at liberty to disclose has my subconscious working overtime, is it a pattern, or just a coincidence ..

please make ready tor application with Firefox 4 .

Comodo was also the company that previous had problems with resellers not checking domains at all - see Mozilla bug 470897. They also allow certificates for domains that don't exist, such as Mozilla bug 526560, and certificates that name companies that don't exist, bug 599856.

Mozilla, at least, refuses to enact any sort of punishment because they think users will blame Firefox when the SSL sites using those certificates stop working. Essentially, they believe they cannot do anything that might possibly actually fix the problem of... well, the CA being crap.

At least these particular changes managed to land just in time for Firefox 4; it is the last commit on the relevant branch.

That wasn’t the sort of security problem that should have been kept secret until patches were shipped — it was clear from the beginning that:

  • bad guys were already able to exploit the bogus certificates,
  • disclosing the existence of the bogus certificates would not have allowed more people to perform the attack they enabled, and
  • waiting to disclose the existence of the bogus certs only aided the ‘cyber-terrorists’ who had obtained them, by allowing the certs to be used for a longer period of time.

On their blog, I asked Mozilla why they agreed to keep this event secret:

But I should ask you a few questions about your conduct in this matter as well:

  1. You wrote above: “Mozilla expressed some concern about disclosure and I offered to embargo this document until Tuesday March 22nd, the launch day for Firefox 4.” What concerns did they express, and why did those concerns convince you to aid in an attack by concealing the existence of these fraudulent certificates?
  2. Did you inform anyone else affiliated with The Tor Project, Inc. before offering to ‘embargo this document’, or was that decision yours alone?
  3. Do you believe that Mozilla would have included Comodo's root certificates as ‘trusted’ CAs in the final release of Firefox 4 if you had disclosed your discovery immediately, rather than delaying your disclosure?

Exactly! Why the embargo ?? This isn't something that should be a traditional responsible disclosure issue. Jake you of all people I assume would've understood this.

I'm sorry that I agreed to the embargo at all at this point.

To reply your your questions:

1) They expressed concerns about being able to ship a fix to Firefox - that is a brand new Firefox binary with hardcoded serials in their blocklist.
1a) We need Firefox to release before we can update the Tor Browser Bundle because our current build process on Windows does not include building Firefox from source. We probably need to change that but it is not my call.

2) I agreed to the embargo on my own - I informed people internally that I was looking for this when I started out and I informed Tor internally when I discovered it, I also again informed Tor when I agreed to the embargo.
2a) I let helix (the main Tor build engineer) know that Firefox would be releasing on Tuesday. Specifically, I explained why and what needs to happen for Tor users to be safe. See point 1a on why this is a PITA for Tor's hard working build engineer.
2b) I started advocating requiring (it's already enabled) OCSP checking from the moment that I realized it would provide a mitigation - I didn't embargo myself for mitigations, only some of the specific details.

3) Mozilla will not give the Comodo CA in question an Internet Death Sentence. We do not know the extent of the compromise and Mozilla did not put off their launch for this extremely serious issue. I think that at the very least, this was a difficult choice for Mozilla. I believe that it was a mistake to not mark the direct signing cert of the certificates in question as untrusted. The rest of the root should have a closer inspection after release, of course. None of that was done for the same reason that OCSP checking isn't required - Mozilla makes these calls and decided they know best.

I obviously disagree with these security choices because they rob the user of their autonomy; users need information to protect their activities on the internet.

I thank you for your work in this area, and also for your warning/recommendation about enabling forced OCSP checking. It's good to know, when there are so many that are trying to tear down, that some are trying to get the protection out there.

Jake did advocate/informed me to enable OCSP in the early days of these events.

I also informed a few friends in Iran. 1 of them reported that he is not able to visit sites with OCSP enabled in firefox.

However the same user 24 hours later, reported that the problem has been resolved automatically :-/

Thank you Jake!

It's funny how when I enabled OSCP in Firefox I couldn't access the EFF link you posted anymore...

Worse, EFF uses wildcard certificate.

CN = *
OU = Comodo PremiumSSL Wildcard
O = Electronic Frontier Foundation
Object Identifier (2 5 4 9) = 454 Shotwell St
L = San Francisco
ST = California
Object Identifier (2 5 4 17) = 94110
C = US

"Last week, a smoking gun came into sight: A Certification Authority appeared to be compromised in some capacity, and the attacker issued themselves valid HTTPS certificates for high-value web sites. With these certificates, the attacker could impersonate the identities of the victim web sites or other related systems, probably undetectably for the majority of users on the internet."

Did they attacker gain access to the additional information required for 'extended validation'? An SSL Cert is only used to secure a site - not to say anything about the site owner. That is unless, they have been issued and EVC.

SSL has always been for both confidentiality and verifying the identity of a server. Why do you think certificates have a subject DN field if they are only for encryption and decryption?

This is obviously a conspiracy between the illuminati, browser vendors, and ioerror. We should add it to…

All software is written by the new world order. The Internet was developed by the US Dept of Defense to build a perfect tracking system. Google, Microsoft, Oracle, Verisign, RSA, Symantec are all members of the same collusion set with the NSA, and other world governments.

Disconnect now, or be subject to their whims and wrath.

I've said too much for the black helicopters approach.

I'm guessing you're british or north american. Your government lies to your face continuously and launches wars in your name. Kills, assassinates, bombs civilians, tortures people or sends them away to secret holes to be tortured by their lackeys.

How about them conspiracies?

1. He's joking
2. Which government do you think the NSA is in, exactly?

ioerror, this is a superb analysis, thanks.

this paper looks interesting in the light of this event:

There is already an implicit trust system built in that may be used in the meantime: DNS. This is what the RFC 4398 CERT resource record could help achieve.

IANA is already a trusted 3rd party. IANA runs IP, AS, and DNS address allocation. If you want to connect to TORPROJECT.ORG,, or AS1 for that matter, look no further than an implicit trust chain emanating from ARIN. The question is, is ARIN more trustworthy than a random CA that Mozilla has mated with? I believe so, and I think many others believe so as well. Let the users decide.

But alas, no implementation of RFC 4398, c. 2006, exists to date... We have all sorts of crazy iPhone apps, but no CERT RR implementation. The RFC likely needs expansive interpretation or outright modification for use in tandem with TLS and other widely used protocols. I hope some programmers that are well versed with browser plugins, libc/libresolv/windns/dnsapi/etc., and protocol considerations will get involved.

Just get something working, it has to be better than this. Otherwise, good article.

DNS-Records without DNSSEC are no replacement for even the weakest CA infrastructure. *Every* possible exploitation scenario that involves a fraudulent certificate in an HTTPS connection -- i.e. every scenario which storing certificates in the DNS would supposedly fix -- give the attacker direct (if the user was lured to the wrong site by some sort of DNS manipulation) or indirect (if the attacker is on the same network as the user or on the path from user to original site) control over the DNS and therefore the CERT records.

DNSSEC is part of DNS as of RFC 4035, c. March 2005, which predates and is accounted for in RFC 4398 CERT resource records. In addition, use of IPSec could possibly bring confidentiality to recursive queries and authentication to stub queries.

In other words, these concerns are largely moot.

No one uses DNSSEC or IPSec or IPv6.

None of them are perfect either, so that they are /designed/ to mitigate things does not mean that they actually do.

We Must Fix HTTPS:

An interesting set of slides on https model.

When Certstar got caught issuing a certificate for without doing valid checks ( people started calling out Comodo because Certstar was a Comodo reseller and Comodo shouldn’t have let that happen. That resulted in Comodo suspending that reseller’s ability to issue certifications because they were not fulfilling their proper contractual obligations to Comodo… During that process people were looking closely at Comodo trust paths, and Comodo was being nice and answering questions in forums to try and clear their name. During this questioning AddTrust came up. If you follow the thread around it reveals that AddTrust AB no longer exists as a company, but Comodo bought the certificates from them, and the rights to continue to use their root CA which bear the AddTrust name.

The history is illustrative, basically the AddTrust company was acquired by ScandTrust (out of Malmö, Sweden), which was a Swedish CA, and then Comodo purchased the AddTrust root from ScandTrust. Then Comodo also purchased UserTrust, which had four roots and those also became part of Comodo when UserTrust became a “Comodo Group Company”.

In both of the above cases the key material was removed from its original sites of operation (in Sweden and Salt Lake City, respectively) and trafserred into Comodo's data centres and backup centres.

This is interesting because defunct companies, that had been sold, transferred their key material around to different parties. I think that this is the wrong way to do this, and that the right way to do this is to revoke the certificates and the CA, but thats an expensive operation. So, I think what they do with these certs is that they actually recognize the lower value of such a CA, and what they do is is sell these certs at a cheaper cost, because they haven't been under the control of Comodo from the beginning.

They call these “Usertrust” CAs and they offer 3rd party vendors, such as Gandi to issue these certs, which are way cheaper than Comodo’s in-house generated ones.


What about phobos' surely phobos and those like him would notice the cert changed?

Can we at least start scoping SSL certs by internet domain to limit the damage? Let a root CA sign for any domain, but create a COM CA that can only sign for *.com, and let it sell a CA to google that can only sign for * and so on...
Or is this too detrimental to the business model to ever happen?

Too many questions unanswered - what about domains that don't want to maintain their own CA? Who operates the *.com?

Why the embargo ? Seems like this isn't a case for responsible disclosure because this isn't a traditional vulnerability.

Yes we could do with a better system based on gpg or something but they're struggling to get dns security right.

Untill it is right and used by all then all a website owner can do in this regard is look after his key, data once received and mail systems. It's a joke the EV and CA costs for a false sense of security which is making it take longer for a general consensus to want to find a new system. I'm glad you can get the certs accepted by browsers for free. I'd use self signed if browsers didn't say the worlds gonna end when you do. I think microsoft probably contemplated shutting your computer down the way IE reacts.

Black helicopters were mentioned, the openbsd list had a discussion recently part of which was that the government can get CAs to cooperate and can MITM the network legally. Assuming they're systems don't have gaping holes, (quite probable running cisco and M$ rubbish at great expense) and they're employees are trustworthy (hmmm), then if you can still be MITM then you likely have far greater problems than SSL.

CAs are dead
Long live, hmmm

bastion hosts

Shouldn't the browsers be able to spot this behavior fairly easily by taking "signatures" of the certs that they are presented with and comparing certs that are supposedly for the same host names? I know this is would be detecting the problem after the fact but it seems fairly reasonable, no?

A source at Microsoft says the following certs were obtained.
• (3 certificates)
• "Global Trustee"

I'm not sure what the "Global Trustee" means.

IP Address:
City: Tehran
State or Region: Tehran
Country: Iran, Islamic Republic of

1. Steal certs
2. Redirect DNS inside Iran to fake sites
3. Block OCSP / access to CRL
4. Capture logins
5. Snoop on gmail, hotmail, yahoo email boxes
6. Round up activists
7. Torture
8. Repeat

9. ???
10. PROFIT!!!

we need a international action toward Iran government

1. It's not necessarily Iran. China has been interfering with GMail lately, for example.
2. Fixing the PKI is cheaper than “international action”

Update: Comodo has published an incident report:

Does anyone know what's Safari's status is with blacklisting revoked certificates?

so we don't trust this douche-b (comodo) to release information about a huge security breach in a timely fashion but we /do/ trust him to name the "state actor" behind the attacks?? what a shocker that they named iran, the USSA's enemy du jour. i dont believe that bullshit for a second, now if u had said USSA's 51st satellite apartheid state, THEN i'd believe you. who was behind STUXNET? who is more inclined to pull off this bullshit and blame it on others (by way of deception....) stop being so fucking stupid.