Detecting Certificate Authority compromises and web browser collusion

Thanks to Ian Gallagher, Seth Schoen, Jesse Burns, Chris Palmer, and other anonymous birds for their invaluable feedback on this writeup.

The Tor Project has long understood that the certification authority (CA) model of trust on the internet is susceptible to various methods of compromise. Without strong anonymity, the ability to perform targeted attacks with the blessing of a CA key is serious. In the past, I’ve worked on attacks relating to SSL/TLS trust models and for quite some time, I’ve hunted for evidence of non-academic CA compromise in the wild.

I’ve also looked for special kinds of cooperation between CAs and browsers. Proof of collusion will give us facts. It will also give us a real understanding of the faith placed in the strength of the underlying systems.

Does certificate revocation really work? No, it does not. How much faith does a vendor actually put into revocation, when verifiable evidence of malice is detected or known? Not much, and that’s the subject of this writing.

Last week, a smoking gun came into sight: A Certification Authority appeared to be compromised in some capacity, and the attacker issued themselves valid HTTPS certificates for high-value web sites. With these certificates, the attacker could impersonate the identities of the victim web sites or other related systems, probably undetectably for the majority of users on the internet.

I watch the Chromium and Mozilla Firefox projects carefully, because they are so important to the internet infrastructure. On the evening of 16 March, I noticed a very interesting code change to Chromium: revision 78478, Thu Mar 17 00:48:21 2011 UTC.

In this revision, the developers added X509Certificate::IsBlacklisted, which returns true if a HTTPS certificate has one of these particular serial numbers:

  • 047ecbe9fca55f7bd09eae36e10cae1e
  • d8f35f4eb7872b2dab0692e315382fb0
  • b0b7133ed096f9b56fae91c874bd3ac0
  • 9239d5348f40d1695a745470e1f23f43
  • d7558fdaf5f1105bb213282b707729a3
  • f5c86af36162f13a64f54f6dc9587c06

A comment marks the first as "Not a real certificate. For testing only." but we don’t know if this means the other certificates are or are not also for testing.

With just these serial numbers, we are not able to learn much about the certificates that Chromium now blocks. To get more information, I started the crlwatch project. Nearly every certificate contains a reference to a Certificate Revocation List (CRL). A CRL is a list of certificates that the CA has revoked for whatever reason. In theory, this means that an attacker is unable to tamper with the certificate to prevent revocation as the browser will check the CRL it finds in a certificate. In practice the attacker simply needs to tamper with the network - this is something they’re already able to do if they are performing a SSL/TLS Machine-In-The-Middle attack. Even if an attacker has a certificate, they generally are unable to modify the certificate without breaking the digital signature issued by the CA. That CA signature is what gives the certificate value to an attacker and tampering takes the attacker back to square zero. So while investigating these serials, we clearly lack the CRL distribution point in the Chrome source. However, the project that I announced on March 17th, crlwatch, was specifically written to assist in finding who issued, and potentially revoked the serial numbers in question. By matching the serial numbers found in the source for Chrome with the serial numbers of revoked certificates, we’re able to link specific serials to specific CA issuers. The more serial numbers we match in revocation lists, the higher our probability of having found the CA that issued the certificates.

About twelve hours (Thursday, March 17, 2011 | 13:00) after the above patch was pushed into source control - Google announced an important Chrome Update that involved HTTPS certificate issues.

This also is mostly uninteresting until we notice that this is not isolated to Google. Mozilla pushed out two patches of interest:

The complete changeset is semi-informative. Mozilla references a private bug in that fix that Mozilla will hopefully disclose. Similar to Chromium, the Mozilla patches create a list of certificate serial numbers that will be treated as invalid. However, the serial numbers from the Mozilla patches are different:

  • 009239d5348f40d1695a745470e1f23f43
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 72032105c50c08573d8ea5304efee8b0
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00d7558fdaf5f1105bb213282b707729a3
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 00f5c86af36162f13a64f54f6dc9587c06
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71

Thus, both Mozilla and Google shipped similar patches to their code at roughly the same time. The two browsers now have partially overlapping certificate blocklists. Here is the union of the two lists:

  • 009239d5348f40d1695a745470e1f23f43
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00d7558fdaf5f1105bb213282b707729a3
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00f5c86af36162f13a64f54f6dc9587c06
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71
  • 72032105c50c08573d8ea5304efee8b0
  • 9239d5348f40d1695a745470e1f23f43
  • b0b7133ed096f9b56fae91c874bd3ac0
  • d7558fdaf5f1105bb213282b707729a3
  • d8f35f4eb7872b2dab0692e315382fb0
  • f5c86af36162f13a64f54f6dc9587c06

Why do the browsers have these blocklists, and why don’t they have the same blocklists?

This returns me to the reason for creating crlwatch last week - I wanted to find the someones who knowingly revoked the above listed special certificates. Anyone looking from the same starting point as I did, obviously lacks the leaf certificates in question and as a result, I had to look in a rather round about manner. Thanks to the EFF’s SSL Observatory, I was able to populate the base list for crlwatch. Armed with a nearly canonical list of all CRLs, I fetched them over Tor and parsed the CRL data into human readable text. The goal was to search for the above serial numbers and to find something linkable.

This is the result from searching the crlwatch data:

  1. <br />
  2. Looking for 009239d5348f40d1695a745470e1f23f43 in parsed CRLs...<br />
  3. Looking for 00b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...<br />
  4. Looking for 00d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...<br />
  5. Looking for 00d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...<br />
  6. Looking for 00e9028b9578e415dc1a710a2b88154447 in parsed CRLs...<br />
  7. Looking for 00f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...<br />
  8. Looking for 047ecbe9fca55f7bd09eae36e10cae1e in parsed CRLs...<br />
  9. Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E<br />
  10. Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E<br />
  11. Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E<br />
  12. Looking for 392a434f0e07df1f8aa305de34e0c229 in parsed CRLs...<br />
  13. Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229<br />
  14. Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229<br />
  15. Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229<br />
  16. Looking for 3e75ced46b693021218830ae86a82a71 in parsed CRLs...<br />
  17. Match! Serial Number: 3E75CED46B693021218830AE86A82A71<br />
  18. Match! Serial Number: 3E75CED46B693021218830AE86A82A71<br />
  19. Match! Serial Number: 3E75CED46B693021218830AE86A82A71<br />
  20. Looking for 72032105c50c08573d8ea5304efee8b0 in parsed CRLs...<br />
  21. Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...<br />
  22. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  23. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  24. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  25. Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...<br />
  26. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  27. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  28. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  29. Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...<br />
  30. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  31. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  32. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  33. Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...<br />
  34. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  35. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  36. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  37. Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...<br />
  38. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />
  39. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />
  40. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />

Huzzah! It appears that we've found a few matches!

Here are the three matching files in human readable format:

Matching entries in the list look like this:

  1. <br />
  2. Serial Number: 392A434F0E07DF1F8AA305DE34E0C229<br />
  3. Revocation Date: Mar 15 20:15:38 2011 GMT<br />

An interesting note is that this date is a bit earlier than the above patches. The CA knew to revoke it on March 15th and the above patches were worked into software a few days later. If the attacker was targeting specific users, the damage to those users may have already been inflicted.

All three of the CRLs in question belong to the same CA:

  1. <br />
  2. issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST<br />
  3. Network/OU=<a href="
  4. [/geshifilter-code" rel="nofollow"><br />
  5. [/geshifilter-code</a>]</p>
  6. <p>This appears to be a reseller or something similar for the <a href="" rel="nofollow">Comodo CA company</a> when the trust chain for USERTRUST is inspected:</p>
  7. <p>[geshifilter-code]<br />
  8. CN = COMODO High Assurance Secure Server CA<br />

We appear to have no initial matches for the following Mozilla specific serials from the data that I gathered during the initial crlwatch data population run:

  • 009239d5348f40d1695a745470e1f23f43
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00d7558fdaf5f1105bb213282b707729a3
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00f5c86af36162f13a64f54f6dc9587c06

Those serial numbers appear to not match, right? Nope. Mozilla appears
to deal with certificate serial numbers in a slightly different manner - Chrome does the same internally but Mozilla exposes a weird quirk of certificate encoding directly in the source. The human readable data does not contain this quirk. Thus if you remove the prefix of ‘00’ from those serial numbers and search for the sixteen byte values, we find what we'd expect:

  1. <br />
  2. Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...<br />
  3. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  4. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  5. Match! Serial Number: 9239D5348F40D1695A745470E1F23F43<br />
  6. Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...<br />
  7. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  8. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  9. Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0<br />
  10. Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...<br />
  11. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  12. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  13. Match! Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  14. Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...<br />
  15. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  16. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  17. Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0<br />
  18. Looking for e9028b9578e415dc1a710a2b88154447 in parsed CRLs...<br />
  19. Match! Serial Number: E9028B9578E415DC1A710A2B88154447<br />
  20. Match! Serial Number: E9028B9578E415DC1A710A2B88154447<br />
  21. Match! Serial Number: E9028B9578E415DC1A710A2B88154447<br />
  22. Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...<br />
  23. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />
  24. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />
  25. Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06<br />

Here's sample from one of those CRLs:

  1. <br />
  2. Serial Number: D7558FDAF5F1105BB213282B707729A3<br />
  3. Revocation Date: Mar 15 20:15:26 2011 GMT<br />

Ironically, after all of this work, the Mozilla patch also leaks the CA name and confirmed my suspicions without question.

In the end, when the lists are merged, we find eleven certificates with two certificates probably acting as testing certificates for the two vendors involved:

  • 077a59bcd53459601ca6907267a6dd1c
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71
  • 72032105c50c08573d8ea5304efee8b0
  • 9239d5348f40d1695a745470e1f23f43
  • b0b7133ed096f9b56fae91c874bd3ac0
  • d7558fdaf5f1105bb213282b707729a3
  • d8f35f4eb7872b2dab0692e315382fb0
  • e9028b9578e415dc1a710a2b88154447
  • f5c86af36162f13a64f54f6dc9587c06

This is evidence of a rather serious event and one that cannot be ignored. If I had to make a bet, I'd wager that an attacker was able to issue high value certificates, probably by compromising USERTRUST in some manner, this was discovered sometime before the revocation date, each certificate was revoked, the vendors notified, the patches were written, and binary builds kicked off - end users are probably still updating and thus many people are vulnerable to the failure that is the CRL and OCSP method for revocation. Even after users update, I'm guessing they may be unequally protected. Mozilla and other browsers should force OCSP verification by default as part of their next release and remove CAs that are unable to handle this requirement. Users of Mozilla Firefox that are concerned about this issue should enable security.OCSP.require in the about:config dialog. The surveillance concerns of enabling OCSP are serious - a CA learns what sites you’re visiting. However, they are nullified by the fact that OCSP checking is enabled by default on Firefox at least; it simply doesn’t provide any security gains for the end user because when it fails, it fails open!

I contacted both Google and Mozilla (bug #643056) for comment after discovering the above data. Mozilla expressed some concern about disclosure and I offered to embargo this document until Tuesday March 22nd, the launch day for Firefox 4. They agreed and I kept this under my hat. After discussions between Comodo and Microsoft, passed on to me by Mozilla, the embargo was to be extended until Wednesday, March 23rd. This extension was ostensibly to ensure that Microsoft would be able to ship their Internet Explorer mitigation pack. After further discussion, Mozilla pushed their blog post about the issue and I now consider the embargo lifted. Google has already shipped a fix to users. Install the latest Firefox to get a patch, if you haven't already. A Tor Browser update is in the works and will be available soon.

Mozilla offered some additional information and disclosed that was one of the certificates acquired by the attacker. In total, nine certificates were acquired. Seven were uniquely named. Two of the certificates were re-issued for a previously issued host name. One certificate was issued for "global trustee" rather than a valid host name. With testing certificates in the list, we have a good accounting of the certificates found in the source code of each browser. Google clarified their discrepancy with the list, acknowledged the duplicate serial mistake and issued subsequent patches. Saving for test hosts, the lists are now identical.

If I had to guess at sites, I'd speculate that Facebook, Skype, Google, Microsoft, Mozilla, and others are worthy of targeting. Comodo should disclose this information and clear up this speculation with very clear information about who was targeted.

Both vendors expressed that the CA in question had done something quite remarkable by disclosing this compromise. The incentives may not be in the favor of the CA for disclosure. Many CAs may fall to similar attacks and simply refuse to disclose. Hopefully crlwatch will provide us with meaningful data regarding revocation events. The EFF and The Tor Project are working on solutions for detecting anomalies in certificates found in the wild. Still, some CAs may simply be unaware of compromises or unwilling to revoke for fear of detection.

Are all other browsers deploying similar countermeasures? Thanks to the free software nature of Firefox and Chrome, we have an answer for at least two projects. I wish that we could say the same for the rest of the browser world. One may assume that the CA in question did their best to contact all impacted vendors and targets.

Comodo has not yet revealed the extent of the compromise to the public - were their signing keys in a hardware security module? How many certificates in total were ever issued by this specific signing key? Wouldn't it be best to remove the specific signing keys from all trust roots to be extra careful given the stakes? Who exactly did they deem important enough for disclosure? The Tor Project, which ships the Tor Browser Bundle was not notified. Clearly some groups are being left out of the loop and this is where even a single attack really causes the entire CA trust model to fall apart.

Comodo should release the full certificates to the internet as well as all of the details relating to the attack. Mozilla and other browsers should open their bug reports, explain their process and lay out a path forward where we won't have to repeat this entire process again.

There is some suspicion that this action was taken by a state level adversary and there are some specific states that have been named. I’ll leave further speculation about which nation states may be involved to others. The mere fact that the web’s system of trust relies on an all or nothing property should be enough of a cause for alarm. It’s obvious that this has been and will continue to be exploited.

The impact on other cryptographic systems, such as S/MIME signatures and other cryptographic systems secured by CAs is entirely un-discussed. Quite seriously, when a CA is compromised, it will impact a great deal more than the web; users of email systems (SMTP, IMAP, POP,etc), Jabber servers, and any other SSL/TLS enabled systems are all at risk. Blocking specific serial numbers or relying on flawed, provably broken methods of revocation will simply not cut it anymore. When the actual protection mechanisms are not enforced, there is little hope of end users being protected.

This should serve as a wake up call to the internet. We need to research, build, and share new methods for ensuring trust, identity, authenticity, and confidentiality on the internet. Proposals such as DANE, CAA, HASTLS, and Monkeysphere are steps in the right direction but they face an uphill battle from entrenched economic interests.

Certification Authorities may continue to provide a piece of the puzzle but it’s high time we ensure that they’re not the alpha and the omega, anymore.

Comodo has issued a statement confirming everything that I've said and more. They believe that this was a targeted attack by a state level actor and they have named Iran as the country they suspect. Mozilla has now opened the bug reports about the issue to the public. Microsoft has now disclosed their report as well.

In the details of their statement we have a confirmation that they have the ability to monitor and thus surveille people who wish to know if certificates are valid.

Comodo also clearly demonstrate a mis-understanding - they believe that checks for revocation are proof positive that certificates are being used. They need to read and understand why this is not true.

The browsers have dropped the ball and they have chosen to fail open in nearly every single case; an attacker who is able to MITM SSL/TLS will also MITM the OCSP/CRL requests. Moxie's sslstrip demonstrated that an attacker would do this automatically and his software has done this for OCSP in public since 2009. Mozilla did not fix this issue at the time and they have once again punted on the issue. An even lower tech attack is possible and it's why revocation does not work: By returning a HTTP 500 error, the browser will the continue on as if revocation checks showed the certificate to be perfectly fine.

The browsers chose a user privacy invasive stance without the user protecting security properties. They did this because they claim that CAs are unable to provide working OCSP/CRL systems for request handling. This is a fair claim if true but it must not stand any longer. If the CA cannot provide even a basic level of revocation, it's clearly irresponsible to ship that CA root in a browser. Browsers should give insecure CA keys an Internet Death Sentence rather than expose the users of the browsers to known problems.

It's probably the case that Mozilla and other browsers should write a secure, caching OCSP server for use when a CA has a failure. It should probably be run by a neutral third party such as the EFF with a strong user privacy stance. This would only serve as a temporary fix and until Browsers get their act together, users are doomed anyway.

OCSP stapling does not fix this issue. The browsers treat revocation errors as soft errors and a MITM is deadly for revocation. The browsers believe they have to treat them as soft errors because the CAs are failing to do their job properly and are almost entirely unaccountable. The browsers are failing users by refusing to hold CAs to account. If OCSP and CRL failures mean the internet doesn't work, we need to create alternatives and not simply sweep these issues under the rug for later analysis. Browsers should hard fail on certificate revocation errors.

Comodo has further failed by:

  • Failing to produce further information about those certificates
  • Selective disclosure to "principal browsers and domain owners"
  • Failing to disclose what sub-CA/intermediate root actually did the signing
  • Believing that the attacker must control DNS for these attacks to succeed
  • Waiting eight days to disclose evidence of a specific targeted attack

I believe that the browsers, such as Mozilla, are doing the best that they can in some ways but the lack of immediate full disclosure is a major failure.


March 29, 2011


Certificate information of 72032105C50C08573D8EA5304EFEE8B0 is the followings.


Subject:,OU=Comodo Trial SSL,OU=TEST USE ONLY - NO WARRANTY ATTACHED,OU=For Testing Purposes Only,O=Kai Engert,STREET=Test Street,L=Test City,ST=Test State,postalCode=12345,C=DE
CN=UTN-USERFirst-Hardware,OU=,O=The USERTRUST Network,L=Salt Lake City,ST=UT,C=US
Serial Number:
Issuer DER Base64:
Serial DER Base64:
Serial DER as C source:
{ 16, "\x72\x03\x21\x05\xc5\x0c\x08\x57\x3d\x8e\xa5\x30\x4e\xfe\xe8\xb0" }
Fingerprint (MD5):
Fingerprint (SHA1):


March 29, 2011


Revocation date of 9 certificates are the followings.

Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E
Revocation Date: Mar 15 19:04:03 2011 GMT
Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Revocation Date: Mar 15 19:04:24 2011 GMT
Serial Number: E9028B9578E415DC1A710A2B88154447
Revocation Date: Mar 15 19:05:26 2011 GMT
Serial Number: 9239D5348F40D1695A745470E1F23F43
Revocation Date: Mar 15 20:15:20 2011 GMT
Serial Number: D7558FDAF5F1105BB213282B707729A3
Revocation Date: Mar 15 20:15:26 2011 GMT
Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Revocation Date: Mar 15 20:15:38 2011 GMT
Serial Number: 3E75CED46B693021218830AE86A82A71
Revocation Date: Mar 15 20:15:47 2011 GMT
Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Revocation Date: Mar 15 20:16:03 2011 GMT
Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Revocation Date: Mar 15 20:19:04 2011 GMT


March 30, 2011


Shouldn't people update their Namoroka (modified Firefox shipped with TOR) to detect these certs?


April 01, 2011


I don't understand sth. Why has the industry chosen to 'ignore' returned errors on the crl queries? Are there a lot of false positives? And is it i.a. a calculated risk they are willing to take?


April 05, 2011


> Why has the industry chosen to 'ignore' returned errors on the crl queries?

It's easy to guess why : end user experience! Joe User wants to get to that site and could care less about faked certs. Browser vendors chose to give access anyway rather than risk the user blame browser.



April 05, 2011


Why don't browsers just check and match domain name/ip registration details and present them to users for approval/refusal?


April 10, 2011


This is going to be a long post, but there are a number of things that are relevant to the discussion and some things I just want to get off my chest (yet some details I can't disclose, not for loyalty to an entity but for fear of that entity knowing my identity). Some of this is in response to comments, of which there are too many to quote

The SSL/TLS model used by HTTPS (browsers), and a number of other protocols, is fundamentally flawed from multiple perspectives. Some have been mentioned in the article, so I will just add onto the list.

First big problem is the typing of encryption and authentication. The fact that I can't encrypt any content without a cert is a major problem. Sure you can use a self-signed cert, or make your own CA, but that causes browsers to throw a fit and is a bad user experience. You can add your cert to the trusted list, but that doesn't scale. Some will argue that encryption is worthless without trust, but I disagree. Rather, it goes the other way. If I need to trust the other party, then surely I want privacy (encryption), but just because I want privacy doesn't mean I have to trust the other party. There are a variety of reasons to want to encrypt more traffic with sites which are fairly anonymous. Many more services would offer an encrypted version without validation if the two were not tied. Alas, currently it is infeasible to do so without spending some cash on a cert, and the reality is that many free projects simply don't have that money to spend (or would rather not fund corrupt CAs for ideological reasons).

Some have stated that there's only validation in EV. That is wrong in principle, EV is extended validation, meaning it extended the validation of standard certs, not that there is none. The entire basis of the certificate is that is proves who you are and to do that there must be validation. Unfortunately, that validation by the commercial CAs is very weak. They actually turned this weakness into part of the marketing of EV certs to organizations that can afford it, but the only real step it adds is necessity of forking over thousands of dollars which, which increasing profits massively, really only stops the casual attacks wherein people obtain certs to show how bad the CAs are (e.g. the two certs from VeriSign a decade ago, the cert from comodo a few years ago).

The CAs are business whose sole purpose is to make money. They say we should trust them, but why? They put the minimal effort into verification because that costs money, reducing profits. They put minimal resources into running reliable CRL and OCSP services, because doing so cuts into profits. They allow resellers with lax rules because more certificates sold means more income, and more stringent checks means fewer certs sold. These companies do not have out security interests in mind, and we know we cannot trusty them, so why do we continue to trust them? Just as I don't trust them to check identity, I don't trust them to protect their keys, to disclose breaches, or to correctly identify the source of problems. I notice I;m not the only one who thinks naming Iran is a little too convenient. I'm actually somewhat surprised they didn't name Libya as part of the crusade to justify yet another war.

The browser vendors also cannot be trusted. They put as many CAs in their root list as possible to make a better user experience. All of them put the most users as top priority, and anything that harms the user experience (for the average non-techie user) is seen as a negative. That is both why they have the default to allow on failure conditions, as well as every CA they can come up with in root list. Worse, the browsers tend to use a union of their list and the OS list. Take a quick look through, and not only will you find the names we know like VeriSign and Comodo, but others that frankly have no reason to be there. How many of you knew AOL and Dell had CAs that your browser trusts? How many are surprised to see a CA from China in their list? Some time ago I interviewed with a certain fruity company for a security manager position and the question came up of who are you trusting when you see the lock icon in your browser. They wanted an answer along the lines of trusting the site, the CA, and everyone in the chain between them involved in issuing the cert. My answer was also the browser vendor, the OS vendor, and anyone who has access (legitimate or not) to add certificates to your root store. That and some other answers didn't help me land that job, but I didn't exactly want it either, I was just playing along because I was intrigued about the existence and purpose of the position as well as why they were calling me for it.

In a past phase of life, I worked on some projects for the US govt involving securing their communications in a mobile environment. Suffice to say that their requirements worked well to negate the effectiveness of their security. In part because certain things that should have been done were not, and in part because the overall combination of reqs made the user experience so bad that many are bound to actively work around it. One thing they did right was they would not validate a cert without checking the CRL. One thing they did wrong was disallow CRL caching (even brief), so the list had to be fetched every time, which is very slow over a public mobile phone network when its several megabytes. By nature, the CRL only grows, it can never shrink. Another problem was no OCSP was allowed. Why would that be? Not because it had shortcomings, but it created a trail that could be audited, and while they were careful to include plenty of downward checks, there was careful avoidance of upward checks. This system is used both by ground troops and by command, extending up to the Chief Executive of the armed forces. Suffice to say, the higher ups don't always want the same scrutiny applied to their activities as they apply to those below. And yes, it all runs on Windows, both the clients and servers, as well as proxies, only some critical routers use Linux because not much else could handle the role.

While working on that project, I had to obtain a certificate for S/MIME use to protect communication in regards to the project. As I was not a member of any part of the govt but an outside contractor, I did not have a .gov address and could not get a certificate from their CA. Instead, I was directed to an ECA (External Certificate Authority), which is a party that has been trusted to issue certificates expressly for the purpose of communicating with certain govt branches. If you have such a certificate, there is an assumption you have some business in sending emails to certain people and they are more likely to respond, so having one when you shouldn't could be abused. The process to get one was much like any other 'high security' certificate; fill out paperwork, include copies of ID (low res black and white xerox), include letter of employment on company letterhead (as if they know what that should look like), get it all notarized (done by the company employed notary that stamps anything delivered to her desk), include payment and send it all off by regular post. The one extra check the ECA does beyond their normal validation is to check if the company is on the list of those who have active projects and thus should be allowed to obtains certs for their employees. A couple weeks after I mailed off the application, I got a call on my office phone from someone at the ECA. He said that everything checked out fine except that out company could not be found on the list from the DoD (I hope they don't actually have the full list, but rather just query it and got back a no response). I told him "Well, we supply to the DoD and some some other departments, but the project which we are actually doing under contract for which I need this certificate originates from NSA, not DoD." His response, which I will never forget, was "Oh, the NSA! They never put anyone on the list. Its all a big secret over there." That was the end of the call and I got my certificate within a day. So if anyone with some social skills who wants to go phishing in the DoD, just remember the magic words "I"m with the NSA." because that's all you need to bypass the regular checks.

In summary, we can't trust any vendors, not the CAs, not the browsers, not the commercial OSes. I think most of us knew that, but for those who didn't, I hope I laid out some convincing points. There are some other bits that bring perspective. We knew the CAs bend for the govt, but the CAs that work in direct cooperation with the govt don't even do a reasonable job in that role. For their role with regular clients, can we expect them to anything other than a worse job? The incompetence of the govt also comes into light, but it shouldn't be news considering the leaks. I think what will be news (for some) is how easily someone so low on the chain could exploit their weaknesses. And once you are in, you are IN. There is NO security once you breach the perimeter...

So, where do we go from here? I already have separate trusted and untrusted browser. I should have a trusted workspace with deeper separation, but Qubes has some ways. There is no absolute security, but the closer you come, the less you can get done. I saw mention of some interesting things, such as peer trust model (not quite the PGP web of trust) to apply after distrusting CAs by default, which I will have to look into. One thing I think we need is an open, trusted CRL aggregator to collect the CRLs from as many CAs as possible (at minimum all those most are default to trust) and make that available for local caching. Another necessity is a local OCSP run from that aggregate CRL to answer questions without leaving a trail outside the locally trusted network. A hosted version of that could be run by and open and trustworthy party as a convenience for those with less strict privacy requirements. As it is, I cannot use public OCSP and leave a record that negates all the effort of using NoScript, AdBlock, cookie blocking, Tor, etc. I can pull the CRLs myself and cache them locally, but a local OCSP would greatly simplify the process as currently it requires catching the CRL requests and doing so comprehensively is non-trivial.

That is still just a stopgap to reduce the problem with CAs. First, we need encryption by default without regard to the partner. In the longer term, we need a replacement for CAs. Neither the current CA system nor the PGP web of trust is sufficient for all cases. The most obvious solution is a hybrid model. At a minimum, more than one CA should have to sign an intermediate and more than one intermediate should have to sign a domain's/site's cert to make it possibly trusted. Further, sites could mark each other trusted, but that should not be the only source of trust. The exact details of how to do this greatly overlaps with a much longer running discussion started (or continuing) some months ago with serious discussion about the future of DNS following the abuses of the US govt. I would not be surprised to see the govt abusing root CAs with greater ease than they have abused DNS recently. Notice that they don't even have to use a commercial CA to forge any site, if you use a commercial OS you already have certs for govt owned CAs in your roots.


April 20, 2011


@ ioerror,

I just came across interesting blog post(s) [1, 2 and worthwhile is 3] on this story and possible missteps (speculation) by the EFF while looking to Google Analytics Opt-Out Firefox Add-on[4] vs the use of Adblock Plus and GreaseMonkey[5] to block Google Analytics[6]. I wonder if you have read the blog post(s) [1,2] and your comments/thoughts, thanks!

[1] "Did the EFF speculate on Comodogate?"…

[2] "Comodogate part 2"

[3] "Comodogate"

[4] Google Analytics Opt-out Browser Add-on

[5] ""

[6] "Google Analytics Opt Out? Not really."…


April 21, 2011


What are the relevant privacy issues when using OSCP in Firefox with Tor? Likewise, what are the privacy issues with using the Firefox Add-on Perspectives with Tor?

I do wish to increase SSL cert security but I worry very much about the privacy issues involved with OSCP and Perspectives ...

Lastly, what is the word on OCSP stapling? Can it be used with Firefox and Tor? As I understand it, OSCP stapling mitigates some (most/all?) privacy issues with OSCP; no? However, there are no currently available OSCP stapling servers that can be used with Firefox, only OSCP servers; no?


October 26, 2011


Interestingly, today, 2011-10-26, the Certificate Patrol plugin in
my browser noticed that Google replaced at least one of its
certificates out of schedule.

My browser shows that the old certificate was valid from 2011-09-05
06:07:38 GMT to 2012-09-05 06:17:38 GMT. The new one is shown as
valid from 2011-20-09 06:08:21 GMT to 2012-09-20 06:18:21
GMT. Normally, certificates are replaced a short time before they
expire, for example a few weeks before. This replacement happened
about a year before expiration, about two weeks after the previous
update, so it appears to have been replaced out of schedule.

Strangely, the *old* certificate was stored by my browser on
2011-10-05, about another two weeks *after* the validity start of
the *new* certificate.

To me, it appears that the old and new certificates essentially
differ in the public keys contained in the certificates. The
private keys associated with this public key are owned by
"*". Other differences such as the serial number are
to be expected.

That private key was replaced. Why? Together with the certificate
itself (which is public), it can be used to impersonate servers in
the domain in question ( Maybe it was in danger of not
being private any more?

The old cert's SHA1 fingerprint was:
Subject Key Identifier:

The new cert's SHA1 is:
Subject Key Identifier:

Here's anoother replacement of the same sort,
this time for

The old cert:
Valid from: 2011-09-20 08:07:19
SHA1 fingerprint:
Subject Key Identifier:
e0 18 c3 8e 7f 2f da 55 9e 50 54 27 a9 2b 34 06 06 c8 72 b5
-- The same as the new certificate valid from 2011-09-20 for

The new cert:
Valid from: 2011-11-10 08:49:06
Subject Key Identifier:
ed d7 71 f8 e2 55 ae ac 2c 70 5d f7 10 9d 83 cd 97 0b ca 2c

Another explanation would be that they rotate keys around their sites. ;-)

And another replacement, encountered at, and,
valid for *, replacing

SHA-1: 3A:26:9A:89:35:8C:99:2E:51:D8:AC:D5:9D:A8:4B:A8:F5:5F:73:9A
issued on 2011-11-10 08:48:51


SHA-1: 8F:C0:79:E8:14:77:7F:68:8B:A4:C8:07:D9:BD:67:D6:2A:F7:1A:EB
issued on 2011-11-21 04:59:22,

which is again *before* Certificate Patrol first stored the older one,
on 2011-11-24 12:58:52 .


December 22, 2011


Yet another replacement, encountered at,
valid for *, replacing the new one described above on October 26th, 2011,

SHA-1: 91:37:CB:62:F5:F4:E4:73:99:A7:18:BD:5F:83:69:D0:0D:AE:90:7D
issued on 2011-09-20 08:08:21


SHA-1: 4D:31:BC:75:D6:BF:D8:5C:63:1D:66:61:CF:36:19:E0:7F:D8:12:D4
issued on 2011-11-21 05:01:16 .

In this case, the old certificate was stored by Certificate Patrol
on 2011-10-26 16:06:06,before the new certificate was issued, as is to be expected.


March 16, 2012



Thank you four your nice writing on Detecting Certificate Authority compromises and web browser collusion .