Detecting Certificate Authority compromises and web browser collusion

Thanks to Ian Gallagher, Seth Schoen, Jesse Burns, Chris Palmer, and other anonymous birds for their invaluable feedback on this writeup.

The Tor Project has long understood that the certification authority (CA) model of trust on the internet is susceptible to various methods of compromise. Without strong anonymity, the ability to perform targeted attacks with the blessing of a CA key is serious. In the past, I’ve worked on attacks relating to SSL/TLS trust models and for quite some time, I’ve hunted for evidence of non-academic CA compromise in the wild.

I’ve also looked for special kinds of cooperation between CAs and browsers. Proof of collusion will give us facts. It will also give us a real understanding of the faith placed in the strength of the underlying systems.

Does certificate revocation really work? No, it does not. How much faith does a vendor actually put into revocation, when verifiable evidence of malice is detected or known? Not much, and that’s the subject of this writing.

Last week, a smoking gun came into sight: A Certification Authority appeared to be compromised in some capacity, and the attacker issued themselves valid HTTPS certificates for high-value web sites. With these certificates, the attacker could impersonate the identities of the victim web sites or other related systems, probably undetectably for the majority of users on the internet.

I watch the Chromium and Mozilla Firefox projects carefully, because they are so important to the internet infrastructure. On the evening of 16 March, I noticed a very interesting code change to Chromium: revision 78478, Thu Mar 17 00:48:21 2011 UTC.

In this revision, the developers added X509Certificate::IsBlacklisted, which returns true if a HTTPS certificate has one of these particular serial numbers:

  • 047ecbe9fca55f7bd09eae36e10cae1e
  • d8f35f4eb7872b2dab0692e315382fb0
  • b0b7133ed096f9b56fae91c874bd3ac0
  • 9239d5348f40d1695a745470e1f23f43
  • d7558fdaf5f1105bb213282b707729a3
  • f5c86af36162f13a64f54f6dc9587c06

A comment marks the first as "Not a real certificate. For testing only." but we don’t know if this means the other certificates are or are not also for testing.

With just these serial numbers, we are not able to learn much about the certificates that Chromium now blocks. To get more information, I started the crlwatch project. Nearly every certificate contains a reference to a Certificate Revocation List (CRL). A CRL is a list of certificates that the CA has revoked for whatever reason. In theory, this means that an attacker is unable to tamper with the certificate to prevent revocation as the browser will check the CRL it finds in a certificate. In practice the attacker simply needs to tamper with the network - this is something they’re already able to do if they are performing a SSL/TLS Machine-In-The-Middle attack. Even if an attacker has a certificate, they generally are unable to modify the certificate without breaking the digital signature issued by the CA. That CA signature is what gives the certificate value to an attacker and tampering takes the attacker back to square zero. So while investigating these serials, we clearly lack the CRL distribution point in the Chrome source. However, the project that I announced on March 17th, crlwatch, was specifically written to assist in finding who issued, and potentially revoked the serial numbers in question. By matching the serial numbers found in the source for Chrome with the serial numbers of revoked certificates, we’re able to link specific serials to specific CA issuers. The more serial numbers we match in revocation lists, the higher our probability of having found the CA that issued the certificates.

About twelve hours (Thursday, March 17, 2011 | 13:00) after the above patch was pushed into source control - Google announced an important Chrome Update that involved HTTPS certificate issues.

This also is mostly uninteresting until we notice that this is not isolated to Google. Mozilla pushed out two patches of interest:

The complete changeset is semi-informative. Mozilla references a private bug in that fix that Mozilla will hopefully disclose. Similar to Chromium, the Mozilla patches create a list of certificate serial numbers that will be treated as invalid. However, the serial numbers from the Mozilla patches are different:

  • 009239d5348f40d1695a745470e1f23f43
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 72032105c50c08573d8ea5304efee8b0
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00d7558fdaf5f1105bb213282b707729a3
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 00f5c86af36162f13a64f54f6dc9587c06
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71

Thus, both Mozilla and Google shipped similar patches to their code at roughly the same time. The two browsers now have partially overlapping certificate blocklists. Here is the union of the two lists:

  • 009239d5348f40d1695a745470e1f23f43
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00d7558fdaf5f1105bb213282b707729a3
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00f5c86af36162f13a64f54f6dc9587c06
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71
  • 72032105c50c08573d8ea5304efee8b0
  • 9239d5348f40d1695a745470e1f23f43
  • b0b7133ed096f9b56fae91c874bd3ac0
  • d7558fdaf5f1105bb213282b707729a3
  • d8f35f4eb7872b2dab0692e315382fb0
  • f5c86af36162f13a64f54f6dc9587c06

Why do the browsers have these blocklists, and why don’t they have the same blocklists?

This returns me to the reason for creating crlwatch last week - I wanted to find the someones who knowingly revoked the above listed special certificates. Anyone looking from the same starting point as I did, obviously lacks the leaf certificates in question and as a result, I had to look in a rather round about manner. Thanks to the EFF’s SSL Observatory, I was able to populate the base list for crlwatch. Armed with a nearly canonical list of all CRLs, I fetched them over Tor and parsed the CRL data into human readable text. The goal was to search for the above serial numbers and to find something linkable.

This is the result from searching the crlwatch data:

Looking for 009239d5348f40d1695a745470e1f23f43 in parsed CRLs...
Looking for 00b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...
Looking for 00d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...
Looking for 00d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...
Looking for 00e9028b9578e415dc1a710a2b88154447 in parsed CRLs...
Looking for 00f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...
Looking for 047ecbe9fca55f7bd09eae36e10cae1e in parsed CRLs...
Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E
Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E
Match! Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E
Looking for 392a434f0e07df1f8aa305de34e0c229 in parsed CRLs...
Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Match! Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Looking for 3e75ced46b693021218830ae86a82a71 in parsed CRLs...
Match! Serial Number: 3E75CED46B693021218830AE86A82A71
Match! Serial Number: 3E75CED46B693021218830AE86A82A71
Match! Serial Number: 3E75CED46B693021218830AE86A82A71
Looking for 72032105c50c08573d8ea5304efee8b0 in parsed CRLs...
Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06

Huzzah! It appears that we've found a few matches!

Here are the three matching files in human readable format:

Matching entries in the list look like this:

Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Revocation Date: Mar 15 20:15:38 2011 GMT

An interesting note is that this date is a bit earlier than the above patches. The CA knew to revoke it on March 15th and the above patches were worked into software a few days later. If the attacker was targeting specific users, the damage to those users may have already been inflicted.

All three of the CRLs in question belong to the same CA:

issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST

This appears to be a reseller or something similar for the Comodo CA company when the trust chain for USERTRUST is inspected:

CN = COMODO High Assurance Secure Server CA

We appear to have no initial matches for the following Mozilla specific serials from the data that I gathered during the initial crlwatch data population run:

  • 009239d5348f40d1695a745470e1f23f43
  • 00b0b7133ed096f9b56fae91c874bd3ac0
  • 00d7558fdaf5f1105bb213282b707729a3
  • 00d8f35f4eb7872b2dab0692e315382fb0
  • 00e9028b9578e415dc1a710a2b88154447
  • 00f5c86af36162f13a64f54f6dc9587c06

Those serial numbers appear to not match, right? Nope. Mozilla appears
to deal with certificate serial numbers in a slightly different manner - Chrome does the same internally but Mozilla exposes a weird quirk of certificate encoding directly in the source. The human readable data does not contain this quirk. Thus if you remove the prefix of ‘00’ from those serial numbers and search for the sixteen byte values, we find what we'd expect:

Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs...
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Match! Serial Number: 9239D5348F40D1695A745470E1F23F43
Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs...
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Match! Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs...
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Match! Serial Number: D7558FDAF5F1105BB213282B707729A3
Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs...
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Match! Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Looking for e9028b9578e415dc1a710a2b88154447 in parsed CRLs...
Match! Serial Number: E9028B9578E415DC1A710A2B88154447
Match! Serial Number: E9028B9578E415DC1A710A2B88154447
Match! Serial Number: E9028B9578E415DC1A710A2B88154447
Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs...
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Match! Serial Number: F5C86AF36162F13A64F54F6DC9587C06

Here's sample from one of those CRLs:

Serial Number: D7558FDAF5F1105BB213282B707729A3
Revocation Date: Mar 15 20:15:26 2011 GMT

Ironically, after all of this work, the Mozilla patch also leaks the CA name and confirmed my suspicions without question.

In the end, when the lists are merged, we find eleven certificates with two certificates probably acting as testing certificates for the two vendors involved:

  • 077a59bcd53459601ca6907267a6dd1c
  • 047ecbe9fca55f7bd09eae36e10cae1e
  • 392a434f0e07df1f8aa305de34e0c229
  • 3e75ced46b693021218830ae86a82a71
  • 72032105c50c08573d8ea5304efee8b0
  • 9239d5348f40d1695a745470e1f23f43
  • b0b7133ed096f9b56fae91c874bd3ac0
  • d7558fdaf5f1105bb213282b707729a3
  • d8f35f4eb7872b2dab0692e315382fb0
  • e9028b9578e415dc1a710a2b88154447
  • f5c86af36162f13a64f54f6dc9587c06

This is evidence of a rather serious event and one that cannot be ignored. If I had to make a bet, I'd wager that an attacker was able to issue high value certificates, probably by compromising USERTRUST in some manner, this was discovered sometime before the revocation date, each certificate was revoked, the vendors notified, the patches were written, and binary builds kicked off - end users are probably still updating and thus many people are vulnerable to the failure that is the CRL and OCSP method for revocation. Even after users update, I'm guessing they may be unequally protected. Mozilla and other browsers should force OCSP verification by default as part of their next release and remove CAs that are unable to handle this requirement. Users of Mozilla Firefox that are concerned about this issue should enable security.OCSP.require in the about:config dialog. The surveillance concerns of enabling OCSP are serious - a CA learns what sites you’re visiting. However, they are nullified by the fact that OCSP checking is enabled by default on Firefox at least; it simply doesn’t provide any security gains for the end user because when it fails, it fails open!

I contacted both Google and Mozilla (bug #643056) for comment after discovering the above data. Mozilla expressed some concern about disclosure and I offered to embargo this document until Tuesday March 22nd, the launch day for Firefox 4. They agreed and I kept this under my hat. After discussions between Comodo and Microsoft, passed on to me by Mozilla, the embargo was to be extended until Wednesday, March 23rd. This extension was ostensibly to ensure that Microsoft would be able to ship their Internet Explorer mitigation pack. After further discussion, Mozilla pushed their blog post about the issue and I now consider the embargo lifted. Google has already shipped a fix to users. Install the latest Firefox to get a patch, if you haven't already. A Tor Browser update is in the works and will be available soon.

Mozilla offered some additional information and disclosed that was one of the certificates acquired by the attacker. In total, nine certificates were acquired. Seven were uniquely named. Two of the certificates were re-issued for a previously issued host name. One certificate was issued for "global trustee" rather than a valid host name. With testing certificates in the list, we have a good accounting of the certificates found in the source code of each browser. Google clarified their discrepancy with the list, acknowledged the duplicate serial mistake and issued subsequent patches. Saving for test hosts, the lists are now identical.

If I had to guess at sites, I'd speculate that Facebook, Skype, Google, Microsoft, Mozilla, and others are worthy of targeting. Comodo should disclose this information and clear up this speculation with very clear information about who was targeted.

Both vendors expressed that the CA in question had done something quite remarkable by disclosing this compromise. The incentives may not be in the favor of the CA for disclosure. Many CAs may fall to similar attacks and simply refuse to disclose. Hopefully crlwatch will provide us with meaningful data regarding revocation events. The EFF and The Tor Project are working on solutions for detecting anomalies in certificates found in the wild. Still, some CAs may simply be unaware of compromises or unwilling to revoke for fear of detection.

Are all other browsers deploying similar countermeasures? Thanks to the free software nature of Firefox and Chrome, we have an answer for at least two projects. I wish that we could say the same for the rest of the browser world. One may assume that the CA in question did their best to contact all impacted vendors and targets.

Comodo has not yet revealed the extent of the compromise to the public - were their signing keys in a hardware security module? How many certificates in total were ever issued by this specific signing key? Wouldn't it be best to remove the specific signing keys from all trust roots to be extra careful given the stakes? Who exactly did they deem important enough for disclosure? The Tor Project, which ships the Tor Browser Bundle was not notified. Clearly some groups are being left out of the loop and this is where even a single attack really causes the entire CA trust model to fall apart.

Comodo should release the full certificates to the internet as well as all of the details relating to the attack. Mozilla and other browsers should open their bug reports, explain their process and lay out a path forward where we won't have to repeat this entire process again.

There is some suspicion that this action was taken by a state level adversary and there are some specific states that have been named. I’ll leave further speculation about which nation states may be involved to others. The mere fact that the web’s system of trust relies on an all or nothing property should be enough of a cause for alarm. It’s obvious that this has been and will continue to be exploited.

The impact on other cryptographic systems, such as S/MIME signatures and other cryptographic systems secured by CAs is entirely un-discussed. Quite seriously, when a CA is compromised, it will impact a great deal more than the web; users of email systems (SMTP, IMAP, POP,etc), Jabber servers, and any other SSL/TLS enabled systems are all at risk. Blocking specific serial numbers or relying on flawed, provably broken methods of revocation will simply not cut it anymore. When the actual protection mechanisms are not enforced, there is little hope of end users being protected.

This should serve as a wake up call to the internet. We need to research, build, and share new methods for ensuring trust, identity, authenticity, and confidentiality on the internet. Proposals such as DANE, CAA, HASTLS, and Monkeysphere are steps in the right direction but they face an uphill battle from entrenched economic interests.

Certification Authorities may continue to provide a piece of the puzzle but it’s high time we ensure that they’re not the alpha and the omega, anymore.

Comodo has issued a statement confirming everything that I've said and more. They believe that this was a targeted attack by a state level actor and they have named Iran as the country they suspect. Mozilla has now opened the bug reports about the issue to the public. Microsoft has now disclosed their report as well.

In the details of their statement we have a confirmation that they have the ability to monitor and thus surveille people who wish to know if certificates are valid.

Comodo also clearly demonstrate a mis-understanding - they believe that checks for revocation are proof positive that certificates are being used. They need to read and understand why this is not true.

The browsers have dropped the ball and they have chosen to fail open in nearly every single case; an attacker who is able to MITM SSL/TLS will also MITM the OCSP/CRL requests. Moxie's sslstrip demonstrated that an attacker would do this automatically and his software has done this for OCSP in public since 2009. Mozilla did not fix this issue at the time and they have once again punted on the issue. An even lower tech attack is possible and it's why revocation does not work: By returning a HTTP 500 error, the browser will the continue on as if revocation checks showed the certificate to be perfectly fine.

The browsers chose a user privacy invasive stance without the user protecting security properties. They did this because they claim that CAs are unable to provide working OCSP/CRL systems for request handling. This is a fair claim if true but it must not stand any longer. If the CA cannot provide even a basic level of revocation, it's clearly irresponsible to ship that CA root in a browser. Browsers should give insecure CA keys an Internet Death Sentence rather than expose the users of the browsers to known problems.

It's probably the case that Mozilla and other browsers should write a secure, caching OCSP server for use when a CA has a failure. It should probably be run by a neutral third party such as the EFF with a strong user privacy stance. This would only serve as a temporary fix and until Browsers get their act together, users are doomed anyway.

OCSP stapling does not fix this issue. The browsers treat revocation errors as soft errors and a MITM is deadly for revocation. The browsers believe they have to treat them as soft errors because the CAs are failing to do their job properly and are almost entirely unaccountable. The browsers are failing users by refusing to hold CAs to account. If OCSP and CRL failures mean the internet doesn't work, we need to create alternatives and not simply sweep these issues under the rug for later analysis. Browsers should hard fail on certificate revocation errors.

Comodo has further failed by:

  • Failing to produce further information about those certificates
  • Selective disclosure to "principal browsers and domain owners"
  • Failing to disclose what sub-CA/intermediate root actually did the signing
  • Believing that the attacker must control DNS for these attacks to succeed
  • Waiting eight days to disclose evidence of a specific targeted attack

I believe that the browsers, such as Mozilla, are doing the best that they can in some ways but the lack of immediate full disclosure is a major failure.


but seriously. very interesting read Jacob!

Thank you.


Comodo has announced acquisition of, the managed domain name system services provider, and the launch of Comodo Authoritative DNS Service one day before loose the certificate.

The verification process for buying a certificate is a joke with most cert vendors. All they care about is the email address registered with the domain registrar. Hijack the domain and you can order yourself all the certs you want for the domain. Next, ISVs and so-called software consultants are constantly implementing code in orgs where certificates are explicitly trusted. Most commercial software developers treat certs as a liability. PKI is totally bankrupt as a trust service.

Why do you think it's even important to validate the email address. An SSL cert is (or at least was) used for the sole purpose of making a site secure. It was never intended to assert the site owners identity, or their trustworthiness. The latter has only come to market with the launch of extended validation services.

This is not really true. Back in the "good old days" in 1994, EV-type stuff was the NORM. I got an SSL cert for Netscape Server from Verisign and it took about 2 weeks to get all the verifications in order. The truth is that CAs got greedy and loosened their processes. More than half of the sites out there get their certs issued in a matter of can set up a reseller account and get certs issued on your clients' behalf with essentially NO verification.

The system relies on honesty, and someone was dishonest. I would imagine there was no exploitation here other than someone trying to get a cert for a high-value domain and succeeding. It was like brute force through multiple vendors.

Encryption is useless without authentication: what is the point of keeping things secret so only your peer can read it if you don't know who the peer is?

I think a lot of people would take issue with your statement that TLS is not meant to assert identity. This is why WebAppSec folk argue that a login page should be on an TLS supported page as opposed to only POSTing to an TLS supported page.

TLS is for making a connection to a site secure, yes. That means you have to know you're talking to the site, and not some system in between. That in turn implies that you have to know that the cert belongs to the site, which implies validation. Drop that, and https is no better than http.

CAs have used the "we must verify who you are" argument for justifying the cost. The crypto is free, process is the only possible justification. Apparently there was only voluntary enforcement, no regulation. We had a race to the bottom and it's too late to go back.

You'd do well to research a little more before you post. CA's not only have to validate your request for a certificate - which, in my experience, is strict to the point of nitpickery, at least among the major CA's - but they also have to host a server farm to process OCSP and CRL requests, process requests to revoke certificates, and have to adhere to a large bundle of rules and regulations. In the US I believe they use FIPS, in Europe it's Directive 99/93/EC which has been worked out into ETSI TS 101 456 and ETSI TS 102 042. These specify requirements of constant internal audits and annual external audits bij ISO-certified independent bodies. Also, CA's must keep records for years in case of a legal dispute where the information is required as evidence. All this isn't free and go a good way to explaining the prices of certificates.

This, in combination with the recent compromise of some of RSA's more important systems and one other event I'm not at liberty to disclose has my subconscious working overtime, is it a pattern, or just a coincidence ..

please make ready tor application with Firefox 4 .

Comodo was also the company that previous had problems with resellers not checking domains at all - see Mozilla bug 470897. They also allow certificates for domains that don't exist, such as Mozilla bug 526560, and certificates that name companies that don't exist, bug 599856.

Mozilla, at least, refuses to enact any sort of punishment because they think users will blame Firefox when the SSL sites using those certificates stop working. Essentially, they believe they cannot do anything that might possibly actually fix the problem of... well, the CA being crap.

At least these particular changes managed to land just in time for Firefox 4; it is the last commit on the relevant branch.

That wasn’t the sort of security problem that should have been kept secret until patches were shipped — it was clear from the beginning that:

  • bad guys were already able to exploit the bogus certificates,
  • disclosing the existence of the bogus certificates would not have allowed more people to perform the attack they enabled, and
  • waiting to disclose the existence of the bogus certs only aided the ‘cyber-terrorists’ who had obtained them, by allowing the certs to be used for a longer period of time.

On their blog, I asked Mozilla why they agreed to keep this event secret:

But I should ask you a few questions about your conduct in this matter as well:

  1. You wrote above: “Mozilla expressed some concern about disclosure and I offered to embargo this document until Tuesday March 22nd, the launch day for Firefox 4.” What concerns did they express, and why did those concerns convince you to aid in an attack by concealing the existence of these fraudulent certificates?
  2. Did you inform anyone else affiliated with The Tor Project, Inc. before offering to ‘embargo this document’, or was that decision yours alone?
  3. Do you believe that Mozilla would have included Comodo's root certificates as ‘trusted’ CAs in the final release of Firefox 4 if you had disclosed your discovery immediately, rather than delaying your disclosure?

Exactly! Why the embargo ?? This isn't something that should be a traditional responsible disclosure issue. Jake you of all people I assume would've understood this.

I'm sorry that I agreed to the embargo at all at this point.

To reply your your questions:

1) They expressed concerns about being able to ship a fix to Firefox - that is a brand new Firefox binary with hardcoded serials in their blocklist.
1a) We need Firefox to release before we can update the Tor Browser Bundle because our current build process on Windows does not include building Firefox from source. We probably need to change that but it is not my call.

2) I agreed to the embargo on my own - I informed people internally that I was looking for this when I started out and I informed Tor internally when I discovered it, I also again informed Tor when I agreed to the embargo.
2a) I let helix (the main Tor build engineer) know that Firefox would be releasing on Tuesday. Specifically, I explained why and what needs to happen for Tor users to be safe. See point 1a on why this is a PITA for Tor's hard working build engineer.
2b) I started advocating requiring (it's already enabled) OCSP checking from the moment that I realized it would provide a mitigation - I didn't embargo myself for mitigations, only some of the specific details.

3) Mozilla will not give the Comodo CA in question an Internet Death Sentence. We do not know the extent of the compromise and Mozilla did not put off their launch for this extremely serious issue. I think that at the very least, this was a difficult choice for Mozilla. I believe that it was a mistake to not mark the direct signing cert of the certificates in question as untrusted. The rest of the root should have a closer inspection after release, of course. None of that was done for the same reason that OCSP checking isn't required - Mozilla makes these calls and decided they know best.

I obviously disagree with these security choices because they rob the user of their autonomy; users need information to protect their activities on the internet.

I thank you for your work in this area, and also for your warning/recommendation about enabling forced OCSP checking. It's good to know, when there are so many that are trying to tear down, that some are trying to get the protection out there.

Jake did advocate/informed me to enable OCSP in the early days of these events.

I also informed a few friends in Iran. 1 of them reported that he is not able to visit sites with OCSP enabled in firefox.

However the same user 24 hours later, reported that the problem has been resolved automatically :-/

Thank you Jake!

It's funny how when I enabled OSCP in Firefox I couldn't access the EFF link you posted anymore...

Worse, EFF uses wildcard certificate.

CN = *
OU = Comodo PremiumSSL Wildcard
O = Electronic Frontier Foundation
Object Identifier (2 5 4 9) = 454 Shotwell St
L = San Francisco
ST = California
Object Identifier (2 5 4 17) = 94110
C = US

"Last week, a smoking gun came into sight: A Certification Authority appeared to be compromised in some capacity, and the attacker issued themselves valid HTTPS certificates for high-value web sites. With these certificates, the attacker could impersonate the identities of the victim web sites or other related systems, probably undetectably for the majority of users on the internet."

Did they attacker gain access to the additional information required for 'extended validation'? An SSL Cert is only used to secure a site - not to say anything about the site owner. That is unless, they have been issued and EVC.

SSL has always been for both confidentiality and verifying the identity of a server. Why do you think certificates have a subject DN field if they are only for encryption and decryption?

This is obviously a conspiracy between the illuminati, browser vendors, and ioerror. We should add it to

All software is written by the new world order. The Internet was developed by the US Dept of Defense to build a perfect tracking system. Google, Microsoft, Oracle, Verisign, RSA, Symantec are all members of the same collusion set with the NSA, and other world governments.

Disconnect now, or be subject to their whims and wrath.

I've said too much for the black helicopters approach.

I'm guessing you're british or north american. Your government lies to your face continuously and launches wars in your name. Kills, assassinates, bombs civilians, tortures people or sends them away to secret holes to be tortured by their lackeys.

How about them conspiracies?

1. He's joking
2. Which government do you think the NSA is in, exactly?

ioerror, this is a superb analysis, thanks.

this paper looks interesting in the light of this event:

There is already an implicit trust system built in that may be used in the meantime: DNS. This is what the RFC 4398 CERT resource record could help achieve.

IANA is already a trusted 3rd party. IANA runs IP, AS, and DNS address allocation. If you want to connect to TORPROJECT.ORG,, or AS1 for that matter, look no further than an implicit trust chain emanating from ARIN. The question is, is ARIN more trustworthy than a random CA that Mozilla has mated with? I believe so, and I think many others believe so as well. Let the users decide.

But alas, no implementation of RFC 4398, c. 2006, exists to date... We have all sorts of crazy iPhone apps, but no CERT RR implementation. The RFC likely needs expansive interpretation or outright modification for use in tandem with TLS and other widely used protocols. I hope some programmers that are well versed with browser plugins, libc/libresolv/windns/dnsapi/etc., and protocol considerations will get involved.

Just get something working, it has to be better than this. Otherwise, good article.

DNS-Records without DNSSEC are no replacement for even the weakest CA infrastructure. *Every* possible exploitation scenario that involves a fraudulent certificate in an HTTPS connection -- i.e. every scenario which storing certificates in the DNS would supposedly fix -- give the attacker direct (if the user was lured to the wrong site by some sort of DNS manipulation) or indirect (if the attacker is on the same network as the user or on the path from user to original site) control over the DNS and therefore the CERT records.

DNSSEC is part of DNS as of RFC 4035, c. March 2005, which predates and is accounted for in RFC 4398 CERT resource records. In addition, use of IPSec could possibly bring confidentiality to recursive queries and authentication to stub queries.

In other words, these concerns are largely moot.

No one uses DNSSEC or IPSec or IPv6.

None of them are perfect either, so that they are /designed/ to mitigate things does not mean that they actually do.

We Must Fix HTTPS:

An interesting set of slides on https model.

When Certstar got caught issuing a certificate for without doing valid checks ( people started calling out Comodo because Certstar was a Comodo reseller and Comodo shouldn’t have let that happen. That resulted in Comodo suspending that reseller’s ability to issue certifications because they were not fulfilling their proper contractual obligations to Comodo… During that process people were looking closely at Comodo trust paths, and Comodo was being nice and answering questions in forums to try and clear their name. During this questioning AddTrust came up. If you follow the thread around it reveals that AddTrust AB no longer exists as a company, but Comodo bought the certificates from them, and the rights to continue to use their root CA which bear the AddTrust name.

The history is illustrative, basically the AddTrust company was acquired by ScandTrust (out of Malmö, Sweden), which was a Swedish CA, and then Comodo purchased the AddTrust root from ScandTrust. Then Comodo also purchased UserTrust, which had four roots and those also became part of Comodo when UserTrust became a “Comodo Group Company”.

In both of the above cases the key material was removed from its original sites of operation (in Sweden and Salt Lake City, respectively) and trafserred into Comodo's data centres and backup centres.

This is interesting because defunct companies, that had been sold, transferred their key material around to different parties. I think that this is the wrong way to do this, and that the right way to do this is to revoke the certificates and the CA, but thats an expensive operation. So, I think what they do with these certs is that they actually recognize the lower value of such a CA, and what they do is is sell these certs at a cheaper cost, because they haven't been under the control of Comodo from the beginning.

They call these “Usertrust” CAs and they offer 3rd party vendors, such as Gandi to issue these certs, which are way cheaper than Comodo’s in-house generated ones.


What about phobos' surely phobos and those like him would notice the cert changed?

Can we at least start scoping SSL certs by internet domain to limit the damage? Let a root CA sign for any domain, but create a COM CA that can only sign for *.com, and let it sell a CA to google that can only sign for * and so on...
Or is this too detrimental to the business model to ever happen?

Too many questions unanswered - what about domains that don't want to maintain their own CA? Who operates the *.com?

Why the embargo ? Seems like this isn't a case for responsible disclosure because this isn't a traditional vulnerability.

Yes we could do with a better system based on gpg or something but they're struggling to get dns security right.

Untill it is right and used by all then all a website owner can do in this regard is look after his key, data once received and mail systems. It's a joke the EV and CA costs for a false sense of security which is making it take longer for a general consensus to want to find a new system. I'm glad you can get the certs accepted by browsers for free. I'd use self signed if browsers didn't say the worlds gonna end when you do. I think microsoft probably contemplated shutting your computer down the way IE reacts.

Black helicopters were mentioned, the openbsd list had a discussion recently part of which was that the government can get CAs to cooperate and can MITM the network legally. Assuming they're systems don't have gaping holes, (quite probable running cisco and M$ rubbish at great expense) and they're employees are trustworthy (hmmm), then if you can still be MITM then you likely have far greater problems than SSL.

CAs are dead
Long live, hmmm

bastion hosts

Shouldn't the browsers be able to spot this behavior fairly easily by taking "signatures" of the certs that they are presented with and comparing certs that are supposedly for the same host names? I know this is would be detecting the problem after the fact but it seems fairly reasonable, no?

Microsoft's response:

A source at Microsoft says the following certs were obtained.
• (3 certificates)
• "Global Trustee"

I'm not sure what the "Global Trustee" means.

IP Address:
City: Tehran
State or Region: Tehran
Country: Iran, Islamic Republic of

1. Steal certs
2. Redirect DNS inside Iran to fake sites
3. Block OCSP / access to CRL
4. Capture logins
5. Snoop on gmail, hotmail, yahoo email boxes
6. Round up activists
7. Torture
8. Repeat

9. ???
10. PROFIT!!!

we need a international action toward Iran government

1. It's not necessarily Iran. China has been interfering with GMail lately, for example.
2. Fixing the PKI is cheaper than “international action”

Update: Comodo has published an incident report:

Does anyone know what's Safari's status is with blacklisting revoked certificates?

so we don't trust this douche-b (comodo) to release information about a huge security breach in a timely fashion but we /do/ trust him to name the "state actor" behind the attacks?? what a shocker that they named iran, the USSA's enemy du jour. i dont believe that bullshit for a second, now if u had said USSA's 51st satellite apartheid state, THEN i'd believe you. who was behind STUXNET? who is more inclined to pull off this bullshit and blame it on others (by way of deception....) stop being so fucking stupid.

I share your concern about Comodo's practices but at this point, I don't think they have much to gain by fudging or hiding info. Their reputation is seriously at stake and if they mess up, they will be history very soon.

Send them another Christmas present for their efforts.

Microsoft update regarding this issue here,

Does the SSL Observatory have any legit certs with global trustee in the cn or san fields?

I asked Seth from the EFF and he reported that he did not find anything with that string in the Observatory data.

They're still missing a few serious points.

I may be completely wrong here, but couldn't an attacker beat revocation by serving a man-in-the-middle victim an old version of the CRL, downloaded before the cert was revoked.
The CRL files are all stored on http servers, not https.

The HTTP 500 trick does that without any crypto trickery.

There are some relevant comments in a paper of mine that will appear at next week's Security Protocols Workshop and that was put online by the workshop organisers today. The direct URL is

Ross Anderson

Oh, that wacky Comodo / Usertrust / UTN CA.

For debian and ubuntu users, here is a quick way to quit trusting this CA. I posted it back in 2008, when they issued a certificate they shouldn't have (see slashdot link above).

sudo sed -ri '/comodo|utn|addtrust/Is/^!*/!/' /etc/ca-certificates.conf; sudo update-ca-certificates

For firefox users on linux, run this with firefox closed:
locate -er '/cert8\.db$' |sed 's;[^/]*$;;' |while read db; do echo $db; certutil -d "$db" -L |egrep -i 'comodo|utn|usertrust|addtrust' |sed -re 's; +[a-zA-Z]*,.*;;' |while read cn; do echo "- $cn"; certutil -d "$db" -M -t cw,cw,cw -n "$cn"; certutil -d "$db" -D -n "$cn"; done; done

For firefox users, using a gui:

Somewhere there is a UserTrust cert that signs itself:

I've marked all my CAs as invalid and use the perspectives firefox extension:

Not perfect, but if I am seeing a cert that doesn't correspond to the notaries, it will inform me. (it doesn't override firefiox xpi signatures, so installs will simply fail).

Wonderful, wonderful. Unearthing this is clearly a sackload of work, for which kudos. Also notice how it is necessary and not something a random user would do. Since it IS necessary, it means safely using the thing is just about impossible except for experts.

This, by the way, is a glaring error, omission, and oversight in security thinking. It keeps on cropping up any time crypto is introduced, and is closely related to the problems detailed in _Why Johnny Can't Encrypt_. Even Appelbaum thinks about this technically. The biggest concern is not that, it's that without deep technical understanding of the system it can be used neither safely nor securely. How's that for userfriendlyness? --bounce

Went to upgrade FF, upgrade failed. Downloaded 4 and installed. Immediately got OCSP errors going to Facebook. Turned on WS and started sniffing packets. Fatal error, bad certificates! Checked against list. Nope different numbers. Did about:config and made sure OCSP was turned on.

It fails if it does not get a response. Good.

Checked the IP where I was getting errors. Skype. Other errors, Facebook.

Read the Freedom To Tinker post, aha. Read this post, aha. Nice.

Noticed that both CA use c=US in their name but do not support the listing of their X.509v3 certs in a c=US X.500 directory when it is just simpler to hard code them into browsers and make the process opaque to users.

The error becomes obvious, if one could easily lookup the right certificates one would not trust the rogue ones. Following the path in browsers has led me down strange alleyways smothered with snake oil, non-working addresses, and some developer who rolled up a demo and it ends up being on a financial site that I am supposed to be able to use to check a balance. Why, because no one is really paying attention, they just want to turn on the crypto, and who cares if it is valid.

But tell that to a browser. Any software with LDAP could fetch that certificate and make a comparison if the CAs supported the name space they have been exploiting. RFC4398...please give me a break. This is a systemic problem that is killing privacy for convenience.

"Comodo also clearly demonstrate a mis-understanding - they believe that checks for revocation are proof positive that certificates are being used"

I might be being dense - but isn't this statement backwards. Shouldn't this be: they believe that NO checks for revocation are proof positive that the certificates are NOT being used.

Report of incident on 15-MAR-2011
An RA suffered an attack that resulted in a breach of one user account of that specific RA.
This RA account was then used fraudulently to issue 9 certificates (across 7 different domains).

This only means that someone had got hold of a RA username and password. Comodo uses 2Tiers for their reselling. This has nothing to do with any CA certificate or breach of the PKI infrastrukture. The Certificate was revoked, as normal procedure in such circumstances. However, customes are protected by the Comodo insurance if they suffered any loss because of this, based on the type of certificate that was issued.

If it was an attack by Iranian authorities, like they suggest, the loss will be that they are hanged slowly in the Evin prison. That's not covered by the insurance, I'm pretty sure.

I propose a very different approach, that does not use public key encryption at all:

Jan-Hein van der Burg.

Infidels! One day great nation of Iran will have the funds to purchase their own CA like your NSA.

Hey, you still getting molested at the airport Jacob?

In a perfect world
people who invest their time in detecting and reporting security holes which affect the entire world,
people who helped reporters from Egypt and Libya to bypass censorship
should deserve a medal.

In this brave new world they only get harassed in US airports.

Jake, in your update you have harsh words for Comodo. This is surely justified.

Yet your own company, Tor, is hardly a sterling example of protecting its users.

The Torbutton extension for Firefox is one of the most frequently downloaded extensions on that site, 4,687,639 downloads to date and 182,056 daily users on March 23rd. Torbutton bundles Tor, Polipo, and Vidalia.

It advertises itself as "currently the only addon that will safely manage your Tor browsing to prevent IP address leakage, cookie leakage, and general privacy attacks", thus giving the user a reasonable expectation of privacy.

Yet it was last updated April 19, 2010 -- nearly a year ago!

Since then there have been many bugfixes to Tor, including fixes to serious security-relevant issues. When will these bugfixes find their way into Torbutton?

By not updating Torbutton, you are putting millions of users of the extension at risk. Some of them may be dissidents in brutal dictatorships and the consequences to them of being found out are dire.

I think you're mistaken and I've pointed Mike at your comment. He'll address them shortly.

You seem to misunderstand what Torbutton does. It only handles the Firefox side of things and has nothing to do with Tor core.

Aside from Firefox 4 compatibility and a couple addon conflicts, not much new in the way of security updates for Torbutton has been needed. If you want Firefox 4 support and fixes for some minor issues, use the alpha series at:

The 'global trustee' cert sounds to me like it could be a signing certificate rather than simply a normal normal site certificate. If so then the attacker could sign further certs of their own. Even worse is that OCSP is generally only applied to the leaf of trust, meaning that even if OCSP worked properly it wouldn't help should this be true.

Lots of talk about Mozilla, IE, and Chrome but what about Opera and Safari? Are they posting patches? Do they automatically check CRLs?

Great post.

1. I have a question here, you mention how browsers soft fail on a OCSP/CRL attempts. Now, assuming they are coded to hard fail. Even then an adversary can MITM the OCSP request and defeat this process right ? In this case, the browsers will be helpless sitting birds, unless the OCSP contact made by browser to CA has some measures in place to avoid a MITM.

2. Opera seems to be maintaining a separate server of their own with certificates etc. Need to dig this deeper, but it seems to evade the whole CA down thing (assuming Opera will try to ensure that those servers are up)

I wonder why if the lead byte of cert serial is > 0x7F they pre-pend that 0x00? Anyone gone through source to see why that is? Not an issue, just curious.

It's a certificate encoding issue - Chrome does the same thing internally.

Great information.Thanks.

Chinese gov action would be possible

fuck all

hello all, i say f4ck all

Thanks all

What makes you guys think that solving for the CA's private key from the public key is all that hard. Have you looked into how much (or how little) computation power is need these days to do this? It certainly is within the capability of large corporations and even small countries.

The whole https security thing is bogus anyway unless the domain name server system can be secured. Most companies are better off using their own (self created) certificates and changing them frequently (like weekly or daily). The browser preapproved CA list just encourages forgery.

It depends on the private key in question - do you know of any keys smaller than 1024bits in major browsers?

There is no proof that finding the private key from the public key is a 'hard' or 'large' problem in the mathematical algorithmic sense - only that most people don't know how to do it efficiently. In any event it is certainly no 'bigger' than the factoring problem (factoring the product of two large primes) in which there has been alot of progress algoritmically. Practically, in light of modern processors, say clusters of GPUs, backsolving private keys is almost personally affordable.

Comodo has not been hacked, but the user credentials of a sovereign account at UserTrust were leaked. The scandal is not the hack, but that these accounts even exist and are allowed to generate certificates at will.

An unknown entity just tried to gain the same level of arbitrary access to the CA system, that western states already have.

Amazing research, fascinating article. Thank you for your work on this issue and on Tor in general.

I have some contacts with human rights activists working in Iran. What kind of precautions do they need to take in light of the possibility that the attack originated from the state?

They should be using Tor with bridges or without to avoid targeted HTTPS MITM attacks. It's very hard to target every gmail user if you're not trying to connect to gmail.

I notice more and more certificates are getting added and updated all the time. Is there a better and more secure and safe way than certificates or can more safeguards be applied to help keep certificates safe. If Iran did indeed have dealings in this then we must in carefully analyze the information and discover how to better safeguard and protect certificates to prevent man in the middle attacks from happening. Furthermore, what other safeguards besides obvious ones like passwords, biometrics, secure limited access, need to know data, etc. can be used to safeguard information. I know certain things could just be done the old fashioned way by pen and paper and safeguarded in a vault and/or an older computer that has access restricted to only a few individuals with supplies carefully protected and fully scanned to entrance to vault. We still need more and better ideas and soon. Thank you.

I notice more and more certificates are getting added and updated all the time. Is there a better and more secure and safe way than certificates or can more safeguards be applied to help keep certificates safe. If Iran did indeed have dealings in this then we must in carefully analyze the information and discover how to better safeguard and protect certificates to prevent man in the middle attacks from happening. Furthermore, what other safeguards besides obvious ones like passwords, biometrics, secure limited access, need to know data, etc. can be used to safeguard information. I know certain things could just be done the old fashioned way by pen and paper and safeguarded in a vault and/or an older computer that has access restricted to only a few individuals with supplies carefully protected and fully scanned to entrance to vault. We still need more and better ideas and soon. Thank you.

So how come security.OCSP.require is set to false in the latest Tor Browser Bundle if you yourself have repeatedly and strongly recommended setting it to true.

I don't produce the TBB. I've argued internally that we should enable this flag and this build fixes the immediate compromise problems at hand.

Anyone who hasn't upgraded should enable that flag by hand, of course.

How about disabling entirely so as not to notify third parties that the site in question is being accessed?

Personally, I think that we should run an anonymous OCSP or CRL caching proxy on a hidden service. I don't have time to write one but it does seem like a reasonable compromise for hard failure modes and privacy concerns.

Don't trust CAs that you do not actually know. Who is Comodo ? I don't know anyone there, hence I don't trust them. Same goes for all the others, I just live without any CAs on my "trusted browser".

I trust my friends CAs, I trust my bank and other services certificates because I manually verify them. For the non important stuff, I take the risk of reading an altered blog post (or use my "non trusted browser").

It appears that the Microsoft update doesn’t just contain the certificate serial numbers, but the whole certificates. They include in the subject:

OU = Hosted by GTI Group Corporation

I’d guess that makes the registration authority GlobalTrust Inc (a division of GTI Group Corporation). They’re in Italy, which matches some news reports. Their websites have also done a disappearing act.

I wonder if this goes some way to explaining the strange "global trustee" certificate. I apologise if this is already common knowledge, but I’ve not seen it mentioned anywhere yet.

The extent of the problem is surely a matter of whether the signing key was obtained or not. If the attackers obtained that, then revoking the certs issued on the CA's computer are almost immaterial as the attacker can continue issuing certs on their own computer. Gah, this is such a mess.

Google's ones :

Mozilla's ones :

G9239... == M009239 ...
Gf5c8 == M00f5c8
Gd755 == M00d755
Gb0b7 == M00b0b7
Gd8f3 = M00d8f3
G047ecb == M047ecb

Every hash in the Google list are found in the Mozilla list : just a matter of visual representation.

Certificate information of 72032105C50C08573D8EA5304EFEE8B0 is the followings.


Subject:,OU=Comodo Trial SSL,OU=TEST USE ONLY - NO WARRANTY ATTACHED,OU=For Testing Purposes Only,O=Kai Engert,STREET=Test Street,L=Test City,ST=Test State,postalCode=12345,C=DE
CN=UTN-USERFirst-Hardware,OU=,O=The USERTRUST Network,L=Salt Lake City,ST=UT,C=US
Serial Number:
Issuer DER Base64:
Serial DER Base64:
Serial DER as C source:
{ 16, "\x72\x03\x21\x05\xc5\x0c\x08\x57\x3d\x8e\xa5\x30\x4e\xfe\xe8\xb0" }
Fingerprint (MD5):
Fingerprint (SHA1):

Revocation date of 9 certificates are the followings.

Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E
Revocation Date: Mar 15 19:04:03 2011 GMT
Serial Number: F5C86AF36162F13A64F54F6DC9587C06
Revocation Date: Mar 15 19:04:24 2011 GMT
Serial Number: E9028B9578E415DC1A710A2B88154447
Revocation Date: Mar 15 19:05:26 2011 GMT
Serial Number: 9239D5348F40D1695A745470E1F23F43
Revocation Date: Mar 15 20:15:20 2011 GMT
Serial Number: D7558FDAF5F1105BB213282B707729A3
Revocation Date: Mar 15 20:15:26 2011 GMT
Serial Number: 392A434F0E07DF1F8AA305DE34E0C229
Revocation Date: Mar 15 20:15:38 2011 GMT
Serial Number: 3E75CED46B693021218830AE86A82A71
Revocation Date: Mar 15 20:15:47 2011 GMT
Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0
Revocation Date: Mar 15 20:16:03 2011 GMT
Serial Number: D8F35F4EB7872B2DAB0692E315382FB0
Revocation Date: Mar 15 20:19:04 2011 GMT

Shouldn't people update their Namoroka (modified Firefox shipped with TOR) to detect these certs?

I don't understand sth. Why has the industry chosen to 'ignore' returned errors on the crl queries? Are there a lot of false positives? And is it i.a. a calculated risk they are willing to take?

> Why has the industry chosen to 'ignore' returned errors on the crl queries?

It's easy to guess why : end user experience! Joe User wants to get to that site and could care less about faked certs. Browser vendors chose to give access anyway rather than risk the user blame browser.


Why don't browsers just check and match domain name/ip registration details and present them to users for approval/refusal?

This is going to be a long post, but there are a number of things that are relevant to the discussion and some things I just want to get off my chest (yet some details I can't disclose, not for loyalty to an entity but for fear of that entity knowing my identity). Some of this is in response to comments, of which there are too many to quote

The SSL/TLS model used by HTTPS (browsers), and a number of other protocols, is fundamentally flawed from multiple perspectives. Some have been mentioned in the article, so I will just add onto the list.

First big problem is the typing of encryption and authentication. The fact that I can't encrypt any content without a cert is a major problem. Sure you can use a self-signed cert, or make your own CA, but that causes browsers to throw a fit and is a bad user experience. You can add your cert to the trusted list, but that doesn't scale. Some will argue that encryption is worthless without trust, but I disagree. Rather, it goes the other way. If I need to trust the other party, then surely I want privacy (encryption), but just because I want privacy doesn't mean I have to trust the other party. There are a variety of reasons to want to encrypt more traffic with sites which are fairly anonymous. Many more services would offer an encrypted version without validation if the two were not tied. Alas, currently it is infeasible to do so without spending some cash on a cert, and the reality is that many free projects simply don't have that money to spend (or would rather not fund corrupt CAs for ideological reasons).

Some have stated that there's only validation in EV. That is wrong in principle, EV is extended validation, meaning it extended the validation of standard certs, not that there is none. The entire basis of the certificate is that is proves who you are and to do that there must be validation. Unfortunately, that validation by the commercial CAs is very weak. They actually turned this weakness into part of the marketing of EV certs to organizations that can afford it, but the only real step it adds is necessity of forking over thousands of dollars which, which increasing profits massively, really only stops the casual attacks wherein people obtain certs to show how bad the CAs are (e.g. the two certs from VeriSign a decade ago, the cert from comodo a few years ago).

The CAs are business whose sole purpose is to make money. They say we should trust them, but why? They put the minimal effort into verification because that costs money, reducing profits. They put minimal resources into running reliable CRL and OCSP services, because doing so cuts into profits. They allow resellers with lax rules because more certificates sold means more income, and more stringent checks means fewer certs sold. These companies do not have out security interests in mind, and we know we cannot trusty them, so why do we continue to trust them? Just as I don't trust them to check identity, I don't trust them to protect their keys, to disclose breaches, or to correctly identify the source of problems. I notice I;m not the only one who thinks naming Iran is a little too convenient. I'm actually somewhat surprised they didn't name Libya as part of the crusade to justify yet another war.

The browser vendors also cannot be trusted. They put as many CAs in their root list as possible to make a better user experience. All of them put the most users as top priority, and anything that harms the user experience (for the average non-techie user) is seen as a negative. That is both why they have the default to allow on failure conditions, as well as every CA they can come up with in root list. Worse, the browsers tend to use a union of their list and the OS list. Take a quick look through, and not only will you find the names we know like VeriSign and Comodo, but others that frankly have no reason to be there. How many of you knew AOL and Dell had CAs that your browser trusts? How many are surprised to see a CA from China in their list? Some time ago I interviewed with a certain fruity company for a security manager position and the question came up of who are you trusting when you see the lock icon in your browser. They wanted an answer along the lines of trusting the site, the CA, and everyone in the chain between them involved in issuing the cert. My answer was also the browser vendor, the OS vendor, and anyone who has access (legitimate or not) to add certificates to your root store. That and some other answers didn't help me land that job, but I didn't exactly want it either, I was just playing along because I was intrigued about the existence and purpose of the position as well as why they were calling me for it.

In a past phase of life, I worked on some projects for the US govt involving securing their communications in a mobile environment. Suffice to say that their requirements worked well to negate the effectiveness of their security. In part because certain things that should have been done were not, and in part because the overall combination of reqs made the user experience so bad that many are bound to actively work around it. One thing they did right was they would not validate a cert without checking the CRL. One thing they did wrong was disallow CRL caching (even brief), so the list had to be fetched every time, which is very slow over a public mobile phone network when its several megabytes. By nature, the CRL only grows, it can never shrink. Another problem was no OCSP was allowed. Why would that be? Not because it had shortcomings, but it created a trail that could be audited, and while they were careful to include plenty of downward checks, there was careful avoidance of upward checks. This system is used both by ground troops and by command, extending up to the Chief Executive of the armed forces. Suffice to say, the higher ups don't always want the same scrutiny applied to their activities as they apply to those below. And yes, it all runs on Windows, both the clients and servers, as well as proxies, only some critical routers use Linux because not much else could handle the role.

While working on that project, I had to obtain a certificate for S/MIME use to protect communication in regards to the project. As I was not a member of any part of the govt but an outside contractor, I did not have a .gov address and could not get a certificate from their CA. Instead, I was directed to an ECA (External Certificate Authority), which is a party that has been trusted to issue certificates expressly for the purpose of communicating with certain govt branches. If you have such a certificate, there is an assumption you have some business in sending emails to certain people and they are more likely to respond, so having one when you shouldn't could be abused. The process to get one was much like any other 'high security' certificate; fill out paperwork, include copies of ID (low res black and white xerox), include letter of employment on company letterhead (as if they know what that should look like), get it all notarized (done by the company employed notary that stamps anything delivered to her desk), include payment and send it all off by regular post. The one extra check the ECA does beyond their normal validation is to check if the company is on the list of those who have active projects and thus should be allowed to obtains certs for their employees. A couple weeks after I mailed off the application, I got a call on my office phone from someone at the ECA. He said that everything checked out fine except that out company could not be found on the list from the DoD (I hope they don't actually have the full list, but rather just query it and got back a no response). I told him "Well, we supply to the DoD and some some other departments, but the project which we are actually doing under contract for which I need this certificate originates from NSA, not DoD." His response, which I will never forget, was "Oh, the NSA! They never put anyone on the list. Its all a big secret over there." That was the end of the call and I got my certificate within a day. So if anyone with some social skills who wants to go phishing in the DoD, just remember the magic words "I"m with the NSA." because that's all you need to bypass the regular checks.

In summary, we can't trust any vendors, not the CAs, not the browsers, not the commercial OSes. I think most of us knew that, but for those who didn't, I hope I laid out some convincing points. There are some other bits that bring perspective. We knew the CAs bend for the govt, but the CAs that work in direct cooperation with the govt don't even do a reasonable job in that role. For their role with regular clients, can we expect them to anything other than a worse job? The incompetence of the govt also comes into light, but it shouldn't be news considering the leaks. I think what will be news (for some) is how easily someone so low on the chain could exploit their weaknesses. And once you are in, you are IN. There is NO security once you breach the perimeter...

So, where do we go from here? I already have separate trusted and untrusted browser. I should have a trusted workspace with deeper separation, but Qubes has some ways. There is no absolute security, but the closer you come, the less you can get done. I saw mention of some interesting things, such as peer trust model (not quite the PGP web of trust) to apply after distrusting CAs by default, which I will have to look into. One thing I think we need is an open, trusted CRL aggregator to collect the CRLs from as many CAs as possible (at minimum all those most are default to trust) and make that available for local caching. Another necessity is a local OCSP run from that aggregate CRL to answer questions without leaving a trail outside the locally trusted network. A hosted version of that could be run by and open and trustworthy party as a convenience for those with less strict privacy requirements. As it is, I cannot use public OCSP and leave a record that negates all the effort of using NoScript, AdBlock, cookie blocking, Tor, etc. I can pull the CRLs myself and cache them locally, but a local OCSP would greatly simplify the process as currently it requires catching the CRL requests and doing so comprehensively is non-trivial.

That is still just a stopgap to reduce the problem with CAs. First, we need encryption by default without regard to the partner. In the longer term, we need a replacement for CAs. Neither the current CA system nor the PGP web of trust is sufficient for all cases. The most obvious solution is a hybrid model. At a minimum, more than one CA should have to sign an intermediate and more than one intermediate should have to sign a domain's/site's cert to make it possibly trusted. Further, sites could mark each other trusted, but that should not be the only source of trust. The exact details of how to do this greatly overlaps with a much longer running discussion started (or continuing) some months ago with serious discussion about the future of DNS following the abuses of the US govt. I would not be surprised to see the govt abusing root CAs with greater ease than they have abused DNS recently. Notice that they don't even have to use a commercial CA to forge any site, if you use a commercial OS you already have certs for govt owned CAs in your roots.

is My device "iPhone"

@ ioerror,

I just came across interesting blog post(s) [1, 2 and worthwhile is 3] on this story and possible missteps (speculation) by the EFF while looking to Google Analytics Opt-Out Firefox Add-on[4] vs the use of Adblock Plus and GreaseMonkey[5] to block Google Analytics[6]. I wonder if you have read the blog post(s) [1,2] and your comments/thoughts, thanks!

[1] "Did the EFF speculate on Comodogate?"

[2] "Comodogate part 2"

[3] "Comodogate"

[4] Google Analytics Opt-out Browser Add-on

[5] ""

[6] "Google Analytics Opt Out? Not really."

What are the relevant privacy issues when using OSCP in Firefox with Tor? Likewise, what are the privacy issues with using the Firefox Add-on Perspectives with Tor?

I do wish to increase SSL cert security but I worry very much about the privacy issues involved with OSCP and Perspectives ...

Lastly, what is the word on OCSP stapling? Can it be used with Firefox and Tor? As I understand it, OSCP stapling mitigates some (most/all?) privacy issues with OSCP; no? However, there are no currently available OSCP stapling servers that can be used with Firefox, only OSCP servers; no?

unfortunately our OIL money consumes against Iran people's.
help us.

i'm living in iran and confirm this.

Interestingly, today, 2011-10-26, the Certificate Patrol plugin in
my browser noticed that Google replaced at least one of its
certificates out of schedule.

My browser shows that the old certificate was valid from 2011-09-05
06:07:38 GMT to 2012-09-05 06:17:38 GMT. The new one is shown as
valid from 2011-20-09 06:08:21 GMT to 2012-09-20 06:18:21
GMT. Normally, certificates are replaced a short time before they
expire, for example a few weeks before. This replacement happened
about a year before expiration, about two weeks after the previous
update, so it appears to have been replaced out of schedule.

Strangely, the *old* certificate was stored by my browser on
2011-10-05, about another two weeks *after* the validity start of
the *new* certificate.

To me, it appears that the old and new certificates essentially
differ in the public keys contained in the certificates. The
private keys associated with this public key are owned by
"*". Other differences such as the serial number are
to be expected.

That private key was replaced. Why? Together with the certificate
itself (which is public), it can be used to impersonate servers in
the domain in question ( Maybe it was in danger of not
being private any more?

The old cert's SHA1 fingerprint was:
Subject Key Identifier:

The new cert's SHA1 is:
Subject Key Identifier:

Here's anoother replacement of the same sort,
this time for

The old cert:
Valid from: 2011-09-20 08:07:19
SHA1 fingerprint:
Subject Key Identifier:
e0 18 c3 8e 7f 2f da 55 9e 50 54 27 a9 2b 34 06 06 c8 72 b5
-- The same as the new certificate valid from 2011-09-20 for

The new cert:
Valid from: 2011-11-10 08:49:06
Subject Key Identifier:
ed d7 71 f8 e2 55 ae ac 2c 70 5d f7 10 9d 83 cd 97 0b ca 2c

Another explanation would be that they rotate keys around their sites. ;-)

And another replacement, encountered at, and,
valid for *, replacing

SHA-1: 3A:26:9A:89:35:8C:99:2E:51:D8:AC:D5:9D:A8:4B:A8:F5:5F:73:9A
issued on 2011-11-10 08:48:51


SHA-1: 8F:C0:79:E8:14:77:7F:68:8B:A4:C8:07:D9:BD:67:D6:2A:F7:1A:EB
issued on 2011-11-21 04:59:22,

which is again *before* Certificate Patrol first stored the older one,
on 2011-11-24 12:58:52 .

Yet another replacement, encountered at,
valid for *, replacing the new one described above on October 26th, 2011,

SHA-1: 91:37:CB:62:F5:F4:E4:73:99:A7:18:BD:5F:83:69:D0:0D:AE:90:7D
issued on 2011-09-20 08:08:21


SHA-1: 4D:31:BC:75:D6:BF:D8:5C:63:1D:66:61:CF:36:19:E0:7F:D8:12:D4
issued on 2011-11-21 05:01:16 .

In this case, the old certificate was stored by Certificate Patrol
on 2011-10-26 16:06:06,before the new certificate was issued, as is to be expected.


Thank you four your nice writing on Detecting Certificate Authority compromises and web browser collusion .


Syndicate content Syndicate content