Yes, we know about the Guardian article


And also the Washington Post article.

We're planning to write up a more detailed analysis later, but for now here's a place to centralize all the "hey did you know about this article" blog comments.

And for the journalists out there who want a statement, here's my quote from the article:

"The good news is that they went for a browser exploit, meaning there's no indication they can break the Tor protocol or do traffic analysis on the Tor network. Infecting the laptop, phone, or desktop is still the easiest way to learn about the human behind the keyboard.

Tor still helps here: you can target individuals with browser exploits, but if you attack too many users, somebody's going to notice. So even if the NSA aims to surveil everyone, everywhere, they have to be a lot more selective about which Tor users they spy on.

Just using Tor isn't enough to keep you safe in all cases. Browser exploits, large-scale surveillance, and general user security are all challenging topics for the average internet user. These attacks make it clear that we, the broader internet community, need to keep working on better security for browsers and other internet-facing applications."

Deterministic Builds Part Two: Technical Details

This is the second post in a two-part series on the build security improvements in the Tor Browser Bundle 3.0 release cycle.

The first post described why such security is necessary. This post is meant to describe the technical details with respect to how such builds are produced.

We achieve our build security through a reproducible build process that enables anyone to produce byte-for-byte identical binaries to the ones we release. Elsewhere on the Internet, this process is varyingly called "deterministic builds", "reproducible builds", "idempotent builds", and probably a few other terms, too.

To produce byte-for-byte identical packages, we use Gitian to build Tor Browser Bundle 3.0 and above, but that isn't the only option for achieving reproducible builds. We will first describe how we use Gitian, and then go on to enumerate the individual issues that Gitian solves for us, and that we had to solve ourselves through either wrapper scripts, hacks, build process patches, and (in one esoteric case for Windows) direct binary patching.

Gitian: What is it?

Gitian is a thin wrapper around the Ubuntu virtualization tools written in a combination of Ruby and bash. It was originally developed by Bitcoin developers to ensure the build security and integrity of the Bitcoin software.

Gitian uses Ubuntu's python-vmbuilder to create a qcow2 base image for an Ubuntu version and architecture combination and a set of git and tarball inputs that you specify in a 'descriptor', and then proceeds to run a shell script that you provide to build a component inside that controlled environment. This build process produces an output set that includes the compiled result and another "output descriptor" that captures the versions and hashes of all packages present on the machine during compilation.

Gitian requires either Intel VT support (for qemu-kvm), or LXC support, and currently only supports launching Ubuntu build environments from Ubuntu itself.

Gitian: How Tor Uses It

Tor's use of Gitian is slightly more automated and also slightly different than how Bitcoin uses it.

First of all, because Gitian supports only Ubuntu hosts and targets, we must cross compile everything for Windows and MacOS. Luckily, Mozilla provides support for MinGW-w64 as a "third tier" compiler, and does endeavor to work with the MinGW team to fix issues as they arise.

To our further good fortune, we were able to use a MacOS cross compiler created by Ray Donnelly based on a fork of "Toolchain4". We owe Ray a great deal for providing his compilers to the public, and he has been most excellent in terms of helping us through any issues we encountered with them. Ray is also working to merge his patches into the crosstools-ng project, to provide a more seamless build process to create rebuilds of his compilers. As of this writing, we are still using his binaries in combination with the flosoft MacOS10.X SDK.

For each platform, we build the components of Tor Browser Bundle in 3 stages, with one descriptor per stage. The first descriptor builds Tor and its core dependency libraries (OpenSSL, libevent, and zlib) and produces one output zip file. The second descriptor builds Firefox. The third descriptor combines the previous two outputs along with our included Firefox addons and localization files to produce the actual localized bundle files.

We provide a Makefile and shellscript-based wrapper around Gitian to automate the download and authentication of our source inputs prior to build, and to perform a final step that creates a sha256sums.txt file that lists all of the bundle hashes, and can be signed by any number of detached signatures, one for each builder.

It is important to distribute multiple cryptographic signatures to prevent targeted attacks against stealing a single build signing key (because the signing key itself is of course another single point of failure). Unfortunately, GPG currently lacks support for verifying multiple signatures of the same document. Users must manually copy each detached signature they wish to verify into its proper .asc filename suffix. Eventually, we hope to create a stub installer or wrapper script of some kind to simplify this step, as well as add multi-signature support to the Firefox update process. We are also investigating adding a URL and a hash of the package list the Tor Consensus, so that the Tor Consensus document itself authenticates our binary packages.

We do not use the Gitian output descriptors or the Gitian signing tools, because the tools are used to sign Gitian's output descriptors. We found that Ubuntu's individual packages (which are listed in the output descriptors) varied too frequently to allow this mechanism to be reproducible for very long. However, we do include a list of input versions and hashes used to produce each bundle in the bundle itself. The format of this versions file is the same that we use as input to download the sources. This means it should remain possible to re-build an arbitrary bundle for verification at a later date, assuming that any later updates to Ubuntu's toolchain packages do not change the output.

Gitian: Pain Points

Gitian is not perfect. In fact, many who have tried our build system have remarked that it is not even close to deterministic (and that for this and other reasons 'Reproducible Builds' is a better term). In fact, it seems to experience build failures for quite unpredictible reasons related to bugs in one or more of qemu-kvm/LXC, make, qcow copy-on-write image support. These bugs are often intermittent, and simply restarting the build process often causes things to proceed smoothly. This has made the bugs exceedingly tricky to pinpoint and diagnose.

Gitian's use of tags (especially signed tags) has some bugs and flaws. For this reason, we verify signatures ourselves after input fetching, and provide gitian only with explicit commit hashes for the input source repositories.

We maintain a list of the most common issues in the build instructions.

Remaining Build Reproducibility Issues

By default, the Gitian VM environment controls the following aspects of the build platform that normally vary and often leak into compiled software: hostname, build path, uname output, toolchain version, and time.

However, Gitian is not enough by itself to magically produce reproducible builds. Beyond what Gitian provides, we had to patch a number of reproducibility issues in Firefox and some of the supporting tools on each platform. These include:

  1. Reordering due to inode ordering differences (exposed via Python's os.walk())

    Several places in the Firefox build process use python scripts to repackage both compiled library archives and zip files. In particular, they tend to obtain directory listings using os.walk(), which is dependent upon the inode ordering of the filesystem. We fix this by sorting those file lists in the applicable places.

  2. LC_ALL localizations alter sorting order

    Sorting only gets you so far, though, if someone from a different locale is trying to reproduce your build. Differences in your character sets will cause these sort orders to differ. For this reason, we set the LC_ALL environment variable to 'C' at the top of our Gitian descriptors.

  3. Hostname and other OS info leaks in LXC mode

    For these cases, we simply patch the pieces of Firefox that include the hostname (primarily for about:buildconfig).

  4. Millisecond and below timestamps are not fixed by libfaketime

    Gitian relies on libfaketime to set the clock to a fixed value to deal with embedded timestamps in archives and in the build process. However, in some places, Firefox inserts millisecond timestamps into its supporting libraries as part of an informational structure. We simply zero these fields.

  5. FIPS-140 mode generates throwaway signing keys

    A rather insane subsection of the FIPS-140 certification standard requires that you distribute signatures for all of your cryptographic libraries. The Firefox build process meets this requirement by generating a temporary key, using it to sign the libraries, and discarding the private portion of that key. Because there are many other ways to intercept the crypto outside of modifying the actual DLL images, we opted to simply remove these signature files from distribution. There simply is no way to verify code integrity on a running system without both OS and coprocessor assistance. Download package signatures make sense of course, but we handle those another way (as mentioned above).

  6. On Windows builds, something mysterious causes 3 bytes to randomly vary
    in the binary.

    Unable to determine the source of this, we just bitstomp the binary and regenerate the PE header checksums using strip... Seems fine so far! ;)

  7. umask leaks into LXC mode in some cases

    We fix this by manually setting umask at the top of our Gitian descriptors. Additionally, we found that we had to reset the permissions inside of tar and zip files, as the umask didn't affect them on some builds (but not others...)

  8. Zip and Tar reordering and attribute issues

    To aid with this and other issues with reproducibility, we created simple shell wrappers for zip and tar to eliminate the sources of non-determinism.

  9. Timezone leaks

    To deal with these, we set TZ=UTC at the top of our descriptors.

Future Work

The most common question we've been asked about this build process is: What can be done to prevent the adversary from compromising the (substantially weaker) Ubuntu build and packaging processes, and further, what about the Trusting Trust attack?

In terms of eliminating the remaining single points of compromise, the first order of business is to build all of our compilers and toolchain directly from sources via their own Gitian descriptors.

Once this is accomplished, we can begin the process of building identical binaries from multiple different Linux distributions. This would require the adversary to compromise multiple Linux distributions in order to compromise the Tor software distribution.

If we can support reproducible builds through cross compiling from multiple architectures (Intel, ARM, MIPS, PowerPC, etc), this also reduces the likelihood of a Trusting Trust attack surviving unnoticed in the toolchain (because the machine code that injects the payload would have to be pre-compiled and present for all copies of the cross-compiled executable code in a way that is still not visible in the sources).

If those Linux distributions also support reproducible builds of the full build and toolchain environment (both Debian and Fedora have started this), we can eliminate Trusting Trust attacks entirely by using Diverse Double Compilation between multiple independent distribution toolchains, and/or assembly audited compilers. In other words, we could use the distributions' deterministic build processes to verify that identical build environments are produced through Diverse Double Compilation.

As can be seen, much work remains before the system is fully resistant against all forms of malware injection, but even in their current state, reproducible builds are a huge step forward in software build security. We hope this information helps other software distributors to follow the example set by Bitcoin and Tor.

Tor and the Silk Road takedown

We've had several requests by the press and others to talk about the Silk Road situation today. We only know what's going on by reading the same news sources everyone else is reading.

In this case we've been watching carefully to try to learn if there are any flaws with Tor that we need to correct. So far, nothing about this case makes us think that there are new ways to compromise Tor (the software or the network). The FBI says that their suspect made mistakes in operational security, and was found through actual detective work. Remember: Tor does not anonymize individuals when they use their legal name on a public forum, use a VPN with logs that are subject to a subpoena, or provide personal information to other services. See also the list of warnings linked from the Tor download page.

Also, while we've seen no evidence that this case involved breaking into the webserver behind the hidden service, we should take this opportunity to emphasize that Tor's hidden service feature (a way to publish and access content anonymously) won't keep someone anonymous when paired with unsafe software or unsafe behavior. It is up to the publisher to choose and configure server software that is resistant to attacks. Mistakes in configuring or maintaining a hidden service website can compromise the publisher's anonymity independent of Tor.

And finally, Tor's design goals include preventing even The Tor Project from tracking users; hidden services are no different. We don't have any special access to or information about this hidden service or any other. Because Tor is open-source and it comes with detailed design documents and research papers, independent researchers can verify its security.

Here are some helpful links to more information on these subjects:

Technical details of hidden services:

Our abuse FAQ:

For those curious about our interactions with law enforcement:

Using Tor hidden services for good:

Regarding the Freedom Hosting incident in August 2013, which is unrelated
as far as we can tell:

Some general hints on staying anonymous:

The Tor Project is a nonprofit 501(c)(3) organization dedicated to providing tools to help people manage their privacy on the Internet. Our focus continues to be in helping ordinary citizens, victims of abuse, individuals in dangerous parts of the world, and others stay aware and educated about how to keep themselves secure online.

The global Tor team remains committed to building technology solutions to help keep the doors to freedom of expression open. We will continue to watch as the details of this situation unfold and respond when it is appropriate and useful.

For further press related questions please contact us at

Tor Weekly News — October 2nd, 2013

Welcome to the fourteenth issue of Tor Weekly News, the weekly newsletter that covers what’s happening in the much-discussed Tor community.

Tor Browser Bundle 3.0alpha4 released

On September 28th, Mike Perry released the fourth alpha of the new Tor Browser Bundle 3.0 series. The main highlights of this series are the important usability improvements that integrate Tor configuration and control into the browser itself, rather than relying on the unmaintained Vidalia interface.

The latest iteration is based on Firefox 17.0.9esr, which brings with it a lot of important security fixes. It also fixes a fingerprinting issue by randomizing the timestamp sent when establishing an HTTPS connection.

Two small but important usability improvements in the new Tor Launcher component were made: users can now directly copy and paste “bridge” lines from the bridge database, while clock-skews that would prevent Tor from functioning properly are now reported to users.

Download your copy, test it, and report any problems you find. If you're feeling adventurous, you can also try out the crucial new security process by independently reproducing the binaries from the publicly-reviewable source code.

Tor mini-hackathon at GNU 30th anniversary

The Tor mini-hackathon at the GNU 30th anniversary event took place over the weekend, and Nick Mathewson sent out a brief report on how things went. As well as working on proposal 220, which involves improvements to Tor server identity keys, Nick merged some small patches into the Tor mainline branch, and collected promises of several more to come. He also directed a few enquiring minds towards Tor's online community, saying “I hope we’ll be seeing more of some of the folks I talked to on our mailing lists and IRC channels soon”.

Tor Stack Exchange page in private beta

The Tor Stack Exchange page, which reached 100% commitment last week, has now been moved into the ‘private beta’ stage. Runa Sandvik clarified that “the purpose behind it is to ensure that users who committed to the site’s proposal have a chance to start asking and answering questions, as well as help with the initial community building activities that will define and shape the site”. She added that “the more experts who participate in the private beta, the more certain it is that our page will move on to the next stage (i.e. the public beta).”

Fruitful discussions are already taking place: Karsten Loesing wrote to the wider community on the question of what to do about contact information for bridge operators after it was posed on Stack Exchange.

Roger Dingledine put out a call for Tor developers and anonymity researchers to participate in answering questions on the site, adding “Steven, Philipp, Jens, and I can't do it by ourselves.” If you have expert knowledge to contribute, please send an email to to get an invitation!

liballium: Pluggable Transports utility library in C

Yawning Angel announced a new library to ease the task of writing pluggable transports. liballium is a “simple library that handles the Tor Pluggable Transport Configuration protocol. The idea is for this library to be the C/C++ equivalent to pyptlib (and maybe more, depending on how much time I have to work on it).”

The code is available for review featuring “a reasonably well commented example.”

Feel free to follow up with “questions, comments, feedback”!

Tor Help Desk Roundup

Multiple users wrote to the help desk asking for guidance setting up hidden service sites. The most straightforward documentation for hidden services is in the torrc file itself. A more in-depth guide can be found on the Tor Project website. The website also documents how hidden services work. Technical details can be found in the Rendezvous Specification document.

Monthly status reports for September 2013

The wave of regular monthly reports from Tor project members for the month of September has begun. Runa Sandvik released her report first, followed by reports from Damian Johnson, Philipp Winter, Sherief Alaa, and Noel David Torres Taño.

Miscellaneous news

Mike Perry published his new GPG public key, adding: “this new key will be used to sign email from me going forward, and will be used to sign software releases until such time as I get around to creating a second set of keys on a hardware token for that purpose”.

David Fifield updated the Pluggable Transports bundles using the latest Tor Browser Bundle. In order to benefit from the improvements and security fixes, please update!

intrigeri sent a release schedule for Tails 0.21. The first release candidate should be out on October 20th.

Roger Dingledine sent out “a list of criteria to consider when evaluating pluggable transports for readiness of deployment to users”, asking for comments on his initial draft.

If you have the necessary hardware and want to help Tails out, please test two upcoming features: persistent printer settings and support for more SD card readers (the “sdio” type).

This issue of Tor Weekly News has been assembled by harmony, Lunar, dope457, and Matt Pagan.

Want to continue reading TWN? Please help us create this newsletter. We still need more volunteers to watch the Tor community and report important news. Please see the project page, write down your name and subscribe to the team mailing list if you want to get involved!

Tor Browser Bundle 3.0alpha4 Released

The third alpha release in the 3.0 series of the Tor Browser Bundle is now available from the Tor Package Archive:

This release includes important security updates to Firefox. Here is the complete ChangeLog:

  • All Platforms:
    • Bug #8751: Randomize TLS HELLO timestamp in HTTPS connections
    • Bug #9790 (workaround): Temporarily re-enable JS-Ctypes for cache
      isolation and SSL Observatory

    • Update Firefox to 17.0.9esr
    • Update Tor to
    • Update NoScript to
    • Update Tor-Launcher to 0.2.2-alpha
      • Bug #9675: Provide feedback mechanism for clock-skew and other early
        startup issues

      • Bug #9445: Allow user to enter bridges with or without 'bridge' keyword
      • Bug #9593: Use UTF16 for Tor process launch to handle unicode paths.
      • misc: Detect when Tor exits and display appropriate notification
    • Update Torbutton to
      • Bug 9492: Fix Torbutton logo on OSX and Windows (and related
        initialization code)

      • Bug 8839: Disable Google/Startpage search filters using Tor-specific urls

    As usual these binaries should be exactly reproducible by anyone with Ubuntu and KVM support. To build your own identical copies of these bundles from source code, check out the official repository and use git tag tbb-3.0alpha4-build1 (commit d1fad5a54345d9dad8f8997f2f956d3f4fdeb0f4).

    These instructions should explain things from there. If you notice any differences from the official bundles, I would love to hear about it!

Tor Weekly News — September 25th, 2013

Welcome to the thirteenth issue of Tor Weekly News, the weekly newsletter that covers what's happening in the well-heeled Tor community.

Reimbursement of exit operators

In July 2012, Roger Dingledine wrote a post on the Tor blog in which he raised the prospect of offering funding to organizations running fast Tor exit nodes. In so doing, Roger wrote, “we will improve the network's diversity as well as being able to handle more users.” He also announced that donors were already interested in financing such a scheme. Then, in April this year, Moritz Bartl stated that was looking to move away from establishing additional exit nodes, in favor of providing support of various kinds to partner organizations running their own exits.

These plans, and the discussion they provoked, are now about to bear fruit in the form of a financial reimbursement scheme directed at's partner organizations. Moritz wrote again on the the tor-relays list to announce that reimbursements are scheduled to begin at the end of this month, drawn from a one-time donation by the U.S. Government's Broadcasting Board of Governors.

The ensuing debate focused both on the technical aspects of reimbursement — that is, how best to determine the division of funds based on information harvested from the network metrics — and the question of the security issues that could potentially arise from such a scheme.

Moritz specified that currently the only organizations to qualify for reimbursements are those that he personally knows: “so, if you’re interested in becoming a partner, start social interaction with me”, he wrote. Questions or comments regarding these proposals are welcome on the tor-relays list, and further announcements and discussion about the reimbursement system will be published on its dedicated mailing lists.

Tails 0.20.1 is out

Tails saw its 33rd release on September 19th. The most visible change might be the upgrade of tor to version, which should result in faster and more reliable access to the network after the sudden bump in Tor clients.

Among other minor bugfixes and improvements, persistence volumes are now properly unmounted on shutdown. This should prevent data loss in some situations, and avoid a sometimes lengthy pause upon activation.

It also fixes several important security issues. It is recommended that all users upgrade as soon as possible.

New Tor Browser Bundles released

A new set of stable and beta Tor Browser Bundles was released on September 20th. The Tor Browser is now based on Firefox 17.0.9esr and fixes several important security issues.

Queries for the default search engine, Startpage, are no longer subject to its invasive “family filter”. The beta branch also include an updated version of HTTPS Everywhere that no longer causes a storm of requests to, an issue reported by many users after the last release.

Once again, it is recommended that all users upgrade as soon as possible.

Tor mini-hackathon at GNU 30th Anniversary Celebration

Nick Mathewson sent an invitation encouraging everyone to attend the GNU 30th Anniversary Celebration on September 28th and 29th at MIT, Cambridge, MA, USA. Part of the event is a hackathon, and Tor is featured alongside a few other projects. If you want to spend some of the weekend helping the Tor community, sign up on the webpage and come along!

Clock skew: false alarm

Small offsets in system time offer an attractive opportunity for fingerprinting Tor clients. In order to eliminate unnecessary exposure, Nick Mathewson has been working on proposal 222.

Unfortunately, this process introduced a bug into the tor daemon which became apparent after the directory authority named “turtles” was upgraded. The result was that relays started to warn their operators of an implausible clock skew. This was, of course, a false alarm.

The issue was quickly worked around, and fixed properly a few hours later.

Tor Help Desk Roundup

One user contacted the help desk for assistance running torbrowser, an application not affiliated with the Tor Project that attempts to mimic the Tor Browser Bundle. The torbrowser application violates the Tor Project’s trademark, and the Tor Project encourages users to avoid it. Multiple Tor Project developers have contacted SourceForge, which hosts this application’s website, attempting to get the project removed. Andrew Lewman has said that lawyers have now been engaged.

A number of University students continued to contact the help desk to report difficulties circumventing their University’s Cyberoam firewall. These students report being unable to access the Tor network even when using the Pluggable Transports Browser with obfs3 bridges. One person reported success circumventing the firewall when using an obfsproxy bridge on port 443. This issue is ongoing, but a bug report has been filed.

Miscellaneous news

Jacob Appelbaum inquired with VUPEN about the Tor Project having the right of first refusal for Tor Browser bugs, in order to protect users.

The proposed Tor page on Stack Exchange has now reached 100% commitment, and will soon be launching as a live beta. Thanks to everyone who signed up!.

sajolida reported on the latest Tails “low-hanging fruits session”. The date and a tentative agenda for the next online contributors meeting have also been set.

As GSoC entered its final phase, Kostas Jakeliunas reported on the searchable metrics archive, Johannes Fürmann on EvilGenius, and Cristian-Matei Toader on Tor capabilities.

How can we provide Tor users an easy way to verify the signatures on Tor software? Sherief Alaa raised this question on the tor-dev mailing list when asking for comments on plans to write a “small” GUI tool.

This issue of Tor Weekly News has been assembled by harmony, Lunar, dope457, Matt Pagan, and Jacob Appelbaum.

Want to continue reading TWN? Please help us create this newsletter. We still need more volunteers to watch the Tor community and report important news. Please see the project page, write down your name and subscribe to the team mailing list if you want to get involved!

Pluggable transports bundles 2.4.17-beta-2-pt3 with Firefox 17.0.9esr

There are new Pluggable Transports Tor Browser Bundles with Firefox 17.0.9esr. They are made from the Tor Browser Bundle release of September 20 and contain important security fixes.

The bundles contain flash proxy and obfsproxy configured to run by default. If you want to use flash proxy, you will have to take the extra steps listed in the flash proxy howto.

These bundles contain the same hardcoded obfs2 bridge addresses as the previous bundles which may work for some jurisdictions but you are strongly advised to get new bridge addresses from BridgeDB.

New Tor Browser Bundles with Firefox 17.0.9esr

The stable and beta Tor Browser Bundles have been updated with Firefox 17.0.9esr. This release of Firefox has many important security updates and all users are strongly encouraged to upgrade.

The beta version includes an updated HTTPS Everywhere which fixes the problems many users were having with the OCSP meltdown.

Tor Browser Bundle (2.3.25-13)

Tor Browser Bundle (2.4.17-beta-2)

  • Update Firefox to 17.0.9esr
  • Update LibPNG to 1.6.3
  • Update HTTPS Everywhere to 4.0development.12
  • Update NoScript to
  • Remove extraneous libevent libraries (closes: #9727)
  • Enable GCC hardening for Tor
  • Firefox patch changes:
    • - Disable filtered results in Startpage omnibox (closes: #8839)
  • Add missing geoip file to Linux bundle
  • (entry missing from regular changelog)

Syndicate content Syndicate content