Making Tor Browser Updates Stable and Reliable with Fastly

by gk | December 14, 2017

tor-fastly
Tor Browser is well-known for its tracking protection and fingerprinting resistance. We have spent a lot of time and energy as well to make the browser bit-by-bit reproducible in order to defend against compromises of our build machines. It is worth mentioning that this includes our update files, too, which are generated during the build process. But having reproducibly built update files, which are properly signed, is only one half of the setup we need: without being able to make them available in a timely and reliable manner our users are not as well-protected as they should be.

Back in 2014 when we introduced the browser updater with Tor Browser 4.0, we hosted the update files ourselves. But it soon became obvious that it would be a challenge to scale our infrastructure to keep up with the user demand and ever growing update sizes (just compare the 50 MB we had on Linux for full updates in 2014 to the 85 MB we have today). Gladly, Fastly stepped up in 2016 to guarantee smooth updates for Tor Browser users. After some experiments, updates were provided over Fastly's infrastructure from June 2016 on. On average roughly 2.1TB/day has been transferred since then, with spikes of over 14TB/day. Thus, a big thank you, Fastly, for hosting our Tor Browser update files for the past 18 months!

We don't know exactly how many Tor Browser users those transferred 2.1TB/day represent, but we started to collect download metrics a while back in order to at least observe trends in user update behavior. If we look at the past 18 months (see image above), we can see that the amount of daily update pings, which check whether an update is available, continually rose up to 2,000,000 in February 2017 and has stayed more or less constant at that level (apart from the outlier between February 2017 and April 2017 which gets investigated in bug 22346). Update requests on the other hand range from 600,000 to 1,200,000 on the day a new Tor Browser stable version gets released (with a sharp drop and a long tail afterwards) which shows the importance of a reliable and robust update infrastructure.

We'd like to emphasize that we've only been using Fastly for Tor Browser updates, which are fetched anonymously over Tor. One reason to focus just on updates is because updates produce the highest bandwidth spikes, which is where Fastly is helpful most to us. Moreover, Fastly does not store customer request logs, so they do not store logs for downloads of Tor Browser updates. This commitment is important as it shows a clear stance towards user privacy which is especially valuable in the update case: an attacker can't show up after the fact and learn update requests and behavior of particular users. This is a good first line of defense which is worth having in a non-update context as well. But Tor Browser takes this a step further by updating its users solely over Tor. Even if there were logs available or someone were able to compromise Fastly's servers, it would not be possible to target a particular Tor Browser user with a malicious update or learn about their update behavior as all users show up as coming from the Tor network on Fastly's systems. This is not only relevant for Tor Browser but is in general an important feature as outlined in an earlier blog post.

Thanks to Fastly we have found a way to solve the stability and reliability issues in our update infrastructure while still protecting our users. However, bandwidth load and update sizes will continue to rise, not only because we will hopefully attract even more users in the future, but also because we deliver Tor Browser to more and more platforms (see for instance the recently started support for 64bit Windows systems). This will likely introduce new scalability challenges given our privacy and security constraints and the need for reliable updates. But we are optimistic to solve them with our friends at Fastly should they arise.

Comments

Please note that the comment area below has been archived.

December 14, 2017

Permalink

[sarcasm] Why didn't you ask your friends over at Cloudflare and Google reCaptcha? I'm sure they would've helped you scale far more reliably than Fastly by making you loose all of your users. [/sarcasm]

:-D :-D :-D Comment of the year award.
Sick of correctly completing ReCaptchas to provide my unpaid labour to train Google's AI that will put me out of a job... only to not even get the respect of a functioning 'Submit' button when using Orfox. This is YEARS that this has been a bug. Google are actively harassing TOR users, in the most passive-aggressive manner, plausible-deniability built-in, of course.

December 14, 2017

Permalink

I wonder how you can reliably scale up by using onion services to distribute updates, I know that update checks are planned, but do you have any idea on how to scale up the updates themselves using onion services?

December 16, 2017

In reply to gk

Permalink

@ gk:

Maybe discuss with Tails Project how Tor Project (or Debian Project?) can help Tails provide iso images for each new edition (i.e. by providing a repository, not via torrents)?

Many thanks for the onion mirrors for Debian!

December 17, 2017

In reply to gk

Permalink

not related to that, but what about choosing randomly what node can become an entry node over a period of time
that can drastically reduce the attack window for us law enforcement

December 14, 2017

Permalink

I have a cybersecurity question about Fastly and TB updates which requires a bit of background.

There has just occurred another highly suspicious BGP incident in which traffic from high-value sites was (apparently with malicious intent) rerouted through this mysterious RU ISP:

AS539523
dv-hyperlink.ru
Vasilyev Ivan Ivanovich
8 Pionerskaya st., office 10
Nekrasovka, Khabarovsk region Russia

The affected traffic included traffic to/from Google, Facebook, Apple, Microsoft, and the illicit BGP changes were picked up by some major US backbone providers including Hurricane Electric in the US and NORDUnet (which has hosted several fast Tor nodes).

> Thus, a big thank you, Fastly, for hosting our Tor Browser update files for the past 18 months!

I think that traffic routed through Fastly would likely be trunked over Hurricane Electric. So a malicious BGP announcement picked up by Hurricane could directly affect TB updates. And according to 2015 Blackhat presentation, a carefully orchestrated BGP hijack could allow an attacker to issue themselves a valid TLS certificate for high value targets such as fastly.com:

http://www.securityweek.com/should-you-be-worried-about-bgp-hijacking-y…
Should You Be Worried About BGP Hijacking your HTTPS?
David Holmes
9 Sep 2015

> A BGP route monitoring firm, Qrator, released a paper at Blackhat 2015 titled “Breaking HTTPS with BGP Hijacking.” I’ll say more about this paper in a little bit, but let’s set up the basics of BGP first... According to the Qrator white paper, with a well-timed BGP hijack, an attacker could issue themselves a real TLS certificate from a real certificate authority. The attacker would advertise that they owned the target’s domain, and request a certificate for that domain from a certificate authority.

Question: if high value websites (such as a website offering legitimate TB updates) used onion sites instead of vanilla https, could this kind of attack be prevented?

If this is a stupid question based upon some misunderstanding of BGP, please explain.

[snip] [I cut the huge background material to make your comment more readable for all users, G.K.]

December 14, 2017

Permalink

For really improving the signature testing on TBB it would be really nice integrate the torproject-signature in TAILS.

Actually, it would be convenient (and potentially reassuring) if Tails and Tor both provided up to date public keys for each other, in case for some reason one offical source is suddenly unavailable to someone.

Obviously, maintaining up to date (and very carefully verified!) copies of public keys of other projects could be a bit tricky, but it might be worth discussing with Tails Project and maybe some others such as Debian and Mozilla.

Points to consider include:

1. By providing copies of keys from other projects, Tails Project would essentially be putting their credibility on the line by saying that these are the real TBB signing keys, and vice versa, so its essential not to make mistakes or to allow the bad guys to maliciously alter anything.

2. Debian Project includes in the Debian repos some packages which hold public keys for some allied projects like Mozilla developers, so this would not be unprecedented.

December 19, 2017

Permalink

How onerous would it be to have exit relays serve updates? Clearly the bandwidth isn't an issue, since the updates are already going through them. The only issue I could see (aside from having to implement it) would be storage, but do people run exit relays on systems without 85 MB to spare?

December 21, 2017

Permalink

Why have not you used IPFS instead? It's more decentralised. But Fastly is a CDN - a global active adversary. Why have you accepted this trojan horse-like gift?

January 06, 2018

Permalink

Fastly , found my other non-tor- browsers and installed a link which
went back to them, then disappeared, I used tcpview.exe and saw this many time and tried to fix
but ended up blocking Fastly servers on my Cisco router, so I have real reservations
that they are helping, tell me they have no moles?????

January 16, 2018

Permalink

A handful of CDNs is like a handful of physical TV distribution networks: playing into the hands of Totalitarians by providing only a few distinct points of pressure in the system.
Now, for example in the UK, we wonder at how supposedly 'independent' TV channels can be pushing complementary propaganda against 'Scrounging' (leeching) Disabled people on welfare.
Which didn't happen before the current (2010 onwards) Government agenda to "compete with the Chinese" (whose human rights and welfare records speak for themselves).
The truth is NOT commonly known and talked-about in the minds of the people of the UK. The propaganda IS. Just one example (and the policy that results from this control and manipulation of the media was condemned at the UN).

https://www.independent.co.uk/news/uk/home-news/government-disabled-peo…

So, using and/or trusting CDNs is arguably helping provide the infrastructure to help our common enemies. Only decentralised control can avoid this. Or, let's have the NSA offer-over some of their vast botnet capacity in the interests of 'freedom', eh? ;-p