Mission: Montreal! (Building the Next Generation of Onion Services)

by asn | May 25, 2016



A few weeks ago, a small group of Tor developers got together in Montreal and worked on onion services for a full week. The event was very rewarding and we wrote this blog post to share with you how we spent our week! For the record, it was our second onion service hackfest, following the legendary Arlington Accords of July 2015.

Our main goal with this meeting was to accelerate the development of the Next Generation Onion Services project (aka proposal 224). We have been working on this project for the past several months and have made great progress. However, it's a huge project! Because of its volume and complexity, it has been extremely helpful to meet and work together in the same physical space as we hammer out open issues, review each other's code, and quickly make development decisions that would take days to coordinate through mailing lists.

During the hackfest, we did tons of work. Here is an incomplete list of the things we did:

  • In our previous hidden service hackfest, we started designing a system for distributed random number generation on the Tor network. A "distributed random number generator" is a system where multiple computers collaborate and generate a single random number in a way that nobody could have predicted in advance (not even themselves). Such a system will be used by next generation onion services to inject unpredictability into the system and enhance their security.

    Tor developers finished implementing the protocol several months ago, and since then we've been reviewing, auditing, and testing the code.

    As far as we know, a distributed random generation system like this has never been deployed before on the Internet. It's a complex system with multiple protocol phases that involves many computers working together in perfect synergy. To give you an idea of the complexity, here are the hackfest notes of a developer suggesting a design improvement to the system:



    Complicated protocols require lots of testing! So far, onion service developers have been testing this system by creating fake small virtual Tor networks on their laptops and doing basic tests to ensure that it works as intended. However, that's not enough to properly test such an advanced feature. To really test something like this, we need to make a Tor network that works exactly like the real Tor network. It should be a real distributed network over the Internet, and not a virtual thing that lives on a single laptop!

    And that's exactly what we did during the Montreal hackfest! Each Tor developer set up their own Tor node and enabled the "distributed random number generation" feature. We had Tor nodes in countries all around the world, just like the real Tor network, but this was a network just for ourselves! This resulted in a "testing Tor network" with 11 nodes, all performing the random number generation protocol for a whole week.

    This allowed us to test scenarios that could make the protocol burp and fail in unpredictable ways. For example, we instructed our testing Tor nodes to abort at crucial protocol moments, and come back in the worst time possible ways, just to stress test the system. We had our nodes run ancient Tor versions, perform random chaotic behaviors, disappear and never come back, etc.

    This helped us detect various bugs and edge cases. We also confirmed that our system can survive network failures that can happen on the real Internet. All in all, it was a great educational experience! We plan to keep our testing network live, and potentially recruit more people to join it, to test even more features and edge cases!

    For what it's worth, here is a picture of the two first historic random values that our Tor test network generated. The number "5" means that 5 Tor nodes contributed randomness in generating the final random value:



  • We also worked to improve the design of next generation onion services in other ways. We improved the clarity of the specification of proposal 224 and fixed inconsistencies and errors in the text (see latest prop224 commits).

    We designed various improvements to the onion service descriptor download logic of proposal 224 as well as ways to improve the handling of clients with skewed clocks. We also brainstormed ways we can improve the shared randomness protocol in the future.



    We discussed ways to improve the user experience of the 55-character-long onion addresses for next generation onion services (compared to the 16-character-long onion addresses used currently). While no concrete design has been specified yet, we identified the need for a checksum and version field on them. We also discussed modifications to the Tor Browser Bundle that could improve the user experience of long onion addresses.

  • We don't plan to throw away the current onion service system just yet! When proposal 224 first gets deployed, the Tor network will be able to handle both types of onion services: the current version and the next generation version.

    For this reason, while writing the code for proposal 224, we've been facing the choice of whether to refactor a particular piece of code or just rewrite it completely from scratch. The Montreal hackfest allowed us to make quick decisions about these, saving tons of time we would have spent trying to decide over mailing lists and bug trackers.



  • We also worked on breaking down further the implementation plan for proposal 224. We split each task into smaller subtasks and decided how to approach them. Take a look at our notes.

All in all, we got crazy amounts of work done and we all have our hands full for months to come.

Finally, if you find these sort of hackfests exciting and you would like to host or sponsor one, don't hesitate to get in touch with us! Contact us at press@torproject.org and we will forward your message to the appropriate people.

Be seeing you!

Comments

Please note that the comment area below has been archived.

May 24, 2016

Permalink

But what 's about the hole in tor bundle (windows only) _ see mozilla vs nsa/justice/government _ ?
Is there a hidden hole in the tbb , a bug in the microcode, an unknown/undetected backdoor, a guilty silence ?
Did your test reveal something like a trace of a suspicious nonrandom behavior ?

May 24, 2016

Permalink

;-)

May 24, 2016

Permalink

"We discussed ways to improve the user experience of the 55-character-long onion addresses for next generation onion services (compared to the 16-character-long onion addresses used currently). While no concrete design has been specified yet, we identified the need for a checksum and version field on them."

What about the distributed onion DNS GSOC project that was never finished, what happened to that?

That project is still ongoing. It's being developed fairly quietly in collaboration with a few others. There were some systemic weaknesses that prompted us to reexamine some parts of the design. The new design should have a much stronger resistance to Sybil and computation attacks.

I have also updated the networking code and replaced raw TCP sockets with JSON-RPC, which involved replacing the massive Boost Asio library with libcurl. This drastically lowers the size of the dependencies, increases reliability, allows asynchronous and out-of-order communication, and reduces the amount of code which means less bugs.

The RNG problem is an issue that I have been stumbling over for the last year or so. I am very excited to hear about this distributed RNG and I will investigate to see if it can be used in OnioNS.

why it is the problem to just feed some local prng data to encrypted stream for egress and feed received data to prng on ingress.

May 24, 2016

Permalink

Will this next-gen onion service be considered strong enough to be approved by the CA/B Forum for deploying non-EV SSL certificates for?

May 24, 2016

Permalink

I hope next generation hidden services are what it takes to bring the Agora people back -- there, I said it! ;)

May 25, 2016

Permalink

> we started designing a system for distributed random number generation on the Tor network. A "distributed random number generator" is a system where multiple computers collaborate and generate a single random number in a way that nobody could have predicted in advance (not even themselves). Such a system will be used by next generation onion services to inject unpredictability into the system and enhance their security.
>
> Tor developers finished implementing the protocol several months ago, and since then we've been reviewing, auditing, and testing the code.
>
> As far as we know, a distributed random generation system like this has never been deployed before on the Internet. It's a complex system with multiple protocol phases that involves many computers working together in perfect synergy.

Awesome!!! This is exactly the kind of innovation we need to enable TP to continue to defend ordinary citizens against the bad guys!

How critical is it to synchronize system times, plus or minus one second?

Can the latest developments in combining two or more sources of poor quality pseudo-random numbers to produce a source of cryptographic quality pseudo-random numbers help?

May 25, 2016

Permalink

Dear tor develop ,will mobile app keyboard likes GOOGL track your typing? we live in heavy censor asia country with lots of hidden carema with passive micophone likes Tabet dala mountain ,can you intereat mobile Vsat for us? the NGN (next generation firewall)(next generation network) can it using different PHY link likes ultra high frequency UHF ,ultra shortwave? or even SATCOM datalink? which cryto process is suggest likes nyse:infrecon trust platform modual 2.0? also any affortable fast cryto switch, cryto router suggest? ASA cost 1mil usd with HA was too expensive with Single sign on , smartcard , can use HDCP content protection with DRM? which manfuacting motherboard with NIC is suggestion? is it rend a VPS offstore middle node or exit node at home with real name cause problem>_< using htc viv Virtual reqlity carema browing .i2p /.cas protect the hidden cqrema from behind ,with next gen tor support 3D printing model ? many thanks

May 25, 2016

Permalink

GREAT work guys i say that loosely in case there are any ladies in the programming part... TOR is the most AMERICAN place on earth. I say that with no disrespect to any other country, I only use America because we are supposed to be the flagship of some sort to freedom... (Yea Right) Thanks for all the hard work.

May 26, 2016

Permalink

>A "distributed random number generator" is a system where multiple computers collaborate and generate a single random number in a way that nobody could have predicted in advance (not even themselves).

diffie-hellman?

Having the time set incorrectly in your OS is dangerous. Your browser can be tricked into accepting expired certificates, for example.

May 27, 2016

Permalink

Great work guys/girls much appreciation for time and creative energy.
All the best!
Freedom fighters..

May 29, 2016

Permalink

Could this be a problem for current or even future Tor network?

http://www.theregister.co.uk/2016/05/27/blue_coat_ca_certs/
Don't panic, says Blue Coat, we're not using CA cert to snoop on you
Symantec and partner say HTTPS certificate-issuing powers used only for testing
Iain Thomson
27 May 2016

> A kerfuffle kicked off this week when it looked as though Blue Coat had been made an intermediate certificate authority, backed by root certificate authority Symantec, in September. This would allow Blue Coat to issue security certs for almost any website it wanted – certificates that would be trusted by browsers and apps on computers, phones and gadgets.
>
> These trusted certs can be used to disguise malicious servers as legit websites; netizens connecting to the systems would think they're using the real deal, but really they're talking to imposters and handing over sensitive information like passwords to strangers. Blue Coat sells network equipment that does just this kind of espionage: the gear intercepts connections to websites and strips the encryption away so secured communications can be monitored. This is useful for corporations that want to keep tabs on their staff at work. Unfortunately, Blue Coat's HTTPS-snooping products have been used by repressive regimes to spy on activists online and quash dissent.
>
> To tear away the encryption and peek inside people's packets, Blue Coat's man-in-the-middle gear masquerades as legit websites – and this is so much easier to pull off when the manufacturer is an intermediate certificate authority because it will have the flexibility to generate trusted certificates as required. It paves the way for seamless surveillance by Blue Coat-built equipment.

Blue Coat told El Reg the cert was "only for testing", but I don't think anyone should simply take their word for it. After Citizen Labs uncovered the role of Blue Coat gear in the cyberwar programs of the current regime in Syria, Blue Coat claimed their equipment had been "pirated", but few activists believe this.

May 29, 2016

Permalink

Next generation or not, I believe that Tor is compromised. All its traffic is recorded which means possibly cracked and decrypted by NSA/CIA its first users and creators. COME BACK TO MOMMY BOY!

It is true that the Snowden leaks confirm that, as many had long suggested, NSA attempts to record and store long term all internet data streams which are encrypted, for example when connecting to an https website. It is true, as the leaks confirm, that NSA hopes to eventually decrypt these data streams when it obtains, by hook or crook or by quantum cryptanalysis or by coercion, the keys needed to decrypt each stream. Possibly decades in the future.

But your conclusion that, as you suggest, everyone should stop using Tor and "come back to momma", does not follow. Indeed, Tor uses perfect forward secrecy precisely to greatly reduce the threat from NSA's ambitions to eventually retroactively decrypt all https protected data streams. PFS doesn't eliminate the threat entirely, but makes it much, much, *much* harder for the bad guys to harm us.

By the way, in recent interviews, former DOJ chief Eric Holder and former NSA and CIA chief Michael Hayden have both stated that they now accept that the Snowden leaks were in the public interest. But Hayden added that he wanted to put Snowden on the USA's "kill list". He was apparently rebuffed. But as the US continues to adopt an increasingly authoritarian, even neo-fascistic government, the danger that political dissidents will be targeted for assassination is increasing. It is probably not a coincidence that US special forces including the SEALS are currently conducting "training missions" in US cities for "direct strike actions", i.e. targeted assassinations.

It is not just Black Lives Matter activists in Brooklyn whose lives are increasingly at risk. The former chief of the US Army's general staff personally commissioned a planning paper, which he personally endorsed, in which the authors strongly hint that Rio and NYC are among a half dozen "megacities" (cities with populations exceeding one million persons) where the Army expects to deploy during the next decade in a "peacekeeping" mission after the collapse of civilian government (or if local governments are unable to limit the activity of drug cartels). So it would appear that expatriate dissidents such as Glenn Greenwald are endangered by the USG's odious and illegal pursuit of state-sponsored global terrorism.

May 30, 2016

Permalink

Hello,
There's something I would like to ask about the circuit my connection goes through Tor. Maybe this is not the appropriate place. Is it just normal that 95% of my connections since some weeks goes through the same node even when I choice "new tor circuit for this site" 15 times? It is the very same node, which is located in the same country from where I access internet.
thanks

Are you talking about the entry node? I think that's expected behavior. Rotating you through more entry nodes increases the chance you'll get a malicious entry node, and since that's the node that actually knows where you are, it's best to keep that risk down.

(Someone please correct me if I'm understanding this wrong.)

Yes, it is the entry node.
"Rotating you through more entry nodes increases the chance you'll get a malicious entry node"
Sure, this is true if the mentioned node is not itself a malicious one.
I am not saying it is. The point is that I just noticed it s the same node all the time few weeks ago. So, I am pretty sure that the entry node was rotating before that.

June 01, 2016

Permalink

If break the Internet connection and reconnect then Tor will crash and no will work. (Whonix tested)

And after hibernation mode no will work.

Fix it all please.

July 01, 2016

Permalink

Consider including a way to better resists traffic analysis from men in the middle (ISP, SigInt):
https://www.mpi-sws.org/~stevens/pubs/sigcomm13.pdf
Aqua: Towards Efficient Traffic-analysis Resistant Anonymity Networks

By grouping peers with identical traffic flows (see pdf) through a Tor Node the traffic achieves better plausible deniability equal to the number of peers in the peer group routing through that (or those series of) Tor Node.

As Tor network works end point traffic analysis (users ISP logs and server logs) have high chance of matching no one else for patterns, which is bad for anonymity and privacy. Grouping this traffic would slow down both in and out traffic for everyone in the peer group but would be an improvement for anonymity against this attack. Even if this was opt in or only for .onion traffic, or only for those running as a node (node+client) it would be an improvement.