A few weeks ago, a small group of Tor developers got together in Montreal and worked on onion services for a full week. The event was very rewarding and we wrote this blog post to share with you how we spent our week! For the record, it was our second onion service hackfest, following the legendary Arlington Accords of July 2015.
Our main goal with this meeting was to accelerate the development of the Next Generation Onion Services project (aka proposal 224). We have been working on this project for the past several months and have made great progress. However, it's a huge project! Because of its volume and complexity, it has been extremely helpful to meet and work together in the same physical space as we hammer out open issues, review each other's code, and quickly make development decisions that would take days to coordinate through mailing lists.
During the hackfest, we did tons of work. Here is an incomplete list of the things we did:
- In our previous hidden service hackfest, we started designing a system for distributed random number generation on the Tor network. A "distributed random number generator" is a system where multiple computers collaborate and generate a single random number in a way that nobody could have predicted in advance (not even themselves). Such a system will be used by next generation onion services to inject unpredictability into the system and enhance their security.
Tor developers finished implementing the protocol several months ago, and since then we've been reviewing, auditing, and testing the code.
As far as we know, a distributed random generation system like this has never been deployed before on the Internet. It's a complex system with multiple protocol phases that involves many computers working together in perfect synergy. To give you an idea of the complexity, here are the hackfest notes of a developer suggesting a design improvement to the system:
Complicated protocols require lots of testing! So far, onion service developers have been testing this system by creating fake small virtual Tor networks on their laptops and doing basic tests to ensure that it works as intended. However, that's not enough to properly test such an advanced feature. To really test something like this, we need to make a Tor network that works exactly like the real Tor network. It should be a real distributed network over the Internet, and not a virtual thing that lives on a single laptop!
And that's exactly what we did during the Montreal hackfest! Each Tor developer set up their own Tor node and enabled the "distributed random number generation" feature. We had Tor nodes in countries all around the world, just like the real Tor network, but this was a network just for ourselves! This resulted in a "testing Tor network" with 11 nodes, all performing the random number generation protocol for a whole week.
This allowed us to test scenarios that could make the protocol burp and fail in unpredictable ways. For example, we instructed our testing Tor nodes to abort at crucial protocol moments, and come back in the worst time possible ways, just to stress test the system. We had our nodes run ancient Tor versions, perform random chaotic behaviors, disappear and never come back, etc.
This helped us detect various bugs and edge cases. We also confirmed that our system can survive network failures that can happen on the real Internet. All in all, it was a great educational experience! We plan to keep our testing network live, and potentially recruit more people to join it, to test even more features and edge cases!
For what it's worth, here is a picture of the two first historic random values that our Tor test network generated. The number "5" means that 5 Tor nodes contributed randomness in generating the final random value:
- We also worked to improve the design of next generation onion services in other ways. We improved the clarity of the specification of proposal 224 and fixed inconsistencies and errors in the text (see latest prop224 commits).
We designed various improvements to the onion service descriptor download logic of proposal 224 as well as ways to improve the handling of clients with skewed clocks. We also brainstormed ways we can improve the shared randomness protocol in the future.
We discussed ways to improve the user experience of the 55-character-long onion addresses for next generation onion services (compared to the 16-character-long onion addresses used currently). While no concrete design has been specified yet, we identified the need for a checksum and version field on them. We also discussed modifications to the Tor Browser Bundle that could improve the user experience of long onion addresses.
- We don't plan to throw away the current onion service system just yet! When proposal 224 first gets deployed, the Tor network will be able to handle both types of onion services: the current version and the next generation version.
For this reason, while writing the code for proposal 224, we've been facing the choice of whether to refactor a particular piece of code or just rewrite it completely from scratch. The Montreal hackfest allowed us to make quick decisions about these, saving tons of time we would have spent trying to decide over mailing lists and bug trackers.
- We also worked on breaking down further the implementation plan for proposal 224. We split each task into smaller subtasks and decided how to approach them. Take a look at our notes.
All in all, we got crazy amounts of work done and we all have our hands full for months to come.
Finally, if you find these sort of hackfests exciting and you would like to host or sponsor one, don't hesitate to get in touch with us! Contact us at email@example.com and we will forward your message to the appropriate people.
Be seeing you!
At the beginning of July, a few of us gathered in Washington DC for the first hidden service hackfest. Our crew was comprised of core Tor developers and researchers who were in the area; mostly attendees of PETS. The aim was to push hidden service development forward and swiftly arrive at decisions that were too tiresome and complex to make over e-mail.
Since we were mostly technical folks, we composed technical proposals and prioritized development, and spent less time with organizational or funding tasks. Here is a snapshot of the work that we did during those 5 days:
- The first day, we discussed current open topics on hidden services and tasks we should be doing in the short-to-medium-term future.
Our list of tasks included marketing and fundraising ones like "Re-branding hidden services" and "Launch crowdfunding campaign", but we spent most of the first day discussing Proposal 224 aka the "Next Generation Hidden Services" project.
- Proposal 224 is our master plan for improving hidden services in fundamental ways: The new system will be faster, use better cryptography, have more secure onion addresses, and offer advanced security properties like improved DoS resistance and keeping identity keys offline. It's heavy engineering work, and we are still fine-tuning the design, so implementation has not started yet.
While discussing how we would implement the system, we decided that we would need to write most of the code for this new protocol from scratch, instead of hooking into the old and rusty hidden service code. To move this forward, we spent part of the following days splitting the proposal into individual modules and figuring out how to refactor the current data structures so that the new protocol can coexist with the old protocol.
- One open design discussion on proposal 224 has been an earlier suggestion of merging the roles of "hidden service directory" and "introduction point" on the hidden service protocol. This change would improve the security and performance as well as simplify the relevant code, and reduce load on the network. Because it changes the protocol a bit, it would be good to have it specified precisely. For this reason, we spent the second and third days writing a proposal that defines how this change works.
- Another core part of proposal 224 is the protocol for global randomness calculation. That's a system where the Tor network itself generates a fresh, unpredictable random value everyday; basically like the NIST Randomness Beacon but decentralized.
Proposal 225 specifies a way that this can be achieved, but there are still various engineering details that need to be ironed out. We spent some time discussing the various ways we can implement the system and the engineering decisions we should take, and produced a draft Tor proposal that specifies the system.
- We also discussed guard discovery attacks, and the various defenses that we could deploy. The fact that many core Tor people were present helped us decide rapidly which various parameters and trade-offs that we should pick. We sketched a proposal and posted it to the [tor-dev] mailing list and it has already received very helpful feedback.
- We also took our old design for "Direct Onion Services" and revised it into a faster and far more elegant protocol. These types of services trade service-side location privacy for improved performance, reliability, and scalability. They will allow sites like reddit to offer their services faster on hidden services while respecting their clients anonymity. During the last days of the hackfest, we wrote a draft proposal for this new design.
- We did more development on OnioNS, the Onion Name System, which allows a hidden service operator to register a human memorable name (e.g. example.tor) that can be used instead of the regular onion address. In the last days of the hackfest we prepared a proof-of-concept demo wherein a domain name was registered and then the Tor Browser successfully loaded a hidden service under that name. That was a significant step for the project.
- We also discussed hidden service statistics and how the two statistics we implemented a few months ago have been very useful. To improve their reliability (since currently only about 3% of the network reports them), we decided to enable them by default in the future.
We also discussed systems for collecting additional statistics in a privacy-preserving manner, using Secure Multiparty Computation or other similar techniques.
- We talked about rebranding the "Hidden Services" project to "Onion Services" to reduce "hidden"/"dark"/"evil" name connotations, and improve terminology. In fact, we've been on this for a while, but we are still not sure what the right name is. What do you think?
- To improve user education, we explored various concepts for a graphical animation explaining hidden services similar in concept to the Tor animation from a few months ago.
And that's only part of what we did. We also wrote code for various tickets, reviewed even more code and really learned how to use Ricochet.
All in all, we managed to fit more things than we hoped into those few days and we hope to do even more focused hackfests in the near future. Email us if you are interested in hosting a hackfest!
If you'd like to get involved with hidden service development, you can contact the hackfest team. Our nicks on IRC OFTC are armadev, asn, dgoulet, kernelcorn, mrphs, ohmygodel, robgjansen, saint, special, sysrqb, and syverson.
Until next time!
JOIN US - Tor Project Boston Hack Day Event - March 20, 2013 - Hosted by Boston University's Department of Computer SciencePosted March 6th, 2013 by kelley
Join us for a unique public hack day event where you will have an opportunity to work in a highly collaborative, interactive environment with Tor's team of technology and research experts. Topics for the day will be determined by the attendees; so bring your ideas, questions, projects and technical expertise with you! Continental breakfast will be provided.
Wednesday, March 20, 2012
9 am until 5 pm
BU Computer Science Dept, 111 Cummington Mall, Boston, MA - ROOM 148
Hosted by Boston University's Department of Computer Science
For more information or questions contact, firstname.lastname@example.org.
Thanks to all who attended and helped make the hackfest in Florence a success. Around 50 people stayed for the two day event. We heard from a team working on a free hardware and software (firmware to drivers) laptop prototype, some Italian legal experts with regards to anonymity, encryption, and chilling effects used to great length in the country, plus a number of Italian hackers from EuroPython 2012 interested in Tor (and vice versa).
On July 5 and 6 we are holding an open hackfest at the Università degli Studi di Firenze in Florence, Italy.
Please attend if you have some interest in programming, advocacy, marketing, or (network security/anonymity/computer science/etc) research with Tor, or are willing to be persuaded to entertain an interest. :) Tor's a small project (in terms of number of developers) that could really use your help.
The majority language will be English, but there will be some Italian speakers at the hackfest.
More details can be found on the Florence Hackfest wiki page.
See you in Florence!
Thanks to the 30+ people that showed up across our hackfests on Wednesday and Thursday. We talked about obfsproxy, torouter, civil liberties and tor, and Andrew spent four-plus hours explaining tor, hidden services, and online privacy risks to a bunch of tor-curious people. It's clear the ice cream sundae/gourmet popcorn bar was a huge success, with people literally squealing their love for Tor (I'm sure it was tor and not the ice cream ;). Thanks to PrivateOnion for writing a song. @INTLRevolver started a twibbon campaign and shared some womens' privacy issues with us and how Tor can help.
And a final thank you to Mel and Kar at the U of Washington for their support, logistics, and general awesomeness overall.
We're having an open hackfest at the University of Washington on Feb 22nd and 23rd; we may hold an additional open hackfest day on Friday, Feb 24th if we feel the demand. This meeting is largely possible due to the support of the UW Security and Privacy Research Lab.
This hackfest coincides with our Winter Developer summit and many Tor developers will be in attendance. As I write Tor developers have already started their travel to Seattle and many will stick around for the following week.
We'd love to welcome everyone interested in attending. We'd especially like people to feel welcome to discuss ideas or proposals, who want to know what's happening in the world of censorship resistance, anonymity, privacy and related topics. Most of all if you're prepared to write software, we're planning to do quite a lot of that next week as well.
I spent two days in Iceland discussing Tor and freedom of information with various people. I talked to a few people, including a member of Icelandic Parliament, about the International Modern Media Institute, http://immi.is/. The goals of IMMI are to secure free speech and defining new operating principles for the global media. They are starting with Iceland and moving on to the world. They already have much success in Iceland, but are running into issues of scale and funding. They could use some help.
The second day I talked to the computer forensics team from the National Police of Iceland about Tor, http://www.logreglan.is/. We discussed all things Tor and their experiences with it. Apparently there are 'computer specialists' traveling Europe talking to law enforcement (for great profit) disparaging any technology that provides security and privacy to citizens as 'for child abuse and organized crime'. These people neglect to mention that all technologies are dual usage and the human behind it determines the good or bad usage of the technology. One of the officers mentions that no one talks about crowbar crime, but everyone is talking about computer crime as if humans aren't involved. Overall, it was a great discussion lasting a few hours.
I then head over to work with the people from 1984, https://1984.is/. They are one of the largest hosting providers in Iceland. And thanks to them, we now have http://torproject.is hosted in the country. I learned more about the physical infrastructure of the Internet in Iceland. We discussed ways to increase competition now that the Iceland Govt bailed out the company that owns nearly all of the fiber in the country. Imagine a country with fiber everywhere (already true in Iceland) and treating it like the road infrastructure with any provider getting access to it. Now mix in successful freedom of expression laws from IMMI.
That night I talked about Tor to the only hackerspace in Iceland at their beer and crypto night at Hakkavélin, http://hakkavelin.is/. Someone showed up and recorded my entire talk until their battery ran out. kapteinnkrokur posted the video at https://www.youtube.com/watch?v=pOayRK48vdE. I covered Tor topics, life under surveillance, and some more advanced topics relating to bridges, ssl filtering, and attempted DHT directory info over Tor. Afterwards, many went out to a bar to talk more until 2 AM. I had a great chat with Bjarni and Ewelina from PageKite, https://pagekite.net/, about Tor marketing, supporting privacy enhancing technology, and peer to peer collaboration for all.
Iceland is a fantastic country and the people are great. I hope to spend more time there, as soon as the volcanoes stop disrupting flights.
Thank you to Björgvin, Birgitta, Berglind, and Mörður for arranging meetings and hosting me for the two days.
Thanks to everyone for attending today. We had some great discussions about The Haven Project, Economic Association for tor relay operators, telecomix, pluggable transports, TAILS, IPv6, sandboxing flash, and of course, tor itself.
And a great bit of gratitude to iis.se for promoting and hosting the hackfest, and for providing food. https://www.iis.se/blogg/hackare-intar-se and the followup, https://www.iis.se/internet-for-alla/reportage/teknikreportage/hackare-i...
We're holding a Tor hackfest on Saturday, May 14th at Ringvägen 100, Stockholm.
We'll be starting at 10 AM . Thanks to https://www.iis.se/ for hosting the event. We're hoping to provide pizza and drinks for lunch. We have a wiki page available to get an idea of what food you want and any topics you wish to see discussed. https://trac.torproject.org/projects/tor/wiki/2011StockholmHackfest
Please attend if you have some interest in programming, advocacy, marketing, or research with Tor, or are willing to be persuaded to entertain an interest. :) Tor's a small project (in terms of number of developers) that could really use your help.
Hope to see you on Saturday!