Secure Messaging with Onion Services, a How-To

 

This post explores how Tor onion services can be integrated into existing web services, making them more secure. This integration will use the “publish / subscribe” pattern over Tor to trigger re-builds of the txtorcon documentation (which is hosted on an onion service). We will use Tor to transport the published messages so the network-location of the machine hosting the onion service remains hidden.

We will use a messaging system called “Web Application Messaging Protocol” or WAMP. This specifies “routed” Remote Procedure Calls (RPC) and Publish/Subscribe (PubSub) messaging over many transports. This means there is one central “router” to which any number of clients connect to communicate with each other (via RPC or PubSub).

We'll also use the Crossbar.io router for WAMP which recently added native support for Tor. We will connect to an onion service the router listens on for WebSocket connections.

Note: there is a companion blog post to this one which goes into much more technical detail about the specifics of the setup of Crossbar.io and includes a code repository; see my personal blog.

Why This Matters

The txtorcon documentation (clearnet) should be re-built whenever GitHub changes happen, and GitHub provides a WebHook service which can notify you whenever interesting changes happen. These notifications take the form of an HTTP POST.

Ordinarily, accepting such an HTTP request and rebuilding the documentation would be straightforward. However, an onion service is trying to hide its network location – giving GitHub some way to call this directly would reveal its location. Instead, we will run an agent on a public IP that listens for GitHub WebHook requests and forwards them as “publish” events over Tor to the Crossbar.io router. There will be another agent connected to the router that will subscribe to these forwarded GitHub events. This second agent rebuilds the documentation that is served by the txtorcon onion service.

Diagram of the setup 

There are three parts in the above diagram that we will deal with here:

  • configure the Crossbar.io router to work over Tor
  • write the GitHub Agent in Python
  • write the Doc Builder in Python

The Crossbar Router

Crossbar.io is a Twisted application that provides a robust implementation of a “WAMP router”, routing RPC and PubSub messages between various connected clients. It supports many transports (stdin/out, Unix sockets, TCP sockets and WebSockets). For our purposes, we will use WebSockets over Tor, which Crossbar natively supports.

The complete details of Crossbar.io setup are in the `companion blog post`. The result of this is that the `Crossbar`_ router will use the ADD_ONION Tor Control Protocol command to add a new service to a running Tor instance. It will then be listening as a Tor onion service on a WebSocket URI that looks like ws://m6dazoly4sqnoqrm.onion:5000/.

The two clients we will write will then use this address to connect. Because onion URIs are self-certifying (see “step two” of Tor’s onion-services description) each client can be sure they’re talking to the correct Crossbar.io router.

The GitHub (WebHook) Agent

Although Crossbar.io has native support for turning WebHook-style requests directly into WAMP “publish” events, we don’t want the crossbar router itself on a public IP address. (One reason is because the private-key of our onion service would then be on a public machine). Therefore we have a simple service that listens on 443 for HTTP POSTs from GitHub and connects to Crossbar over Tor to the onion service (“GitHub Agent” in the diagrams).

This will do two things:

  • listen on 443 for HTTPS connections from GitHub;
  • and connect to Crossbar via Tor turning any WebHook calls into “publish” events

The documentation builder will subscribe to these topics and trigger builds when the source code changes. The setup will include a WebSocket connection via Tor onion service (speaking WAMP) and a web server listening for incoming GitHub WebHook POST requests.

Again, see the `companion blog post` for the full details; we use a Klein web server and txacme to get a (free!) Let’s Encrypt certificate for our server. So we accept TLS connections and verify the GitHub signatures before publishing any valid events as a WAMP “publish” message.

Building Documents

So now we have: a Crossbar.io router on a private machine and a WebHook agent on a public machine. The Crossbar.io machine is connected only via Tor, with no incoming connections allowed (all Tor connections are outbound).

The next step is to add the agent that actually re-builds the documentation on the machine serving the txtorcon documentation. This machine also is connected to the outside world only via Tor. This agent will connect to Crossbar.io via Tor and listen for “publish” events – that is, the GitHub WebHook announcements that are now being turned into WAMP “publish” messages.

For the full details, see the repository and `companion blog post`. What this agent does is to simply wait for “push” events to the “master” branch and then update the code and run the build commands.

Onions for All

We have set up a Crossbar.io router listening on a `Tor Project` onion service with a publicly-facing component that verifies and forwards GitHub “WebHook” requests. These requests are acted on by a second “builder” component that re-builds the txtorcon documentation. This keeps the network-location of the documentation host unknown.

Anyone can set up an onion service to integrate Tor into their software. Onion services are often considered “for anonymity only”, but they provide a number of other benefits:

  • no incoming connections at all need be allowed on the onion service (in this case the Crossbar.io router machine)
  • clients can be assured they’re talking to the correct host because onion domains are self-certifying (the address is a hash of the public-key of the service) without the need for a centralized Certificate Authority (CA)
  • onion services are end-to-end encrypted, from the client to the service provider and traffic never leaves the Tor network; and
  • onion services are difficult to block.

It’s well-known that onion services are more secure than services on the wider internet. This is why several large public web services like Facebook and DuckDuckGo provide consumers with the option to connect over Tor, even though they have no need of the anonymity feature. If you want to configure your own onion service, start here.

Anonymous

October 10, 2017

Permalink

startpage uses proxy not tor _ as option.
why do you not convince them to add the Tor_option ?
and sks ? experimental onion !
why do you not convince them to add the Tor_option ?
and mailing-list ?
they retain logs & posts & block/censure a lot : morality ? < 0
why do you not convince them to add the Tor_option ?
yes, these webservice must first configure their own onion service and show us on which side they run.

That's impossible, software will always have bugs. If you want to better protect yourself from such compromise situations you'd have to look at Qubes OS. But again, even Qubes can be hacked although it's much much more difficult than your typical browser's RCE because of the modest attack surface.

That connection means to illustrate the connection between the "GitHub Agent" (a Crossbar client) and the Crossbar router. In this case, it is using the "system Tor", but yes there is a Tor client running on the same machine as the Crossbar router.

The "GitHub Agent" is running a HTTPS site as well as the connection to Crossbar -- so GitHub just makes a normal HTTPS POST request which is then turned into a WAMP message (over Tor, to the Crossbar router).

Anonymous

October 11, 2017

Permalink

> Instead, we will run an agent on a public IP that listens for GitHub WebHook requests and forwards them as “publish” events over Tor to the Crossbar.io router.

Would it also have been possible to put up a tor2web instance on a public IP, and to the post-to-publish conversion on the onion side (in crossbar itself, or a separate shim program)?

Crossbar itself does have built-in support for turning WebHook-style ReST requests directly into WAMP "publish" events -- but in this case I wanted the Crossbar instance to only be connected via Tor (and so it couldn't be on a public IP address for GitHub to make the POST to).

Similarly, I believe a tor2web instance could have been made to work as well -- and give GitHub a URL that would go via tor2web.

Yes, it CAN run on Windows desktop, BUT, you have to install Google Chrome.

As far as I am concerned, Google and the NSA are one and the same, making installing and using any Google product or service anathema.

In other words, it would be a cold day in hell before I ever installed any Google product, including and especially Chrome.

Never mind that they are working with the communist Chinese to censor their internet, bust dissidents, and working overtime to bring the same blessings of communism to America.

Here is but one example:
NSA uses Google cookies to pinpoint targets for hacking
https://www.washingtonpost.com/news/the-switch/wp/2013/12/10/nsa-uses-g…

Anonymous

October 11, 2017

Permalink

> clients can be assured they’re talking to the correct host because onion domains are self-certifying

Incorrect. Any powerful computer can use scallion or other tool to generate a private_key of any onion domains.

If A is using scallion to generate "test(.*).onion", B could run the tool to generate "test(.*).onion".
I hope you guys prevent any tool to generate "v3" onion name.

It is true that you can use brute-force to generate domains that look "similar" -- and this is especially true for "vanity" domains as you suggest above. However, I don't think this diminishes from the fact that the domains *are* self-certifying. Bear this in mind when considering your own vanity domain...

Also, "Proposition 224" domains are even longer (and are deployed right now) so this situation will only improve. I mean: it's untenable to expect a human to remember a prop224 domain, so they should "almost always" be cut/pasted or 'remembered' via bookmark (or similar).

I don't understand. Verification is used against the onion URL. Onion URLs are 16 character and each character is at most 6bits but appear to be 5. That is 96chars at most but likely 80bits. 80 and 96bits isn't very strong. Google broke SHA1 which is 160bits I'm sure 80 and 96 would make them laugh.

Yes, the actual (version 2) .onion domains are a truncated hash of the public-key. Proposition 224 (version 3) onions are now deployed and use the full key for the domain (among other security benefits).

I believe that you are correct that it would be feasible for people with "very large" computers to brute-force search for a public/private keypair where the resulting version-2 .onion domain was the same as an existing one. Luckily, there are now version-3 Onion services!

Perhaps someone with more in-depth knowledge of the cryptography here can comment...

i think this is a nice improvement , it takes times & feedback for be certain that it is'unbreakable".
in fact , they use more tricks & cheat than brute force but what will be the next technical tips used by the quantum computer ? i do not know.
They can also built a mirror (fake cluster/virtual/docker & others) of onions , like a 'parallel' net, like a hidden isolated circuit outside of the net but it is another story.
*i tried post on the mailing-list : reserved at registered user.
*an anti-Tor blog is need for all these doubts coming especially from guys who are not u.s resident : fud.
*unfortunately, encryption & usage of a p.c is prohibited in most part of the world.

Tor criticism is certainly welcome, and there's been a lot of "breaking Tor"-type research published; probably one of the mailing-lists would be most appropriate for this. Tor's threat-model doesn't defend against a "global passive adversary" (e.g. some entity that can observe all/most network traffic).

Mixnets may provide a good answer for threat models including such an adversary, but traditionally haven't been fast enough for "interactive" (e.g. Web) use like Tor is. Recent research may have solved this.

I am not a US resident (nor citizen).

Anonymous

October 12, 2017

Permalink

Can you build tor browser by using the Crossbar router so it wil b very difficult to hack by the fbi? can u do tat making tor more secured? .Please reply

Where did you get https://bridges.torproject.org/bridges?transport=meek from? Generally, meek does not work in the same way as "usual" pluggable transports/bridges which is why you should not get meek as one of the options BridgeDB gives you. See: https://trac.torproject.org/projects/tor/wiki/doc/meek for information about that (and how you change the front domain).

Anonymous

October 13, 2017

Permalink

if you are using the government's internet, they can watch you doing it. why would you think otherwise?

Anonymous

October 22, 2017

Permalink

Hello , 2 years ago I was a SR2 vendor and also I did sale on other marketplace's...Now my computer etc is gone by burglary and Now I can't find the TOR chat Forum about TOR and other Stuff...could somebody Help me out ? Kind regards Holland Online aka Jumbo

Join the discussion...

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

8 + 11 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.