performance

Investigating http proxy performance with Tor

phobos — Wed, 19 Aug 2009 14:57:17 -0700

A while ago there was a thread on OR-TALK that devolved into

"why does Tor still ship ancient privoxy?"

and

"why are you shipping polipo with the Tor Browser Bundle instead of current privoxy?"

For those interested, the thread is here, http://archives.seul.org/or/talk/Jul-2009/msg00063.html.

Scott had a good argument for why we should update the bundles to the latest privoxy, and I agree, we should. But then I started thinking about why we needed a proxy at all. Almost all browsers support socks5 direct, isn't that faster than a middleman proxy?

This got me thinking about why polipo is in the TBB, but not the other packages. The TBB "feels faster" when using Tor than using the installed Tor, Vidalia, and Privoxy. However, I couldn't find any actual testing of performance of polipo vs. privoxy vs. socks5 direct.

So I did it myself, in a loose manner. I wanted to quantify "feels faster".

The raw data from all the testing is :

Tamper Data as xml,
proxy config files,
and results in a spreadsheet.

This is all contained in http://freehaven.net/~phobos/polipo-v-privoxy.tar.gz {.asc). There is a README as well. And yes, the ruby script is a quick and dirty hack.

I tested a few scenarios:

1) native polipo and privoxy without using Tor.
2) polipo and privoxy forwarding to Tor localhost:9050.
3) firefox socks5 direct to Tor via localhost:9050.

The summary of results:

Native polipo is 54.5% faster on average than native privoxy. This could be due to polipo's caching, http 1.1 pipelining, and it can serve bits as fast as they come in from the network. Privoxy needs to load the whole page, scan it, and then send it to the client. Even if privoxy filtering is disabled, it still works the same way.
Polipo caching shines with Tor usage. Performance is still about 40% faster with polipo than privoxy. Common images are cached, and served from the memory cache in single-digit millisecond ranges. Privoxy needs to wait for Tor to wholly deliver the bits. Caching is faster, this we know already. However, from a user perspective, it's just faster to load pages.
socks5 in Firefox 3.5.2 did better than I expected. It was about twice as slow as polipo, but still twice as fast as privoxy. I chalk this up to the tor circuit variability more than anything else.
I tried testing a click to a second page to see how much polipo caching helps people reading different pages on the same site. It helps, but not as much as I expected. Polipo ranged from slower than privoxy to 40-100% faster. Too much variability to make a real determination.

Caveats: Testing under tor is highly variable. I used the same circuits for both the polipo and privoxy tests to minimize variability. However, I can't control node load and congestion.

Out of 23 get requests for the Torproject.org/index.html.en, 17 are for the country flags. Perhaps we should load these last at the bottom of the page, or do something else to speed up the torproject page load.

As I was doing this, I kept thinking of other ways to do it better;

time requests and bits between tor, the http proxy, and the browser. How long does each request take to get from the browser, to the proxy, to tor and back across each layer? how much latency does each piece of software add to the request and delivery?
automate testing and let it run on a normal tor client over weeks. This will average out tor network variability and show "typical" user experience.
Pick a sampling of the top 100 websites by visits worldwide and measure their performance with the three methods, fully instrumented as in #1.
Do user experience measurements. Pay/ask/bribe people to sit in front of a computer, video record their browsing and feedback, and ask for a rating of each configuration (socks5, polipo, privoxy, and a placebo).
re-run #2 and run gcov to watch the code paths used in each piece of software, and figure out what can be optimized for performance.
test various "private browsing modes" through tor to see which browser is faster; firefox, safari, chromium, torfox, or torora.
how can we better tune polipo caching dynamically based on system ram config? Does having 1GB of cache provide significant benefits over the default?

I'm sure there are lots of things wrong with my measurements, minimal analysis, and results.

Constructive criticism is welcome.

Roger's HAR2009 talk on Tor performance

arma — Wed, 19 Aug 2009 13:30:38 -0700

Jake, Mike, Karsten, Sebastian, and I attended Hacking at Random last week in The Netherlands. I did a talk on Tor performance challenges — basically walking through the key pieces of the "Why Tor is Slow" document that we wrote in March.

As usual with European hacking cons, they produced a really well-done video just days after my talk. So if you want to get the highlights on what we're doing to speed up Tor and what roadblocks remain, take a look at the video and also the slides that come with it.

On the Recent Growth of the Tor Network

karsten — Mon, 22 Jun 2009 18:07:03 -0700

In the past few days the Tor network is seeing a lot of new users coming from Iran. At the same time we have heard from many people who want to support the Tor network by setting up more relays and bridges. Now we wanted to know, are these just promises, or did the network really grow? Here are the results:

The number of relays has grown by 359 or 24% to now 1827 relays within only one week to the highest number of relays the network has ever seen. Likewise, the number of bridges has increased by an incredible 69 % from 255 to 432 within one week. Awesome!

So, does this mean everything is taken care of and no more relays or bridges are needed? Not at all! As you may know, Tor has some performance issues that are, among other things, the result of too few bandwidth capacity for too many clients. If you can, go setup more relays and/or bridges and keep them running even when the conflicts in the world do not hit the headlines. Be sure that your support is much appreciated!

TorFlow Node Capacity, Integrity and Reliability Measurements at HotPETS

mikeperry — Sun, 21 Jun 2009 02:32:49 -0700

Like Karsten, I too am presenting at HotPETS in Seattle in August. My presentation will cover my work with my TorFlow suite - a python library and utility set to assist measuring and adjusting performance on the Tor network, and to scan the network for malfunctioning and misbehaving exits.

One of the highlights is a system for performance feedback: a set of training wheels of sorts for Tor's load balancing algorithms that once deployed, should greatly improve the performance of the network for 0.2.1 clients. I am very much excited about this and am looking forward to throwing the switch on it in the coming weeks.

Also of note is the discovery that nodes that use Tor's built in rate-limiting are far less likely to fail circuits than those that do not, especially for Windows nodes. So if you are a Windows node operator, please rate limit your node slightly below your connection's capacity using either the Vidalia GUI or the BandwidthRate torrc config option.. I will be emailing the contact info addresses of nodes that are most affected by this soon.

The paper for the presentation describes the library, utilities, and their results in much further detail.

Measuring the Tor Network from Public Directory Information

karsten — Tue, 16 Jun 2009 11:25:31 -0700

On this year's HotPETs workshop (August 5-7 in Seattle, WA, USA) I'm going to present some results on Measuring the Tor Network from Public Directory Information. The main idea is to observe trends in the Tor network without having to measure any data other than public directory information. These data are there anyway as they are required for clients to make good path selection decisions and build circuits. The results of this paper reveal problems in the current Tor network that need to be addressed, e.g., by lowering requirements for assigning certain flags, facilitating the upgrade process, improving support for dynamic IP addresses, possibly calculating bandwidth capacity more reliably, and clarifying legal issues for running relays in view of data retention laws. The next step in understanding the problems of the Tor network requires an extension of network measurements to improve performance and blocking-resistance of Tor.

Performance measurements and blocking-resistance analysis in the Tor network

karsten — Thu, 21 May 2009 13:19:22 -0700

The Tor network has grown to more than one thousand relays and millions of casual users over the past few years. We are proud of our network's popularity, but with growth has come increasing performance problems and attempts by some countries to block access to the Tor network. In order to address these problems, we need to learn more about the Tor network. In this post, I describe the current state of network measurements in Tor and some proposed additions to help us understand the network better.

Right now, relays, bridges, and directories gather the following data for statistical purposes:

Relays and bridges count the number of bytes that they have pushed in 15-minute intervals over the past 24 hours. They include these data in extra-info documents that they send to the directory authorities whenever they publish their server descriptor. See Figure 3 in the analysis of directory archives that shows that roughly half of the available bandwidth capacity is utilized.
Bridges further include a rough number of clients per country that they have seen in the past 48 hours in their extra-info documents. We added this feature in version 0.2.0.13-alpha to help us learn when a given country is trying to block connections to bridges. It also lets us understand bridge adoption better: see for an example an analysis of bridge users by countries.
Directories since version 0.2.1.1-alpha can be configured to count the number of clients they see per country in the past 24 hours and to write them to a local file. We have used these data in the past to estimate total numbers and origins of clients.

It turns out that we need to learn more about the Tor network to make it more useful for everyone. In particular, we are trying to identify performance bottlenecks and want to be ready to notice if any countries start blocking access to the Tor network. Therefore, I am planning to extend network measurements by the following kinds of data:

Entry guards should count the number of clients per country seen in the past 24 hours and include these numbers in their extra-info documents. These data are similar to what bridges already gather about their clients as well as directories if configured accordingly. (Remember, because Tor paths are more than one hop, the entry guard knows you are using Tor but does not know anything about your destinations.) We need country counts from entry guards to learn how many clients are connected to a single entry guard and if there are or start to be any restrictions for clients connecting from specific countries.
Relays should determine statistics about the number of bytes and cells waiting in their local queues and report them to the directory authorities in their extra-info documents. We need to learn more about buffer sizes of relays at various loads to identify current and future performance problems and fix them.
Exit nodes should include the number of bytes and streams they pushed over the past 24 hours broken down by exiting port in their extra-info documents. These data are important for us to identify load-balancing problems with respect to exit policies.

These approaches have been designed so that none of the network data can be used to deanonymize our users. All network data are aggregated before being uploaded to the directory authorities: client addresses are resolved to countries, added up over at least 24 hours, and rounded up to the next multiple of a fixed number (currently 8). All network data should be made available via the directories just as the current statistical data can be obtained from downloading extra-info documents. The details of the network measurements as outlined here will be specified in proposals and discussed on the developer mailing list. You can check out our results on the metrics project page as we make progress.

We are excited to finally start tackling the performance problems and to prepare ourselves to notice when countries start blocking access to the Tor network. We would love to have help from the rest of the research community in discussing safe ways to measure the described network data and to analyze them later on.

Why Tor is slow and what we're going to do about it

arma — Fri, 13 Mar 2009 03:33:01 -0700

I've just finished writing up an explanation of all the various reasons why the Tor network is slow, and what we can do about each. Part of it comes down to design flaws; some of it is that a handful of users are overloading the network; and there's also simply not enough capacity to go around.

Specifically, we've identified six categories of problems to address, and laid out some steps to resolve each of them.

You can read the pdf here:
https://svn.torproject.org/svn/tor/trunk/doc/roadmaps/2009-03-11-performance.pdf

Andrew has also put together a real live press release to go with it.

Now all that remains is to do everything. So if you want to help, or especially if you know any organizations that can help with funding, please help us make this happen!

[Edit, 19 August 2009: you can watch the HAR2009 video explaining our key performance plans, if you aren't a fan of 25-page pdfs.]

Overhead from directory info: past, present, future

arma — Sun, 15 Feb 2009 23:05:16 -0800

A growing number of people want to use Tor in low-bandwidth contexts (e.g. modems or shared Internet cafes in the Middle East) and mobile contexts (start up a Tor client, use it for a short time, and then stop it again). Currently Tor is nearly unusable in these situations, because it spends too many bytes fetching directory info. This post summarizes the steps we've taken so far to reduce directory overhead, and explains the steps that are coming next.

First, what do I mean by "directory info"? Part of the Tor design is the _discovery_ component: how clients learn about the available Tor relays, along with their keys, locations, exit policies, and so on. Tor's solution so far uses a few trusted directory authorities that sign and distribute official lists of the relays that make up the Tor network.

History of v1, v2, v3 dir protocols

Over the years we've had several different "directory protocols", each more bandwidth-friendly than the last, and often providing stronger security properties as well. In Tor's first directory design (Sept 2002), each authority created its own list of every relay descriptor, as one flat text file. A short summary of relay status at the top of the file told clients which relays were reachable. Every Tor client fetched a copy from an authority every 10 minutes.

Tor 0.0.8 (Aug 2004) introduced "directory caches", where normal relays would fetch a copy of the directory and serve it to others, to take some burden off the authorities. Tor 0.0.9 (Dec 2004) let clients download "running routers" status summaries separately from the main directory, so they could keep more up-to-date on reachability without needing to refetch all the descriptors. It also added zlib-style compression during transfer. At this point everybody fetched the whole directory every hour and the running-routers document every 15 minutes.

There were two big flaws with the v1 directory scheme: a security problem and an overhead problem. The security problem was that even though there were three authorities, you just went with the most recent opinion you could find, so a single evil authority could screw everybody. The overhead problem was that clients were fetching a new directory even when very little of it had changed. In Dec 2004 there were 57 relays and the uncompressed directory was 172KB; but by May 2006 we were up to 749 relays and the full directory was almost 1MB compressed. Even though we'd lowered the period for fetching new copies to 2 hours (20 minutes for caches), this was not good.

We introduced the v2 directory design in Tor 0.1.1.20 in May 2006. Each authority now produced its own "network status" document, which listed brief summaries of each relay along with a hash of the current relay descriptor. Clients fetched all the status documents (there were 5 authorities by now) and ignored relays listed by less than half of them. Clients only fetched the relay descriptors they were missing. Once bootstrapped, clients fetched one new status document (round robin) per hour. Peter Palfrader produced a graph of bandwidth needed to bootstrap and then keep up-to-date over a day.

All of the improvements so far were oriented toward saving bandwidth at the server side: we figured that clients had plenty of bandwidth, and we wanted to avoid overloading the authorities and caches. But if we wanted to add more directory authorities (a majority of 5 is still an uncomfortably small number), bootstrapping clients would have to fetch one more network status for every new authority. By early 2008, each status document listed 2500 relay summaries and came in around 175KB compressed, meaning you needed 875KB of status docs when starting up, and then another megabyte of descriptors after that. And we couldn't add more authorities without making the problem even worse.

The v3 directory design in Tor 0.2.0.30 (Jul 2008) solved this last problem by having the authorities coordinate and produce an hourly "consensus" network status document, signed by a majority of the six v3 authorities. Now clients only have to fetch one status document.

Next fixes

Based in part on sponsorship by NLnet to reduce Tor's directory overhead, there are two directions we're exploring now: reducing relay download overhead and reducing consensus networkstatus download overhead.

We've cut relay descriptor size by 60% by moving some of the descriptor info to separate "extra-info" descriptors that clients don't need to fetch, and we've cut the consensus size by 40% by leaving out non-Running relays (since clients won't fetch descriptors for them anyway).

We spent the second half of 2008 working on a much more radical design change where we move all the descriptor components that are used for path selection into the consensus, and then clients would download each relay descriptor "just in time" as part of circuit extension. This design would have two huge benefits: a) clients download zero descriptors at startup, greatly speeding bootstrapping, and b) the total number of relay descriptor downloads would be based on the number of circuits built, regardless of how many relays are in the network.

Alas, we have backed off on this proposal: Karsten Loesing's work on hidden service performance concluded that the circuit extension is the main performance killer, and our "just in time" download proposal would add more round-trips and complexity into exactly that step. So we have a new plan: we're still going to move all the path-selection information into the consensus, but then we're going to put the remaining pieces into a new microdescriptor that will hopefully change at most weekly. That means the initial bootstrap costs will still be there (though the microdescriptor is maybe a third the size of the normal descriptor), but so long as you can keep a disk cache, descriptor maintenance will be reduced to roughly zero. We're aiming to roll this change out in Tor 0.2.2.x, in mid 2009.

We also have plans to further reduce the consensus download overhead. Since the consensus doesn't actually change that much from one hour to the next, clients should fetch consensus diffs rather than fetching a whole new consensus. We could expect another 80% reduction in size here. We hope to roll this step out in mid to late 2009. Alas, this goal is stymied by the fact that we haven't found any small portable BSD-licensed C diff libraries. Anybody know one?

So these two changes together mean an initial bootstrap cost of maybe 100KB+300KB, and then a maintenance cost of maybe 20KB/hour. But actually, once we've gotten the maintenance level so low, we should think about updating the consensus more often than once an hour. The goal would be to get relays that change IP addresses back into action as soon as possible -- currently it takes 2 to 4 hours before a new relay (or a relay with a new location) gets noticed by clients. Since one-third of Tor relays run on dynamic IP addresses, bringing that level down to 30 to 60 minutes could mean a lot more network capacity.

Down the road, there are even more radical design changes to consider. One day there will be too many relays for every client to know about all of them, so we will want to partition the network (see Section 4.4 of Tor's development roadmap). When we do that, we can bound the amount of directory info that each client has to maintain. Another promising idea is to figure out ways to let clients build paths through the network while requiring less information about each relay. There's definitely a tradeoff here between centralized coordination (which is easier to design, and can more easily provide good anonymity properties) and scaling to many tens of thousands of relays. But that, as they say, is a story for another time.

Two incentive designs for Tor

arma — Sat, 17 Jan 2009 12:42:56 -0800

One big challenge to making Tor fast is providing incentives for users to act as relays. So far we've been getting more relays by 1) building community through interacting more with relay operators, listing the fast ones prominently in the Tor status pages, and generally making it clear that you will make the Tor network better if you do, and 2) making it really easy to configure and run a relay by adding a simple GUI interface in Vidalia and adding UPnP support. But we should also consider more direct incentive approaches, for example where Tor is faster for you if you're a relay.

There are two papers that came out in 2008 that everybody pondering incentives in Tor should read. The first is "Building Incentives into Tor", a tech report I coauthored with Johnny Ngan and Dan Wallach from Rice University. The second is "Payment for Anonymous Routing", published at PETS 2008 by Androulaki et al from Columbia University.

The first paper proposes that Tor's directory authorities should spot check relays to make sure they're behaving well, and assign "gold star" flags to the good ones in the networkstatus consensus. Then relays give priority service to connections from people who have gold stars. We ran simulations of the idea for various combinations of users and strategies (selfish, cooperative, adaptive, etc), and showed that in general the performance for gold-star users stays good even with many other users and a heavy traffic load on the network. The other main goal of the design uses an economic argument: not only does it provide the (explicit) incentive to run a relay, but it also aims to grow the overall capacity of the network, so even non-relays will benefit.

However, this incentive design has a serious flaw: the set of gold-star users is public. Over time an attacker can narrow down which relays are always the ones online when a certain activity (e.g. posting to a blog) happens. One fix might be to make the gold star status persist for a few weeks after the relay stops offering service, to dampen out fluctuations in the anonymity sets. But I fear that this narrowing-down attack (also known in research papers as the "intersection attack") is still going to work really well against users who only relay traffic around the times they want good performance.

It would seem that any incentives scheme that treats currently running relays specially will fall prey to this attack. We need to find some way to greatly increase the set of people who might be getting priority service. That's where the Columbia paper comes in: they propose that Tor clients use e-cash (digital coins) to pay for high-priority circuits. The bulk of the paper is in working out how to both a) make sure users can't cheat too often, and b) make sure relays can't use the payments to trace users. They use a hybrid digital cash design, where clients don't mind identifying themselves to the first hop, but then use anonymous digital cash when paying the later hops in the circuit. Most anonymous credential schemes involve way too much computational overhead, so having one that's more practical is a great step.

Because anybody can buy the digital coins, it's not so easy to build a set of suspects when you see somebody use a high-priority circuit. Of course, once real money gets involved, things get more complex. One big problem is the bank. Actually building a centralized place where people turn dollars into bits and back is a daunting exercise, and other projects have learned the lesson that it's hard to get right. Plus there are still unsolved anonymity questions -- if we see Alice use a credit card to buy some anonymous coins, and then a few minutes later some anonymous person spends some anonymous coins, what did we learn?

On top of that are the social implications of adding money into the system. Nick keeps reminding me of sociological studies saying that rewarding volunteers with t-shirts makes them feel good about their contribution, whereas rewarding them with a small amount of cash makes them subconsciously start to value their contribution based on the cash you give them. So they're more likely to stop volunteering, as they don't feel their effort is properly appreciated. More details here, here, and here. It's hard to say how right this research is, but it seems a rough set of variables to add in if we can avoid it.

Beyond that, paying relays introduces other problems. For one, relays now have new incentives to cheat, or to minimize their traffic costs compared to the payments. How do we achieve a good decentralized network if everybody gravitates to the same cheapest hosting provider? Money can even change the legal status of relays in some cases.

So how to proceed? My current idea is a combination of the two designs. The directory authorities give out digital coins in exchange for being a good relay, and the coins can be used to build high-priority circuits. The relays track the coins just enough to prevent too much double-spending (using a coin more than once), and then discard them. Now there is no bank, and no real money involved. It's just a resource management approach.

Will a secondary market appear, where people sell their coins on eBay? Perhaps. Fine with me if so. I think that's a different situation than having the protocol itself designed to transfer dollars from users to relays.

The single-use coins make me uncomfortable, because there's a lot of crypto infrastructure and performance questions in getting all those coins right. Worse, if we're really spending coins for every circuit, we will want to rethink our current "feel free to build a bunch of circuits and just use a few" approach that we're getting even more attached to with proposal 151. In my ideal world we could give out coins (credentials) that could be used as much as you like in a given time period, so we don't need the whole anonymous cash infrastructure. But if somebody posts their credential to Slashdot, I want some way either to revoke it and/or to notice and not give that guy any more credentials in the future, and that seems hard. So it looks like it'll be single-use coins or bust.

Of course, lest I appear too optimistic, there are a few more barriers to getting this right. We need to make sure Tor's network design can scale to make use of many more relays. We've been making some progress lately at decreasing the bandwidth required for directory downloads, but many other aspects of this problem still need to be solved. We also need a better way to actually implement priority circuits: our current approach sometimes accidentally gives high priority to other circuits too. Lastly, we might find that per-circuit accounting is not sufficient to handle the load that some users want to put on the network. If so, somebody will need to start doing design and research on per-byte accounting.

Improvements on Hidden Service Performance -- or not?

karsten — Thu, 15 Jan 2009 12:32:56 -0800

During the past eight months we have been trying pretty hard to improve hidden service performance. This work was part of the project to Speed Up Tor Hidden Services, generously funded by the NLnet Foundation. As of today, we know that we have succeeded in our attempts -- well, or not?

We started in June 2008 by measuring how long it takes until hidden services are registered in the network, how long it takes for clients to connect to them, and similar metrics. An analysis of these data revealed what problems could be responsible for the experienced delays: First of all, we found in total 10 bugs, some of them delaying hidden services significantly. These bugs are now fixed in the 0.2.1.x series (and the major bugs also in the 0.2.0.x series). Second, we identified certain substeps as the bottlenecks of the hidden service protocol, all of them related to building circuits on demand. We proposed a few design changes to overcome these bottlenecks by reducing circuit-build timeouts or by building circuits in parallel. These design changes have been discussed, evaluated, specified, and finally implemented in the 0.2.1.x series.

As of today, we know that we were successful in our attempts -- well, at least in parts. In December 2008, we performed a second set of measurements with a Tor version that had all bugfixes and design changes. Today we finished the analysis of these new data by comparing them with the June data. We found that the time until hidden services are advertised in the network halved from 2:12 minutes to 58 seconds in the mean. This improvement is by far better than we had expected. With this improvement we might even think about reducing an internal descriptor stabilization time from 30 seconds to a lower value, which would further reduce service publication time. However, we also found that the time that clients need to wait while connecting to hidden services remained unchanged at 56 seconds in the mean. Even though our design changes were effective, the distributed storage for hidden service descriptors has counter-balanced all gains achieved here. This deteriorating effect was not expected when we started. In the future, we should therefore focus on speeding up downloads from the distributed hidden service directory, for example by parallelizing requests.

All in all, these improvements are an important step in making hidden services faster. Now we know pretty well what the main problem of hidden services is: on-demand circuit building. Of course, this problem is not specific to hidden services, but a general problem of Tor. Fortunately, hidden services would benefit from all improvements that are made to circuit building in Tor. In the future we are focusing on improvements to circuit building in general.