metrics

Atlas Recent Improvements

Atlas is a web application to learn about currently running Tor relays and bridges. You can search by fingerprint, nickname, country, flags and contact information and be returned information about its advertised bandwidth, uptime, exit policies and more.

I'm taking this opportunity to introduce myself. I'm Iain R. Learmonth, or just irl on IRC. I began contributing to Atlas in June last year, and I'm currently serving as the maintainer for Atlas. We have made some usability improvements to Atlas recently that we are happy to share with you today.

Thanks to the work of Raphael and anonymous contributors for their help in producing patches. We will continue to work through the open tickets, and if you have a feature you would like to see or spot something not working quite correctly, please do feel free to open a ticket for that. If you would like to contribute to fixing some of our existing tickets, we have a new guide for contributing to Atlas.

Improved Error Handling

  • Added a new message to warn users when the Onionoo backend is unavailable [#18081]
  • Added a new message for the case where Onionoo is serving outdated data [#20374]
  • No longer attempts to display AS or geolocation information when it's not available [#18989]

UX Improvements

  • Added tooltips to give descriptions of the meaning for flags [#9913]
  • Made it easy to distinguish between "alleged" and "effective" family [#20382]
  • Removed the graphs for which the data backend will never have any data [#19553]
  • Graphs that have no data, but which may have data in the future, now give a "No Data Available" message [#21430]
  • Relay and bridge fingerprints will now wrap when on smaller screens [#12685]
  • Tooltips are repositioned to avoid them being clipped off on smaller displays [#21398]

Standards Compliance

  • Now HTML 5 compliant according to the W3C Validator (including generated HTML) [#21274]

Tor Browser in numbers

Tor Browser is the secure and anonymous way to browse the web and access onion services. Tor Metrics' new visualization of Tor Browser downloads and updates shows that Tor Browser is downloaded 100,000 times from the Tor website every day! These could be new Tor users or existing users who are downloading it again.

The Signature downloads subgraph shows that between 5,000 and 15,000 users per day tried to verify that Tor Browser was signed by our developers after downloading it. Verifying the signature is the surest way to know that that executable is the legitimate version from Tor and not a benign or malicious third-party one. It is important to increase the number of users that verify their downloads in the future through education and assistance, and knowing the numbers is the first step.

The Update pings subgraph shows ~2,000,000 checks for a new Tor Browser version being made every day. Each running instance of Tor Browser makes a minimum of two such requests per day, and another request at the start of each session. As of now, we don't have any data on how long a typical Tor Browser session lasts or how often users restart their browser. But the update number is still useful to observe trends. For instance, look at the sharp drop of update pings at the end of January. We don't yet know what happened there, though it coincides with the Tor Browser 6.5 release, and the pattern looks similar to what happened when the first version of the 6.0 series was released. We use these graphs to recognize such anomalies, investigate them, and track our explanations here.

Lastly, the Update requests subgraph shows spikes every few weeks with peaks between 750,000 and 1,000,000 requests. This happens when a new Tor Browser version is released, which tells us that automated updates are working!

We sourced the data used above from Tor Project web server logs. Don't worry—we don't record what we do not need (your IP addresses or time of day of requests) and remove potentially identifying information (such as request parameters and the user agent string) before processing. We also delete the original logs afterwards and only keep a sanitized version.

Come back to Tor Metrics often! All of our graphs and tables are updated daily, and we are working to add additional ones in the future. We also encourage you to dig through the data we use and tell us if you find something interesting.

We would like to thank the generous community donations for funding our work. Donations to Tor Project not only help fund new work, but lessen our dependencies on institutions for funding. Keep us independent by donating today!

Metrics Reloaded

in

If you haven’t noticed already, https://metrics.torproject.org/ has a new look. The underlying data, graphing engine, and graphs remain the same.

The goal for this project was to make Tor metrics easier to use and more useful. Our process involved usability inspections, feature brainstorming, rough wireframes, and iterative prototypes. This page documents our process in detail.

We restructured, redesigned, and added content to:

  • Alleviate pain points in using the interface for better workflow and navigation.
  • Aggregate resources for journalists, developers, relay operators, and researchers.
  • Increase compatibility with phones and tablets through responsive design.

It’s truly a place where you can learn interesting facts about the Tor network! We’re especially excited about the news page, which lists various world events with measured anomalies. We hope that the operation, development, and research pages help our many valued Tor community members to find the resources they need. Feel free to email metrics-team@lists.torproject.org with suggestions.

This work was sponsored by Mozilla's Open Source Support. The objectives were to 1) determine the usability of Tor Metrics and 2) address the most pressing usability issues identified (milestone 6.1 and 6.2 of this contract).

Tor's Innovative Metrics Program Receives Award from Mozilla

Good news for data enthusiasts who trust numbers more than words: The Tor Project has just received an award from Mozilla's Open Source Support program to improve Tor metrics over the next 12 months.

While some analytics programs collect data in ways that violate the privacy of users, Tor's metrics program seeks to keep users safe as we collect and analyze data. We use the data to develop ways to allow more people to access the free Internet via Tor, and we make all data available to the world, so that Tor users, developers, journalists, and funders can see and understand the ways that people use Tor worldwide.

Mozilla's mission is to ensure the Internet is a global public resource, open and accessible to all. Mozilla Open Source Support (MOSS) is an awards program specifically focused on supporting the Open Source and Free Software movement. Their Mission Partners track is open to any open source/free software project undertaking an activity which significantly furthers Mozilla's mission.

Over the coming year, our main goals for this project will be:

1. To make CollecTor (our primary data collection service) more resilient to single-point failures, by enabling multiple CollecTor instances to gather data independently and exchange it in an automated fashion. Doing this will reduce the number of gaps in our data, and make it less likely that an error at one server will make the data invalid.

2. To create an easy-to-use observation kit containing DescripTor (our library for parsing and analyzing Tor servers' descriptions of themselves) together with user-friendly tutorials for evaluating Tor network data. This will make it easier for programmers to write tools that examine historical and current data about the servers that make up the Tor network.

3. To set up more instances of the network status service Onionoo to improve its availability, and work on the most pressing usability issues of the Atlas network status service;

4. To further reduce the amount of sensitive usage data (such as bandwidth totals and connections-per-country) stored on Tor relays and reported to the Tor directory authorities. While we believe that this data is safe the way we handle it today, we believe that improved cryptographic and statistical techniques would allow us to store and share even less data.

5. To improve the accuracy of performance measurements by developing better methods and tools to analyze and simulate average user behavior;

6. To make the Tor Metrics website more usable, so that users, developers, and researchers can more easily find, compare, and interpret information about Tor's usage and performance.

We're excited about this news for a great many reasons.

First, it is one more important step in diversifying Tor's funding.

Second, while the project focuses on improving six important aspects of Tor metrics, it also aims at more general improvements to make Tor metrics software more stable, scalable, maintainable, and usable. These improvements are typically harder to "sell" in funding proposals because their results are less visible to funders. It's reassuring that Mozilla understands that these improvements are important, too.

Third, this award is the first one awarded to Tor's young metrics team, only established 12 months ago in June, 2015. It's an appreciation of the initial work done by the metrics team and a very good basis for the upcoming 12 months.

Writing the award proposal was a successful cooperation of a number of Tor people: it would simply not have happened without Isabela, who made contact with Mozilla people; it would not have been readable without Cass's remarkable ability to translate from tech to English; it would not have contained as many good reasons for getting accepted without iwakeh's invaluable input; and it would not have been accepted without Shari's efforts in asking a leading security expert to write an endorsement of our award request. Finally, this blog post would certainly not have been as readable without Kate's and Nick's editorial capabilities. And now let's go write some code.

10 years of collecting Tor directory data

Today is the 10th anniversary of collecting Tor directory data!

As the 2004 Tor design paper says, "As of mid-May 2004, the Tor network consists of 32 nodes (24 in the US, 8 in Europe), and more are joining each week as the code matures."

In fact, we still have the original relay lists from back then. The first archived Tor directory dates back to May 15, 2004. It starts with the following lines which are almost human-readable:

signed-directory
published 2004-05-15 07:30:57
recommended-software 0.0.6.1,0.0.7pre1-cvs
running-routers moria1 moria2 tor26 incognito jap dizum
  cassandra metacolo poblano ned TheoryOrg Tonga
  peertech hopey tequila triphop moria4 anize rot52
  randomtrash


As of today, May 15, 2014, there are about 4,600 relays in the Tor network and another 3,300 bridges. In these 10 years, we have collected a total of 212 GiB of bz2-compressed tarballs containing Tor directory data. That's more than 600 GiB of uncompressed data. And of course, the full archive is publicly available for download.

Here's a small selection of what people do with this fine archive:

If people want to use the Tor directory archive for their research or for building new applications, or want to help out with the projects listed above, don't hesitate to contact us!

Happy 10th birthday, Tor directory archive!

Shadow v1.6.1 released, adds multi-threading support

Rob just tagged Shadow v1.6.1, and added a link on the download page. This is really more like a pre-1.7.0 release, but he wanted to get out some exciting new support for running multi-threaded simulations earlier!

This release includes:

  • Support for running multi-threaded simulations! (use the “-w” flag to specify the number of worker threads)
  • A few bugfixes

In preliminary testing, the biggest improvements have been seen when using between 4 and 8 worker threads. Happy simulating!

March 2012 Progress Report

Our progress report for March 2012 is now available. Highlights include lots and lots of metrics work, bridge infrastructure work, new tor alpha release, support queue stats, and some press and speaking slots.

Available as a pdf with full color graphs, https://archive.torproject.org/monthly-report-archive/2012-March-Monthly...

or as a plain text file for portability and readability, https://archive.torproject.org/monthly-report-archive/2012-March-Monthly...

January 2012 Progress Report

Our progress report for January 2012 is available now. Highlights include two Tails releases, a summary of support calls for the past six months with actual user stories, a trip to Egypt for the 'Change your world' summit, updated metrics codebase, discussion of a new voting method, and lots of translation updates.

Available as a pdf with full color graphs, https://archive.torproject.org/monthly-report-archive/2012-January-Month...

or as a plain text file for portability and readability, https://archive.torproject.org/monthly-report-archive/2012-January-Month...

January 2011 Progress Report

New releases

Design, develop, and implement enhancements that make Tor a better
tool for users in censored countries.

Architecture and technical design docs for Tor enhancements
related to blocking-resistance.

Hide Tor's network signature.

Grow the Tor network and user base. Outreach.

Bridge relay and bridge authority work.

  • Karsten did some work to publish sanitized bridge pool assignments. We're
    going to publish the information which distribution pool a bridge is assigned
    to. The distribution pool defines whether we're giving out bridges via HTTP,
    via email, or not at all (reserved pool). The plan is to remove all sensitive
    information from bridge pool assignments before making them available on
    https://metrics.torproject.org/data.html. The discussion was started on
    the or-dev list
    at http://archives.seul.org/or/dev/Jan-2011/msg00033.html.

Scalability, load balancing, directory overhead, efficiency.

  • We released an updated version of Tor
    Weather, https://weather.torproject.org. Tor Weather is a web application used
    to allow tor relay operators to sign up for notices when their relay is offline,
    drops below a threshold of bandwidth served, and receive notifications when a
    new version of tor is released. This version of the web application was written
    by the Wesleyan University Humantarian Free and Open Source Software (HFOSS)
    team working on Tor for their summer project, http://hfoss.wesleyan.edu/.
  • Karsten started improving metrics-db performance, so that it can scale to
    five years of data with 10K relays and 5K bridges. This included a few tricks
    to avoid parsing the same data twice. Also changed the database schema to use
    SQL arrays to store bandwidth histories, which is apparently a less used part of
    PostgreSQL, because he found a confirmed bug in PostgreSQL 8.2 (released
    2006-12-05).
  • Karsten found two major, if not blocking, bugs in Torouter when run on the
    suggested Buffalo hardware. The Excito hardware does not have these problems.
    The bug numbers are 2334,
    https://trac.torproject.org/projects/tor/ticket/2334, and 2376,
    https://trac.torproject.org/projects/tor/ticket/2376.
  • Karsten found and fixed a problematic bridge sanitizer bug that made us
    keep original IP addresses in reject lines. Updated metrics-db and sanitized
    all bridge descriptors since May 2008 once again. The latter kept two of our
    computers busy for 2.5 weeks.
  • Karsten started with exporting bridge pool assignments and restarted
    discussion about preserving hashed IP addresses in bridge descriptors.
  • Karsten upgraded Torperfs to output information about which circuits they
    used for measuring download times. Made data available on metrics website.
    Added new graphs combining all Torperf sources and showing the fraction of
    timeouts and failures. Started Torperfs with custom entry guard selection
    strategies.
  • Karsten talked to Björn Scheuermann and Florian Tschorsch about
    performance improvements in Tor. Working on a patch with them to be included in
    Tor 0.2.3.x.
  • Karsten improved graphs on metrics-web by adding more countries and by
    allowing users to customize the graph image resolution.

More reliable download mechanism.

  • Sebastian and Erinn started to tackle Thandy and Hudson work. They solved
    the Hudson issue on Windows and made a good deal of progress on getting Thandy
    set up, understanding the different roles and responsibilities of each in the
    Thandy system. Installing files by copying into the right place works, but the
    packaging db that would be required for TBB is not yet working.

Translations

  • Updated translations for the following languages: af ak am arn ast be bg bn
    bn_IN csb cy dz eo eu fil fur ga gl gun ha he hi ht hu is it km kn kw lb ln lo
    lt lv mg mi mk ml mn mr ms mt nah nap ne nn nso oc pa pap pms ps sco son sw ta
    te tg th ti tk uk ur ve wa zh_HK zu.

Measuring Tunisian Tor Usage

I've been following the crackdown on Internet freedom in Tunisia over the past few weeks. I run an unpublished tor bridge for some Tunisian activists. It's been used fairly well for the past 18 months, however it has not seen traffic for a week. Tor usage from Tunisia has never been very high, but for those who need it, it's been a lifesaver; or so I've been told. An example of what's going on in the country: Slim Amamou arrested, Global Voices' coverage of Tunisia, Global Voices Advocacy: Tunisia, and Tunisia's bitter cyberwar.

Out of interest, I wondered how Tor usage in Tunisia has fared over 2010. I wonder if Facebook, Twitter, and other social network services are seeing an increase of users logging into Tunisian social networks from Tor.

It appears we're having an impact in Tunisia. More help is needed.

This is graph of Tor clients directly connecting to the rest of the network:

This is a graph of Tor clients connecting through bridges to the rest of the network:

Syndicate content Syndicate content