Tor Weekly News — September 24th, 2014

Welcome to the thirty-eighth issue in 2014 of Tor Weekly News, the weekly newsletter that covers what’s happening in the Tor community.

The EFF concludes its 2014 Tor Challenge

As Tor Weekly News reported in June, over the last few months the Electronic Frontier Foundation has been holding its second Tor Challenge to improve the strength and diversity of the Tor network by inspiring people to run Tor relays. The 2014 Challenge is now over, and Rainey Reitman of the EFF posted some thoughts on the campaign and its outcome.

1635 Tor relays (including 326 exit relays) were started up or had their capacity increased as part of the 2014 Tor Challenge, compared to 549 at the end of the last campaign in 2011. As Rainey wrote, this number “far exceeded our hopes”; the success can be attributed to a coordinated promotional effort by the EFF, the Free Software Foundation, the Freedom of the Press Foundation, and the Tor Project, as well as to “the 1,000 individuals who cared enough to help contribute bandwidth to the Tor network.” Thanks to everyone who participated!

It’s important to remember, though, that new relays only benefit Tor users as long as they stay running. Advice and support from experienced relay operators can always be found on the #tor IRC channel or the tor-relays mailing list; if you missed out on the Tor Challenge this year but still want to contribute to a stronger, more stable Tor network, take a look at the Tor website for advice on how to get started.

Guardiness and Tor’s directory authorities

When a Tor relay is first assigned the Guard flag by the directory authorities (or “dirauths”) it sees a dip in the amount of traffic passing through it, because Guard capacity is a scarce resource on the Tor network and, as Roger Dingledine explained last year, “all the rest of the clients back off from using you for their middle hops, because when they see the Guard flag, they assume that you have plenty of load already from clients using you as their first hop”, an assumption which is only correct after clients have had enough opportunity to select the new guard. With the recent move to single entry guards, an even longer period of time may pass before a young guard can be selected as a first hop by old clients.

“Guardiness”, or GuardFraction, is a proposed measurement to let dirauths, and therefore clients, work out how much of a relay’s capacity is being used for first hops by clients, and how much for second and third hops, by finding the fraction of recent consensuses in which the relay has been given the Guard flag; the “dead period” following the assignment of the flag can then be avoided. George Kadianakis published an analysis of ways in which dirauths’ votes could be extended to include this guardiness measurement, taking into account the time and effort required to parse large numbers of Tor consensuses very quickly. The initial proposal was to ask dirauths to run a script each hour that would extract the data required for parsing into “summary files”: Sebastian Hahn asked how this measure might fail in different situations, and Peter Palfrader suggested that loading every consensus into a database for later querying might be more efficient.

“This feature is by far the trickiest part of prop236 (guard node security) and I wanted to inform all dirauths of our plan and ask for feedback on the deployment procedure”, wrote George. If you have any comments to add to the discussion so far, please send them to the tor-dev mailing list.

Miscellaneous news

The Tails team wants to make sure that all the Debian packages on which Tails relies are “in good shape” before Jessie, the next Debian release, is frozen on 5th November. To that end, the team called for testing both of the software itself and its translations — if you’d like to help, find full instructions and links to the “barely-working” experimental disk images in the announcements.

meek, the pluggable transport that routes Tor traffic through platforms which are “too big to block”, now works with Microsoft Azure in addition to the already-supported Google App Engine and Amazon Web Services. David Fifield posted the announcement, which contains instructions for those who want to start using the new front domain.

Sebastian Hahn announced that gabelmoo, the Tor directory authority which he administers, has moved to a new IP address. “You should not notice any kind of disturbance from this, and everything should continue to work as normal.”

Released in December 2013, the SafePlug is a $49 router that promises its users “complete security and anonymity” online by sending all of their traffic through Tor. Annie Edmundson from Princeton University released a summary of research presented during FOCI’14 in which the authors point out several security problems in the implementation of the SafePlug administration interface, and also highlight other structural issues. “The most crucial problem with a torifying proxy is that it uses a bring-your-own-browser system, as opposed to a hardened browser, and therefore is susceptible to browser-based privacy leaks. This is why it’s better to use the Tor Browser Bundle […]”, wrote Annie.

The upcoming Tor Messenger is based on Instantbird. One key feature that was identified as missing in the latter is support for Off-the-Record encryption. After months of discussions and reviews to determine the right programming interface, Arlo Breault got the necessary core modifications merged.

Roger Dingledine wrote up a walkthrough of the controller events you might see when accessing Tor hidden services. “In theory the controller events should help you understand how far we got at reaching a hidden service when the connection fails. In practice it’s a bit overwhelming”.

In the first message posted to the recently-created onionoo-announce mailing list, Karsten Loesing explained a minor improvement that should allow Onionoo clients to determine when they need to be upgraded to a new protocol version.

Leiah, whose design work has featured on many of Tor’s company publications, posted a mock-up of a possible new look for the Tor blog.

Patrick Schleizer announced the release of version 9 of Whonix, the anonymous operating system based on Tor, Debian, and security-by-isolation.

Tor help desk roundup

The help desk has been asked how to configure a VPN to prevent a website from learning that a user is using Tor. We consider positioning a VPN between one’s exit node and the destination site to be totally unsafe, and not much more anonymous than using a VPN without Tor. By design, Tor allows the destination site to know that a visitor is using Tor. The better solution is to email the website owner and ask them to stop blocking Tor. The longer-term solution is that Tor needs someone willing to coordinate with websites to design engagement solutions that work for Tor users and for big websites.

News from Tor StackExchange

Jobiwan has a machine on their network which should act as a SOCKS proxy. When Tor Browser is configured to use this proxy, it complains that Tor is not working in this browser. However, Jobiwan is able to visit hidden services with these settings, and wants to know why this message is printed and if it is safe to use Tor Browser this way. Do you know a good answer to this question? If so, please share your thoughts.

Andy Smith asks if slow relays are useful for the Tor network. Roya suggests that a large number of slow relays is better than a small number of fast relays, at least anonymity-wise, because this helps to grow diversity in the network and makes it harder for an attacker to deanonymize users. On the other hand, user194 and Relay Operator write that a slow relay does not provide much benefit for the network. They recommend spending a few dollars more to rent a fast virtual server.

Easy development tasks to get involved with

The tor daemon has a SafeLogging configuration option that removes all potentially sensitive parts of log messages and replaces them with “[scrubbed]”. However, this option does not cover hidden services operated by the tor daemon. Extending this option involves scanning through some code, but Nick says it could be some interesting code; if you’re up to reading and patching some C code and then reading some (hopefully scrubbed) logs, this ticket may be for you. Be sure to post your branch for review on the ticket.

This issue of Tor Weekly News has been assembled by harmony, Lunar, qbi, Matt Pagan, Karsten Loesing, Arlo Breault, and Roger Dingledine.

Want to continue reading TWN? Please help us create this newsletter. We still need more volunteers to watch the Tor community and report important news. Please see the project page project page, write down your name and subscribe to the team mailing list if you want to get involved!

Tor Weekly News — September 17th, 2014

Welcome to the thirty-seventh issue in 2014 of Tor Weekly News, the weekly newsletter that covers what’s happening in the community around Tor, the anonymity network that makes full use of its library card.

tor is out

Nick Mathewson announced the first release candidate in tor’s 0.2.5.x series. This version “fixes several regressions from earlier in the 0.2.5.x release series, and some long-standing bugs related to ORPort reachability testing and failure to send CREATE cells”; relay operators running it will also receive a warning if they try to configure a hidden service on the same process as their relay, as the public nature of much information about Tor relays can help identify services running on the same machine. As ever, you can read the full list of improvements and fixes in Nick’s announcement, and download the source code from the Tor Project’s distribution directory.

Tor protects library patrons’ right to privacy

April Glaser and Alison Macrina published an article for BoingBoing on efforts by Massachusetts librarians to guarantee their patrons’ right to access information without fear of surveillance or censorship. Macrina and her colleagues, in partnership with the ACLU of Massachusetts, have been giving workshops on the use of privacy-preserving technologies to other librarians, and spreading the word about the risk that pervasive surveillance poses to freedom of thought and intellectual inquiry.

As the authors remark, “it’s no secret that libraries are among our most democratic institutions. Libraries provide access to information and protect patrons’ right to explore new ideas, no matter how controversial or subversive […] and protecting unfettered access to information is important whether that research is done using physical books or online search engines. But now it has become common knowledge that governments and corporations are tracking our digital lives, and that surveillance means our right to freely research information is in jeopardy”.

Tor and Tails are a natural fit for any response to this problem, and BoingBoing reports that not only have “multiple Massachusetts libraries […] installed the Tor browser on all of their public PCs” following the workshops, some have even “set up Tor middle relays on their libraries’ networks”.

It would be a shame, however, if these exciting developments were restricted to the state of Massachusetts. If you are a library user concerned about this issue, share the article with your local librarians. If you work in a library, contact the authors of the article at the addresses they provide to find out how you can offer privacy workshops and tools to your own community!

Hidden service enumeration and how to prevent it

When a Tor user wants to connect to a hidden service, their client makes a request over the Tor network to a relay acting as a “hidden service directory”, or HSDir. In return, the client receives a hidden service “descriptor” containing the information necessary for a connection to be made, including the set of Introduction Points that the hidden service is currently using.

Hidden services would ideally not be discoverable unless the address is public or has been shared directly, but one of the weaknesses of the current protocol is that hidden service directories know which services they are serving descriptors for; this same shortcoming was an element of the “RELAY_EARLY” traffic confirmation attack discovered in July. Although the full set of descriptors is not published to all directories at once — at any given time, six directories are responsible for a service’s descriptor — the list is rotated frequently, so it would not be hard for an adversary to run a relay stable enough to gain the HSDir flag, and harvest hidden service addresses as they are uploaded to it in turn.

Fabio Pietrosanti informed the tor-talk mailing list of an experiment designed to detect this enumeration of hidden services: he set up thirty new hidden services, keeping their addresses secret, with each service running a script to report any attempts at access from outside. As the existence of these services was not disclosed to anyone, any requests to the service could only come from a client that had obtained the address from a directory which had previously held the descriptor, possibly “a malicious Tor relay acting as a TorHS directory, with Tor’s code modified to dump from the RAM memory the TorHS list, then harvest them with an http client/script/crawler”. After approximately a month, according to Fabio’s message, a client did indeed try to access one of the “honeypot” services.

Regular readers of Tor Weekly News will know that the hidden service protocol is being fully redesigned, and this “next-generation” proposal already suggests defenses against this kind of attack, but (as ever) more eyes are needed. If you’re interested, see George Kadianakis’ introduction to the issues facing hidden services; those familiar with cryptography in C are welcome to review the discussion of this particular issue on the bug tracker.

Miscellaneous news

Nathan Freitas announced version of Orbot, the Tor client for Android. The highlights of this release are an upgrade to tor (see above), which solves an issue with the “airplane mode” feature, as well as a number of improvements to do with transparent proxying. Find the full changelog and download links in Nathan’s message.

Juha Nurmi described the current state of, the search engine for hidden services, following a successful Google Summer of Code project. The post includes notes on the design, content statistics, and plans for future work.

David Fifield called for a volunteer operating a “big fast bridge” to take over the running of the meek pluggable transport: “I want to do this both to diffuse trust, so that I don’t run all the infrastructure, and because my bridge is not especially fast and I’m not especially adept at performance tuning”.

David also wondered why the number of FTE users appeared to dip in late August, and explored possible reasons behind the correlation in usage statistics for meek and Flashproxy, whose backends both run on the same bridge. Karsten Loesing suggested that the latter was because “we’re counting consensuses downloaded from a bridge via any supported transport, and then we’re attributing those downloads to specific transports based on what fraction of IPs connected per transport”.

Tim reported on progress made towards a “fuzzer” for Tor, based on the Tor research framework previously announced by Gareth Owen, including a draft design for the process and a list of patches against Tor made during development.

Matt Pagan submitted his status report for August, while Roger Dingledine sent out the report for SponsorF.

Karsten Loesing posted the minutes of last week’s Globe/Atlas developer IRC meeting.

This issue of Tor Weekly News has been assembled by harmony, Lunar, Roger Dingledine, George Kadianakis, and special.

Want to continue reading TWN? Please help us create this newsletter. We still need more volunteers to watch the Tor community and report important news. Please see the project page, write down your name and subscribe to the team mailing list if you want to get involved!

Tor is out

Tor fixes several regressions from earlier in the 0.2.5.x release series, and some long-standing bugs related to ORPort reachability testing and failure to send CREATE cells. It is the first release candidate for the Tor 0.2.5.x series.

The tarball and signature file are currently available from
and packages and bundles will be available soon.

Changes in version - 2014-09-11

  • Major bugfixes (client, startup):
    • Start making circuits as soon as DisabledNetwork is turned off.
      When Tor started with DisabledNetwork set, it would correctly
      conclude that it shouldn't build circuits, but it would mistakenly
      cache this conclusion, and continue believing it even when
      DisableNetwork is set to 0. Fixes the bug introduced by the fix
      for bug 11200; bugfix on

    • Resume expanding abbreviations for command-line options. The fix
      for bug 4647 accidentally removed our hack from bug 586 that
      rewrote HashedControlPassword to __HashedControlSessionPassword
      when it appears on the commandline (which allowed the user to set
      her own HashedControlPassword in the torrc file while the
      controller generates a fresh session password for each run). Fixes
      bug 12948; bugfix on

    • Warn about attempts to run hidden services and relays in the same
      process: that's probably not a good idea. Closes ticket 12908.
  • Major bugfixes (relay):
    • Avoid queuing or sending destroy cells for circuit ID zero when we
      fail to send a CREATE cell. Fixes bug 12848; bugfix on 0.0.8pre1.
      Found and fixed by "cypherpunks".

    • Fix ORPort reachability detection on relays running behind a
      proxy, by correctly updating the "local" mark on the controlling
      channel when changing the address of an or_connection_t after the
      handshake. Fixes bug 12160; bugfix on
  • Minor features (bridge):
    • Add an ExtORPortCookieAuthFileGroupReadable option to make the
      cookie file for the ExtORPort g+r by default.
  • Minor features (geoip):
    • Update geoip and geoip6 to the August 7 2014 Maxmind GeoLite2
      Country database.
  • Minor bugfixes (logging):
    • Reduce the log severity of the "Pluggable transport proxy does not
      provide any needed transports and will not be launched." message,
      since Tor Browser includes several ClientTransportPlugin lines in
      its torrc-defaults file, leading every Tor Browser user who looks
      at her logs to see these notices and wonder if they're dangerous.
      Resolves bug 13124; bugfix on

    • Downgrade "Unexpected onionskin length after decryption" warning
      to a protocol-warn, since there's nothing relay operators can do
      about a client that sends them a malformed create cell. Resolves
      bug 12996; bugfix on 0.0.6rc1.

    • Log more specific warnings when we get an ESTABLISH_RENDEZVOUS
      cell on a cannibalized or non-OR circuit. Resolves ticket 12997.

    • When logging information about an EXTEND2 or EXTENDED2 cell, log
      their names correctly. Fixes part of bug 12700; bugfix

    • When logging information about a relay cell whose command we don't
      recognize, log its command as an integer. Fixes part of bug 12700;
      bugfix on

    • Escape all strings from the directory connection before logging
      them. Fixes bug 13071; bugfix on Patch from "teor".
  • Minor bugfixes (controller):
    • Restore the functionality of CookieAuthFileGroupReadable. Fixes
      bug 12864; bugfix on

    • Actually send TRANSPORT_LAUNCHED and HS_DESC events to
      controllers. Fixes bug 13085; bugfix on Patch
      by "teor".
  • Minor bugfixes (compilation):
    • Fix compilation of test.h with MSVC. Patch from Gisle Vanem;
      bugfix on

    • Make the nmake make files work again. Fixes bug 13081. Bugfix on Patch from "NewEraCracker".

    • In routerlist_assert_ok(), don't take the address of a
      routerinfo's cache_info member unless that routerinfo is non-NULL.
      Fixes bug 13096; bugfix on Patch by "teor".

    • Fix a large number of false positive warnings from the clang
      analyzer static analysis tool. This should make real warnings
      easier for clang analyzer to find. Patch from "teor". Closes
      ticket 13036.
  • Distribution (systemd):
    • Verify configuration file via ExecStartPre in the systemd unit
      file. Patch from intrigeri; resolves ticket 12730.

    • Explicitly disable RunAsDaemon in the systemd unit file. Our
      current systemd unit uses "Type = simple", so systemd does not
      expect tor to fork. If the user has "RunAsDaemon 1" in their
      torrc, then things won't work as expected. This is e.g. the case
      on Debian (and derivatives), since there we pass "--defaults-torrc
      /usr/share/tor/tor-service-defaults-torrc" (that contains
      "RunAsDaemon 1") by default. Patch by intrigeri; resolves
      ticket 12731.
  • Documentation:
    • Adjust the URLs in the README to refer to the new locations of
      several documents on the website. Fixes bug 12830. Patch from
      Matt Pagan.

    • Document 'reject6' and 'accept6' ExitPolicy entries. Resolves
      ticket 12878.

Tor Weekly News — September 10th, 2014

Welcome to the thirty-sixth issue in 2014 of Tor Weekly News, the weekly newsletter that covers what is happening in the Tor community.

More monthly status reports for August 2014

The wave of regular monthly reports from Tor project members for the month of August continued, with reports from Yawning AngelGeorge KadianakisIsis LovecruftColin C., and Griffin Boyce.

Arturo Filastò reported on behalf of the OONI team.

Miscellaneous news

Nathan Freitas announced the release of Orbot 14.0.8, containing “some fixes for people who like to fiddle with transproxy/iptables settings, which can lead to the device getting into a bad network state”, as well as for “a common freak crash that was occuring on app exit in some cases.” See Nathan’s message for a full changelog and download links.

Mike Perry asked for comments on his proposal to drop Tor Browser support for Mac OS X 10.6, which is no longer receiving security updates from Apple. This means that the Tor Browser team would only have to distribute standard-sized 64-bit builds for Mac OS X rather than the oversized 32+64-bit set. Users who are unable to upgrade their operating system would still be able to use Tails for their Tor browsing needs.

Hartmut Haase reported that Tor Browser occasionally fails to open, despite a successful connection being made to the Tor network; several other users confirmed that they are also experiencing this problem. Georg Koppen suggested that the issue is the one covered by bug ticket #10804: “Solving this is high on the priority list, but alas not as high as getting everything ready for the switch to ESR31.”

Thanks to Peter Ludikovsky and goll for running mirrors of the Tor Project website and software archive!

Andrew Lewman published the results of a test he ran to answer the question “Why not just use CloudFlare for mirrors of the Tor Project website?”: “The results are that using CloudFlare doesn’t offload the binaries, which are what make up the bulk of traffic on the mirror […] I’ve started to look at CDN providers to see if there are affordable services which can offload the entire site itself.”

As part of an ongoing effort to rescue the Tor blog from rot and ruin caused by broken Drupal code, ultrasandwich set up an unofficial preview of a possible blog based on the Jekyll static site generator. If you want to contribute to the revamp of the Tor Project website, including the blog, the www-team mailing list awaits your comments and ideas!

Tor help desk roundup

Users want to know if their personal information is safe when they use Tor Browser. Personal accounts are no less secure using Tor Browser than they are using the web without Tor: the problem of authenticating websites and preventing eavesdropping has been addressed outside of the Tor context through HTTPS. That’s why the Tor Browser ships with the HTTPS-Everywhere browser extension — for every website you visit, HTTPS-Everywhere checks whether or not that website is known to have an HTTPS version, and if so it connects to the site using HTTPS instead of HTTP. Tor + HTTPS provides full end-to-end encryption when visiting any site that offers its content via HTTPS. Using HTTPS with Tor helps keep users’ web accounts secure.

Easy development tasks to get involved with

If a single human or organization runs more than one relay, they should configure all their relays to be in the same “family”, the goal being to prevent clients from using more than one of these relays in the same circuit. However, the config option used for this, MyFamily, only accepts relay fingerprints that are preceeded by $, unlike most other config options. It would be great if this option accepted fingerprints preceeded by $, as well as without it. Nick Mathewson says this ticket would be pretty easy, so why not give it a try? It does sound like some fun C hacking. Be sure to post your patch to the ticket.

Back in the day, the tor daemon, which is the core of the Tor network, compiled and ran on Windows 98. But that’s history, and aren’t we all glad? Somebody should identify and drop support code for all Windows versions prior to Windows XP. Nick says “this is mainly going to be a matter of identifying cases where we use LoadLibrary and GetProcAddress to find always-present-functions in always-present DLLs.” If the previous sentence made any sense to you, maybe you’re a good person to help with this! Be sure to comment on the ticket if you have a branch to review.

This issue of Tor Weekly News has been assembled by harmony, Matt Pagan, Karsten Loesing, and Lunar.

Want to continue reading TWN? Please help us create this newsletter. We still need more volunteers to watch the Tor community and report important news. Please see the project page, write down your name and subscribe to the team mailing list if you want to get involved!

Ahmia search after GSoC development

The Google Summer of Code (GSoC) was an excellent opportunity to improve on the Ahmia search engine. With Google's stipend and friendly mentoring from The Tor Project, I was able to concentrate on development of my search engine project. Thank you all!

GSoC 2014 is over, but I am sticking around to continue developing and maintaining Ahmia.

Here is the current status of ahmia after GSoC development:


Ahmia is open-source search engine software for Tor hidden service websites. You can test the running search engine at

Building a search engine for anonymous web sites running inside the Tor network is an interesting problem. Tor enables web servers to hide their location and Tor users can connect to these authenticated hidden services while the server and the user both stay anonymous. However, finding web content is hard without a good search engine and therefore a search engine is needed for the Tor network.

Web search engines are needed to navigate and search the web. There were no search engines for searching hidden service web content, so I decided to build a search engine specially for Tor. I registered and started development on it as a side project in 2010.

This development involved programming and testing web crawlers, thinking of ways to find hidden service addresses (since the protocol does not allow enumeration), learning about the Tor community, and implementing a filtering policy. Moreover, I implemented an API that empowers other Tor services that publish content to integrate with Ahmia.

As a result, Ahmia is a working search engine that indexes, searches and catalogs content published on Tor Hidden Services. Furthermore, it is an environment to share meaningful statistics, insights and news about the Tor network itself.

Interesting Summer of Code

One of my best memories from the summer is the Tor Project's Summer 2014 Developers meeting that was hosted by Mozilla in Paris, France. I have always admired the people who are working on the Tor Project.

I also loved the coding itself. Finally I had time to improve the Ahmia search engine and its many features. I did a lot of work and liked it.

Some journalist were very interested in my work: Carola Frediani asked if I could analyze the content of hidden services. I coded a script that fetches every front page's HTML, I gathered all the keywords, headers and description texts and made a simple word cloud visualization.

Hidden website content visualization.

It is a simple way to glance what is published on the hidden websites.

Carola found this data useful and used it in her presentation at on June 11th.

Technical design of ahmia

The Ahmia web service is written using the Django web framework. As a result, the server-side language is Python. On the client-side, most of the pages are plain HTML. There are some pages that require JavaScript, but the search itself works without client-side JavaScript.

The components of Ahmia are:

  • Django front-end site
  • PostgreSQL database for the site
  • Custom scripts to download data about hidden services
  • Django-Haystack connection to Solr database
  • Apache Solr for the crawled data
  • OnionBot crawler that gathers data to Solr database

Technical architecture.

See installation and developing tutorial


The full-text search is implemented using Django-Haystack. The search is using crawled website data that is saved to Apache Solr.


OnionDir is a list of known online hidden service addresses. A separate script gathers this list and fetches information fields from the HTML (title, keywords, description etc.). Furthermore, users can freely edit these fields.

We've also started a convention where hidden service admins can add a file to their website, called description.json, to offer an official description of their site in Ahmia.

As a result, this information is shown in the OnionDir page and over 80 domains are already using this method.


We are gathering statistics from hidden services. As a result, we can represent and share meaningful data about hidden services and visualize it.

We are gathering three types of popularity data:

  1. Tor2web nodes share their visiting statistics to Ahmia
  2. Number of public WWW backlinks to hidden services
  3. Number of clicks in the search results

The click counter tells the total number of clicks on a search result in


We have decided to filter any sites related to child porn from our search results. Ahmia is removing everything related to these websites. These websites may not be actual child porn sites. They are rather sites where users can post content (forums, file and image uploads etc.) and as the result there have been, momentarily at least, some suspicious content that has not been moderated in a reasonable period of time. does not have the time to monitor these sites carefully and we are banning sites from our public index if we see any evidence of child abuse. Of course, the ban is removed if the site itself contacts us and we review the website to be OK.

In practice, Ahmia calculates the MD5 sums of the banned domains for use as a filtering policy. Moreover, we are sharing this list and Tor2web nodes can use the list to filter out pages.

At the moment, there seems to be 1228 hidden website domains online and 7 of them has been filtered because they are possibly sharing child porn content.


OnionBot is a crawler for hidden service websites based on the Scrapy framework. It crawls the Tor network and passes data to the search database. OnionBot requires the Tor software (using Tor2web mode) and Polipo. The results are saved to Apache Solr.

Apache Solr

Apache Solr is a popular, open source enterprise search platform. Its major features include powerful full-text search, hit highlighting, faceted search, and near real-time indexing.

The schema.xml file contains all of the details about which fields your documents can contain, and how those fields should be dealt with when adding documents to the index, or when querying those fields.

Security measures for privacy

In the software

  • We do not log any IP addresses, see Apache configuration
  • We are gathering real-time clicks, however, this data is not shown accurately

In the host

  • Backend servers are run separately and they do not have any knowledge about the end-users
  • All servers are hosted in countries with strong privacy laws. For example, Finland and the Netherlands
  • Communication between servers is encrypted
  • Only a few trustworthy people know the locations of the back-end servers and are able to access them

Future work

GSoC 2014 was fun and productive!

There is a lot more to do. However, I do not have time to do everything myself. Of course, I am coding when I have time and maintaining the search engine.

In addition, I am going to write a scientific article about the implementation.

Is there anyone who would be interested in developing

Is anyone familiar with Solr and would know how to tweak it for full text search?

Furthermore, any kind of help would be most welcome. There are always Linux admin duties, HTML/CSS design, bug fixing, Django development, etc...

For further information, please don't hesitate to contact me by e-mail:

Tor Browser 3.6.5 and 4.0-alpha-2 are released

Tor Browser 3.6.5

The fifth pointfix release of the 3.6 series is available from the Tor Browser Project page and also from our distribution directory.

This release features important security updates to Firefox.

This release also features improvements to the canvas image extraction permissions prompt, and will now log offending script urls to the browser console. It also restores the missing RELRO hardening option to the Linux bundles, and disables NTLM and Negotiate HTTP auth (which can leak sensitive information about the computer). To avoid resolution fingerprinting, popups are also opened in new tabs by default.

Here is the complete changelog for 3.6.5:

  • All Platforms
    • Update Firefox to 24.8.0esr
    • Update NoScript to
    • Update HTTPS Everywhere to 4.0.0
    • Update Torbutton to
      • Bug 12684: New strings for canvas image extraction message
      • Bug 8940: Move RecommendedTBBVersions file to
      • Bug 9531: Workaround to avoid rare hangs during New Identity
    • Bug 12684: Improve Canvas image extraction permissions prompt
    • Bug 7265: Only prompt for first party canvas access. Log all scripts
      that attempt to extract canvas images to Browser console.

    • Bug 12974: Disable NTLM and Negotiate HTTP Auth
    • Bug 2874: Remove Components.* from content access (regression)
    • Bug 9881: Open popups in new tabs by default
  • Linux:
    • Bug 12103: Adding RELRO hardening back to browser binaries.

Tor Browser 4.0-alpha-2

In addition, we are also releasing the second alpha in the 4.0 series, available for download on the extended downloads page.

This release also includes important security updates to Firefox.

In addition to including the changes in 3.6.5, this release also is the first Tor Browser release to enable the in-browser Firefox-based updater. This means that if all goes well, 4.0-alpha-2 users will notified of an available update via a notification similar to that in Firefox. You will then be able to download and install it directly via the browser UI. By default, neither the download nor the update will happen automatically, so if you are not feeling adventurous, you need not allow it to update in this way. Even if you are feeling adventurous, you should probably back up your Tor Browser directory before updating.

In addition to the updater, this release should also re-enable the basic hardening features on Windows, including ASLR, DEP, and SSP.

Furthermore, the NoScript behavior in this release has changed. Selecting "Temporarily allow scripts" will now automatically allow all scripts in a page. This was done for usability reasons, to make it easier for novice users to run Tor Browser with scripting disabled most of the time. This will also hopefully make it possible for more people to use the "High Security" setting in our upcoming Security Slider, which will have Javascript disabled globally via NoScript by default.

Here is the complete changelog for 4.0-alpha-2:

  • All Platforms
    • Update Firefox to 24.8.0esr
    • Update NoScript to
    • Update Tor Launcher to
      • Bug 11405: Remove firewall prompt from wizard.
      • Bug 12895: Mention as a valid bridge request email address
      • Bug 12444: Provide feedback when “Copy Tor Log” is clicked.
      • Bug 11199: Improve error messages if Tor exits unexpectedly
    • Update Torbutton to
      • Bug 12684: New strings for canvas image extraction message
      • Bug 8940: Move RecommendedTBBVersions file to
    • Bug 12684: Improve Canvas image extraction permissions prompt
    • Bug 7265: Only prompt for first party canvas access. Log all scripts
      that attempt to extract canvas images to Browser console.

    • Bug 12974: Disable NTLM and Negotiate HTTP Auth
    • Bug 2874: Remove Components.* from content access (regression)
    • Bug 4234: Automatic Update support (off by default)
    • Bug 9881: Open popups in new tabs by default
    • Meek Pluggable Transport:
      • Bug 12766: Use TLSv1.0 in meek-http-helper to blend in with Firefox 24
  • Windows:
    • Bug 10065: Enable DEP, ASLR, and SSP hardening options
  • Linux:
    • Bug 12103: Adding RELRO hardening back to browser binaries.

The list of frequently encountered known issues is also available in our bug tracker.

Tor Weekly News — September 3rd, 2014

Welcome to the thirty-fifth issue of Tor Weekly News in 2014, the weekly newsletter that covers what is happening in the Tor community.

Tor Browser 3.6.5 and 4.0-alpha-2 are out

The Tor Browser team put out two new releases of the privacy-preserving web browser. Among the major changes, version 3.6.5 upgrades Firefox to 24.8.0esr, and includes an improved prompt to help users defend against HTML5 canvas image fingerprinting, following a patch by Isis Lovecruft. Version 4.0-alpha-2 additionally includes the code for the forthcoming Tor Browser auto-updater (switched off by default) and “better hardening for Windows and Linux builds”.

As ever, you can download the new releases along with their signature files from the Tor Project’s distribution directory. Please upgrade as soon as you can.

Tails 1.1.1 is out

The Tails team released version 1.1.1 of the Debian- and Tor-based live operating system. As well as upgrading key components like Tor, Iceweasel, and Linux, this release disables I2P by default when Tails is booted, in response to the vulnerability recently disclosed by Exodus Intelligence. Like Truecrypt, “i2p” must now be specified as a parameter on booting by users who wish to use it.

A number of other security fixes and routine improvements make this an important update for all Tails users. See the full changelog in the team’s announcement, then update from a running copy of Tails 1.1 if you have one, or head to the download page if you don’t.

Helping Internet services accept anonymous users

Without a large and diverse network, run by thousands of dedicated volunteers, Tor would be nowhere near as useful or popular as it currently is. Although the current situation might at times seem fragile, there are still many places where it is feasible to host Tor exit nodes.

However, Tor would become much less attractive to users if they found themselves unable to reach or interact with their favorite websites while using it, a situation that is unfortunately growing more common as site administrators and engineers react negatively to instances of abusive Tor traffic by banning anonymous connections outright. Tor users and developers, as well as members of other online communities (such as Wikimedia), have tried to address the issue before, but real progress has yet to be made.

Roger Dingledine wrote a “call to arms” explaining the problem in detail and exploring possible paths to a solution: “Step one is to enumerate the set of websites and other Internet services that handle Tor connections differently from normal connections […]. Step two is to sort the problem websites based on how amenable they would be to our help”.

Since the problem involves humans as much as it does machines, anyone working on it will have to be both “technical” but also ”good at activism”. If you fit that description, OTF has expressed interest in funding work on this issue through their Information Controls Fellowship Program. Please read Roger’s blog post in full for more details.

Monthly status reports for August 2014

The wave of regular monthly reports from Tor project members for the month of August has begun. Damian Johnson released his report first, followed by reports from Georg Koppen, Sherief Alaa, Noel Torres, Kevin P Dyer, Nick Mathewson, Lunar, Arthur D. Edelstein, Karsten Loesing, Andrew Lewman, Arlo Breault, Pearl Crescent, and Michael Schloh von Bennewitz.

Lunar also reported on behalf of the help desk, and Mike Perry did the same for the Tor Browser team.

Miscellaneous news

Yawning Angel released a new set of experimental Tor Browser builds containing the proposed obfs4 pluggable transport, along with a changelog; “questions, comments, feedback” are welcome on the email thread or the bug ticket tracking the deployment of obfs4.

Arturo Filastò announced the release of version 1.1.0 of oonibackend, the tool “used by ooniprobe to discover the addresses of test helpers (via the bouncer) to submit reports to (via the collector) and to perform some measurements that require a backend system to talk to (via test helpers)”.

meejah posted a list of tasks to be completed in order to bring Tor Weather to a deployable state, following the recent rewrite effort and the Google Summer of Code project by Sreenatha Bhatlapenumarthi.

Israel Leiva submitted a summary of work completed as part of the “Revamp GetTor” Google Summer of Code project: “The plan for now is to keep doing tests and deploy it asap (hopefully during September).”

Mike Perry posted an updated version of the proposal for website fingerprinting countermeasures which he co-authored with Marc Juarez as part of the latter’s Google Summer of Code project.

Lunar gave a talk at this year’s DebConf on the effort to build Debian packages deterministically, which is inspired in large part by Tor Browser’s use of the same technology. Major progress was achieved during the conference.

David Fifield submitted a breakdown of the costs incurred by the infrastructure that supports the meek pluggable transport since its introduction. The total to date from both the Google App Engine and Amazon AWS front domains? $6.56.

Thanks to P D and Daniel Pajonzeck for running mirrors of the Tor Project website and software!

Also on the subject of mirrors, Roger Dingledine alerted the tor-mirrors mailing list to the fact that the Tor Project website (specifically the distribution directory) will shortly be increasing in size to eight or nine gigabytes, as a result of the soon-to-be-implemented Tor Browser updater. Mirror operators will need to ensure that they can provide enough disk space to accommodate the change.

whonixqubes announced the release of an integrated version of the Whonix and Qubes operating systems: “I look forward to helping make Qubes + Whonix integration even tighter and more seamless throughout the future.”

Tor help desk roundup

The help desk has been asked if Tor can make a website visit appear to come from China. Tor connections appear to originate from the country where the exit relay in use is located; since Tor is blocked in China, there are zero exit relays in China. A visualization of the different country-locations of exit relays can be found on Tor’s metrics page.

News from Tor StackExchange

Anony Mouse wanted to know why Facebook shows the location of the user’s last login over Tor as Baghdad or Dhaka, instead of the real location of the exit relay. qbi posted a screenshot showing this issue. According to Facebook, this information is based on an approximation, but this approximation locates all Tor exit relays either in Baghdad or in Dhaka.

user3500 wants to contribute to Tor and asks how this can be done as an inexperienced developer. Jens Kubieziel replied with several possibilities, including reading the volunteer page and Tor Weekly News: in particular, the section containing easy development tasks might be a good start. Roya pointed out that any contribution is better than no contribution, and encouraged user3500 to just get started. Umut Seven recommended writing unit tests.

Kras wants to use FoxyProxy in connection with Tor Browser Bundle and asks if it is safe to do so. At the moment, there is only an answer saying “yes”, without any explanation. What is your experience? Is it safe for a user to install and use FoxyProxy?

This issue of Tor Weekly News has been assembled by harmony, Matt Pagan, Lunar, qbi, and Arlo Breault.

Want to continue reading TWN? Please help us create this newsletter. We still need more volunteers to watch the Tor community and report important news. Please see the project page, write down your name and subscribe to the team mailing list if you want to get involved!

Tails 1.1.1 is out

Tails, The Amnesic Incognito Live System, version 1.1.1, is out.

All users must upgrade as soon as possible: this release fixes numerous security issues.

Download it now.


Notable user-visible changes include:

  • Security fixes
    • Upgrade the web browser to 24.8.0esr-0+tails1~bpo70+1 (Firefox 24.8.0esr + Iceweasel patches + Torbrowser patches).
    • Add an I2P boot parameter. Without adding "i2p" to the kernel command line, I2P will not be accessible for the Live user. I2P was also upgraded to, and stricter firewall rules are applied to it, among other security enhancements.
    • Upgrade Tor to (fixes CVE-2014-5117).
    • Upgrade Linux to 3.14.15-2 (fixes CVE-2014-3534, CVE-2014-4667 and CVE-2014-4943).
    • Prevent dhclient from sending the hostname over the network (ticket #7688).
    • Override the hostname provided by the DHCP server (ticket #7769).
  • Bugfixes
    • Don't ship OpenJDK 6: I2P prefers v7, and we don't need both (ticket #7807).
    • Prevent Tails Installer from updating the system partition properties on MBR partitions (ticket #7716).
  • Minor improvements
    • Upgrade to Torbutton
    • Install gnome-user-guide (ticket #7618).
    • Install cups-pk-helper (ticket #7636).
    • Update the SquashFS sort file, which should speed up boot from DVD (ticket #6372).
    • Compress the SquashFS more aggressively (ticket #7706) which should make the Tails ISO image smaller.

See the online Changelog for technical details.

Known issues

Longstanding known issues.

I want to try it or to upgrade!

Go to the download page.

What's coming up?

The next Tails release is scheduled for October 14.

Have a look to our roadmap to see where we are heading to.

Do you want to help? There are many ways you can contribute to Tails. If you want to help, come talk to us!

Support and feedback

For support and feedback, visit the Support section on the Tails website.

Syndicate content Syndicate content