Tor 0.3.2.1-alpha is released, with support for next-gen onion services and KIST scheduler

by nickm | September 18, 2017

And as if all those other releases today were not enough, this is also the time for a new alpha release series!

Tor 0.3.2.1-alpha is the first release in the 0.3.2.x series. It includes support for our next-generation ("v3") onion service protocol, and adds a new circuit scheduler for more responsive forwarding decisions from relays. There are also numerous other small features and bugfixes here.

You can download the source from the usual place on the website. Binary packages should be available soon, with an alpha Tor Browser likely by the end of the month.

Remember: This is an alpha release, and it's likely to have more bugs than usual. We hope that people will try it out to find and report bugs, though.

Below are the changes since Tor 0.3.1.7.

Changes in version 0.3.2.1-alpha - 2017-09-18

  • Major feature (scheduler, channel):
    • Tor now uses new schedulers to decide which circuits should deliver cells first, in order to improve congestion at relays. The first type is called "KIST" ("Kernel Informed Socket Transport"), and is only available on Linux-like systems: it uses feedback from the kernel to prevent the kernel's TCP buffers from growing too full. The second new scheduler type is called "KISTLite": it behaves the same as KIST, but runs on systems without kernel support for inspecting TCP implementation details. The old scheduler is still available, under the name "Vanilla". To change the default scheduler preference order, use the new "Schedulers" option. (The default preference order is "KIST,KISTLite,Vanilla".)

      Matt Traudt implemented KIST, based on research by Rob Jansen, John Geddes, Christ Wacek, Micah Sherr, and Paul Syverson. For more information, see the design paper at http://www.robgjansen.com/publications/kist-sec2014.pdf and the followup implementation paper at https://arxiv.org/abs/1709.01044. Closes ticket 12541.

  • Major features (next-generation onion services):
    • Tor now supports the next-generation onion services protocol for clients and services! As part of this release, the core of proposal 224 has been implemented and is available for experimentation and testing by our users. This newer version of onion services ("v3") features many improvements over the legacy system, including:

      a) Better crypto (replaced SHA1/DH/RSA1024 with SHA3/ed25519/curve25519)

      b) Improved directory protocol, leaking much less information to directory servers.

      c) Improved directory protocol, with smaller surface for targeted attacks.

      d) Better onion address security against impersonation.

      e) More extensible introduction/rendezvous protocol.

      f) A cleaner and more modular codebase.

      You can identify a next-generation onion address by its length: they are 56 characters long, as in "4acth47i6kxnvkewtm6q7ib2s3ufpo5sqbsnzjpbi7utijcltosqemad.onion".

      In the future, we will release more options and features for v3 onion services, but we first need a testing period, so that the current codebase matures and becomes more robust. Planned features include: offline keys, advanced client authorization, improved guard algorithms, and statistics. For full details, see proposal 224.

      Legacy ("v2") onion services will still work for the foreseeable future, and will remain the default until this new codebase gets tested and hardened. Service operators who want to experiment with the new system can use the 'HiddenServiceVersion 3' torrc directive along with the regular onion service configuration options. We will publish a blog post about this new feature soon! Enjoy!

 

  • Major bugfixes (usability, control port):
    • Report trusted clock skew indications as bootstrap errors, so controllers can more easily alert users when their clocks are wrong. Fixes bug 23506; bugfix on 0.1.2.6-alpha.
  • Minor features (bug detection):
    • Log a warning message with a stack trace for any attempt to call get_options() during option validation. This pattern has caused subtle bugs in the past. Closes ticket 22281.
  • Minor features (client):
    • You can now use Tor as a tunneled HTTP proxy: use the new HTTPTunnelPort option to open a port that accepts HTTP CONNECT requests. Closes ticket 22407.
    • Add an extra check to make sure that we always use the newer guard selection code for picking our guards. Closes ticket 22779.
    • When downloading (micro)descriptors, don't split the list into multiple requests unless we want at least 32 descriptors. Previously, we split at 4, not 32, which led to significant overhead in HTTP request size and degradation in compression performance. Closes ticket 23220.
  • Minor features (command line):
    • Add a new commandline option, --key-expiration, which prints when the current signing key is going to expire. Implements ticket 17639; patch by Isis Lovecruft.
  • Minor features (control port):
    • If an application tries to use the control port as an HTTP proxy, respond with a meaningful "This is the Tor control port" message, and log the event. Closes ticket 1667. Patch from Ravi Chandra Padmala.
    • Provide better error message for GETINFO desc/(id|name) when not fetching router descriptors. Closes ticket 5847. Patch by Kevin Butler.
    • Add GETINFO "{desc,md}/download-enabled", to inform the controller whether Tor will try to download router descriptors and microdescriptors respectively. Closes ticket 22684.
    • Added new GETINFO targets "ip-to-country/{ipv4,ipv6}-available", so controllers can tell whether the geoip databases are loaded. Closes ticket 23237.
    • Adds a timestamp field to the CIRC_BW and STREAM_BW bandwidth events. Closes ticket 19254. Patch by "DonnchaC".
  • Minor features (development support):
    • Developers can now generate a call-graph for Tor using the "calltool" python program, which post-processes object dumps. It should work okay on many Linux and OSX platforms, and might work elsewhere too. To run it, install calltool from https://gitweb.torproject.org/user/nickm/calltool.git and run "make callgraph". Closes ticket 19307.
  • Minor features (ed25519):
    • Add validation function to checks for torsion components in ed25519 public keys, used by prop224 client-side code. Closes ticket 22006. Math help by Ian Goldberg.
  • Minor features (exit relay, DNS):
    • Improve the clarity and safety of the log message from evdns when receiving an apparently spoofed DNS reply. Closes ticket 3056.
  • Minor features (integration, hardening):
    • Add a new NoExec option to prevent Tor from running other programs. When this option is set to 1, Tor will never try to run another program, regardless of the settings of PortForwardingHelper, ClientTransportPlugin, or ServerTransportPlugin. Once NoExec is set, it cannot be disabled without restarting Tor. Closes ticket 22976.
  • Minor features (logging):
    • Improve the warning message for specifying a relay by nickname. The previous message implied that nickname registration was still part of the Tor network design, which it isn't. Closes ticket 20488.
    • If the sandbox filter fails to load, suggest to the user that their kernel might not support seccomp2. Closes ticket 23090.
  • Minor features (portability):
    • Check at configure time whether uint8_t is the same type as unsigned char. Lots of existing code already makes this assumption, and there could be strict aliasing issues if the assumption is violated. Closes ticket 22410.
  • Minor features (relay, configuration):
    • Reject attempts to use relative file paths when RunAsDaemon is set. Previously, Tor would accept these, but the directory- changing step of RunAsDaemon would give strange and/or confusing results. Closes ticket 22731.
  • Minor features (startup, safety):
    • When configured to write a PID file, Tor now exits if it is unable to do so. Previously, it would warn and continue. Closes ticket 20119.
  • Minor features (static analysis):
    • The BUG() macro has been changed slightly so that Coverity no longer complains about dead code if the bug is impossible. Closes ticket 23054.
  • Minor features (testing):
    • The default chutney network tests now include tests for the v3 hidden service design. Make sure you have the latest version of chutney if you want to run these. Closes ticket 22437.
    • Add a unit test to verify that we can parse a hardcoded v2 hidden service descriptor. Closes ticket 15554.
  • Minor bugfixes (certificate handling):
    • Fix a time handling bug in Tor certificates set to expire after the year 2106. Fixes bug 23055; bugfix on 0.3.0.1-alpha. Found by Coverity as CID 1415728.
  • Minor bugfixes (client, usability):
    • Refrain from needlessly rejecting SOCKS5-with-hostnames and SOCKS4a requests that contain IP address strings, even when SafeSocks in enabled, as this prevents user from connecting to known IP addresses without relying on DNS for resolving. SafeSocks still rejects SOCKS connections that connect to IP addresses when those addresses are _not_ encoded as hostnames. Fixes bug 22461; bugfix on Tor 0.2.6.2-alpha.
  • Minor bugfixes (code correctness):
    • Call htons() in extend_cell_format() for encoding a 16-bit value. Previously we used ntohs(), which happens to behave the same on all the platforms we support, but which isn't really correct. Fixes bug 23106; bugfix on 0.2.4.8-alpha.
    • For defense-in-depth, make the controller's write_escaped_data() function robust to extremely long inputs. Fixes bug 19281; bugfix on 0.1.1.1-alpha. Reported by Guido Vranken.
  • Minor bugfixes (compilation):
    • Fix unused-variable warnings in donna's Curve25519 SSE2 code. Fixes bug 22895; bugfix on 0.2.7.2-alpha.
  • Minor bugfixes (consensus expiry):
    • Check for adequate directory information correctly. Previously, Tor would reconsider whether it had sufficient directory information every 2 minutes. Fixes bug 23091; bugfix on 0.2.0.19-alpha.
  • Minor bugfixes (directory protocol):
    • Directory servers now include a "Date:" http header for response codes other than 200. Clients starting with a skewed clock and a recent consensus were getting "304 Not modified" responses from directory authorities, so without the Date header, the client would never hear about a wrong clock. Fixes bug 23499; bugfix on 0.0.8rc1.
    • Make clients wait for 6 seconds before trying to download a consensus from an authority. Fixes bug 17750; bugfix on 0.2.8.1-alpha.
  • Minor bugfixes (DoS-resistance):
    • If future code asks if there are any running bridges, without checking if bridges are enabled, log a BUG warning rather than crashing. Fixes bug 23524; bugfix on 0.3.0.1-alpha.
  • Minor bugfixes (format strictness):
    • Restrict several data formats to decimal. Previously, the BuildTimeHistogram entries in the state file, the "bw=" entries in the bandwidth authority file, and the process IDs passed to the __OwningControllerProcess option could all be specified in hex or octal as well as in decimal. This was not an intentional feature. Fixes bug 22802; bugfixes on 0.2.2.1-alpha, 0.2.2.2-alpha, and 0.2.2.28-beta.
  • Minor bugfixes (heartbeat):
    • If we fail to write a heartbeat message, schedule a retry for the minimum heartbeat interval number of seconds in the future. Fixes bug 19476; bugfix on 0.2.3.1-alpha.
  • Minor bugfixes (linux seccomp2 sandbox, logging):
    • Fix some messages on unexpected errors from the seccomp2 library. Fixes bug 22750; bugfix on 0.2.5.1-alpha. Patch from "cypherpunks".
  • Minor bugfixes (logging):
    • Remove duplicate log messages regarding opening non-local SocksPorts upon parsing config and opening listeners at startup. Fixes bug 4019; bugfix on 0.2.3.3-alpha.
    • Use a more comprehensible log message when telling the user they've excluded every running exit node. Fixes bug 7890; bugfix on 0.2.2.25-alpha.
    • When logging the number of descriptors we intend to download per directory request, do not log a number higher than then the number of descriptors we're fetching in total. Fixes bug 19648; bugfix on 0.1.1.8-alpha.
    • When warning about a directory owned by the wrong user, log the actual name of the user owning the directory. Previously, we'd log the name of the process owner twice. Fixes bug 23487; bugfix on 0.2.9.1-alpha.
    • The tor specification says hop counts are 1-based, so fix two log messages that mistakenly logged 0-based hop counts. Fixes bug 18982; bugfix on 0.2.6.2-alpha and 0.2.4.5-alpha. Patch by teor. Credit to Xiaofan Li for reporting this issue.
  • Minor bugfixes (portability):
    • Stop using the PATH_MAX variable, which is not defined on GNU Hurd. Fixes bug 23098; bugfix on 0.3.1.1-alpha.
  • Minor bugfixes (relay):
    • When uploading our descriptor for the first time after startup, report the reason for uploading as "Tor just started" rather than leaving it blank. Fixes bug 22885; bugfix on 0.2.3.4-alpha.
    • Avoid unnecessary calls to directory_fetches_from_authorities() on relays, to prevent spurious address resolutions and descriptor rebuilds. This is a mitigation for bug 21789. Fixes bug 23470; bugfix on in 0.2.8.1-alpha.
  • Minor bugfixes (tests):
    • Fix a broken unit test for the OutboundAddress option: the parsing function was never returning an error on failure. Fixes bug 23366; bugfix on 0.3.0.3-alpha.
    • Fix a signed-integer overflow in the unit tests for dir/download_status_random_backoff, which was untriggered until we fixed bug 17750. Fixes bug 22924; bugfix on 0.2.9.1-alpha.
  • Minor bugfixes (usability, control port):
    • Stop making an unnecessary routerlist check in NETINFO clock skew detection; this was preventing clients from reporting NETINFO clock skew to controllers. Fixes bug 23532; bugfix on 0.2.4.4-alpha.
  • Code simplification and refactoring:
    • Extract the code for handling newly-open channels into a separate function from the general code to handle channel state transitions. This change simplifies our callgraph, reducing the size of the largest strongly connected component by roughly a factor of two. Closes ticket 22608.
    • Remove dead code for largely unused statistics on the number of times we've attempted various public key operations. Fixes bug 19871; bugfix on 0.1.2.4-alpha. Fix by Isis Lovecruft.
    • Remove several now-obsolete functions for asking about old variants directory authority status. Closes ticket 22311; patch from "huyvq".
    • Remove some of the code that once supported "Named" and "Unnamed" routers. Authorities no longer vote for these flags. Closes ticket 22215.
    • Rename the obsolete malleable hybrid_encrypt functions used in TAP and old hidden services, to indicate that they aren't suitable for new protocols or formats. Closes ticket 23026.
    • Replace our STRUCT_OFFSET() macro with offsetof(). Closes ticket 22521. Patch from Neel Chauhan.
    • Split the enormous circuit_send_next_onion_skin() function into multiple subfunctions. Closes ticket 22804.
    • Split the portions of the buffer.c module that handle particular protocols into separate modules. Part of ticket 23149.
    • Use our test macros more consistently, to produce more useful error messages when our unit tests fail. Add coccinelle patches to allow us to re-check for test macro uses. Closes ticket 22497.
  • Deprecated features:
    • Deprecate HTTPProxy/HTTPProxyAuthenticator config options. They only applies to direct unencrypted HTTP connections to your directory server, which your Tor probably isn't using. Closes ticket 20575.
  • Documentation:
    • Clarify in the manual that "Sandbox 1" is only supported on Linux kernels. Closes ticket 22677.
    • Document all values of PublishServerDescriptor in the manpage. Closes ticket 15645.
    • Improve the documentation for the directory port part of the DirAuthority line. Closes ticket 20152.
    • Restore documentation for the authorities' "approved-routers" file. Closes ticket 21148.
  • Removed features:
    • The AllowDotExit option has been removed as unsafe. It has been deprecated since 0.2.9.2-alpha. Closes ticket 23426.
    • The ClientDNSRejectInternalAddresses flag can no longer be set on non-testing networks. It has been deprecated since 0.2.9.2-alpha. Closes ticket 21031.
    • The controller API no longer includes an AUTHDIR_NEWDESCS event: nobody was using it any longer. Closes ticket 22377.

Comments

Please note that the comment area below has been archived.

September 18, 2017

Permalink

Have you made some performances tests to compare curve to rsa? I bet the CPU consumption difference is significant.

September 18, 2017

Permalink

Woohoo! Finally something stable enough to have fun with V3 onion services :)

By the way an important question: how does making Tor fully multithreaded (many people and relays use 4 cores and even more including mobile users) fit in your future plans?

September 19, 2017

Permalink

When will they add measurements against traffic correlation attacks?

September 19, 2017

Permalink

congratulations
- what 's new about quantum resistance ?
- will the next TBB be update with all these new improvements ?

September 19, 2017

Permalink

(experimental)
- does "KIST" work or is it a proof of concept ?
- SHA3 ! 25519 ! _ fantastic _
- does it mean that in a near future the onions site must be set with a 56 length ?
- could i set my o.s & my kernel with some of these features (e.g. TCP buffers) ?
-------------------------------------------------------------------------------------------------------------
- https-everywhere runs well with tor but is still broken on firefox.
- is it true that the relays are running with a google dns ?

> does "KIST" work or is it a proof of concept ?

KIST works and is in 0.3.2.1-alpha.

> does it mean that in a near future the onions site must be set with a 56 length

Legacy ("v2") are still the default type. Still the 16-char length addresses you're familiar with.

Legacy ("v2") onion services will still work for the foreseeable future, and will remain the default until this new codebase gets tested and hardened. Service operators who want to experiment with the new system can use the 'HiddenServiceVersion 3' torrc directive along with the regular onion service configuration options.

> could i set my o.s & my kernel with some of these features (e.g. TCP buffers)

Yes you can tweak low level stuff like that. But that's not what KIST does, it isn't how KIST works, and doesn't really have anything to do with Tor.

> is it true that the relays are running with a google dns

DNS only matters for exit relays.

The way DNS queries are resolved is up to the individual exit relay operator. SOME exit relays operators choose to use Google's public DNS service. Many don't.

September 19, 2017

Permalink

Does it exist yet a library, a cypher, an algorithm, a 'savoir-faire' for implementing quantum resistance on line ?
Is it so irrelevant or outside the scoop ?

KIST counts & dispatches (at least the result lol) but how-to be certain that data recollected are true ?
The more you add the more the count is false : it is the famous mathematics law of errors.

Is it not dangerous to built a system from scratch without a board/box dedicated ?
should not it be better if the relays & the users should use the same 'special Tor box/board' ?

Could tor be implemented in a ssd manufactured as a ram (a fake ddr but a real micro-harddisk) [or as an electronic component that i should insert in the mother card] that the user could set/update from a special program ?
The new laws , almost applied everywhere , say that even an usb-flash can be seized as soon as police can see it. Masked as a ddr_mini ssd, the user could plug it and be maybe more anonymized. I should prefer buy this 'special ssdram/ddr' than a tor box _ mini computer _ router _ .

Could it be in the kernel running like an ime friendly user element ? Should it be possible if ime should become opened to implemented tor inside the kernel as a function in the hands of the user ? Should not it be a monster doing the opposite (privacy/anonymity) that it should do : a layer against this invasive & destructive "insane curiosity" ?

Could nova file system improve the usage of tor ?

September 20, 2017

Permalink

Added new GETINFO targets "ip-to-country/{ipv4,ipv6}-available", so controllers can tell whether the geoip databases are loaded. Closes ticket 23237.

I'm sorry for my stupid question, but I need to use tor's geoIP in my scripts to decide which country each particular Tor node belongs to. I now use geoiplookup tool to map IP to country, but that relies on its own DB. I would like to use Tor's DB for GeoIP. How I can do that? Does this option gives me the result I need directly?

Sorry again, I understood answer to my question. Indeed, this option does exactly what I want! Now I use getinfo ip-to-country/X.X.X.X in my scripts, and it works fine. Thanks a lot for this improvement!

Improvement? Browser status bars getting throttled to death waiting for bizarre chained Url's? How does that relate to improvement? You have not got lost at jpg4.us.

September 20, 2017

Permalink

Any compile manual here? baniry is too late, always release with the TBB, but TBB release time is too long.

So the new hidden services don't actually have any improvement that makes it harder to locate them.

My guess is that you have no idea on how important "Improved directory protocol, leaking much less information to directory servers" is, where the HSDir wont be able to gather and discover new onion services than before.

OP is right, harvesting is presence and has nothing to do with locating the known... attackers can still drop sybils as guards, start modulating their own test traffic to hs, eventually they get a pattern hit through guard to the onions IP.... game over.

September 23, 2017

Permalink

Since this mornig, saturday 23th september, TOR is very unstable and quit about each 10 minutes… I reload et I'm quite for another 10 minutes. I'm in France, I use Tor browser on Macintosh. When reloaded, TOR works fine, but there this problem anytime (almost 20 times today).
Has anybody experienced this problem ?

September 25, 2017

In reply to gk

Permalink

My problem, since saturday 23th is continuing. Approximately every ten minutes (more or less), a small window informs me that TOR has unexpectedly left.
This small window has two buttons, exit and restart. I click on "restart" and everything works normally again.
I use the last version of Tor browser on Mac OS X, Tor 7.0.5.

September 23, 2017

Permalink

I think that they have implemented guard pinning, layering, and rotation periods that make it harder for an attacker to move up the proxy chain to achieve a direct connection to the HS.

This basically means that the HSs' guards are themselves guarded by two layers of guards. Each layer having a different rotation period. Im unsure what the client side protections are aside from making end to end correlation more difficult.

It would be nice if the devs would elaborate on the new protections the new HS design affords its users on both ends. A thorough blog addressing this would be great. A video would be ideal.

September 29, 2017

Permalink

>SHA3
NSA-backed

>ECC
>NSA-backed

ooook....

September 30, 2017

Permalink

Hi

October 10, 2017

Permalink

How to use Tor as a tunneled HTTP proxy? I added the HTTPTunnelPort statement in my torrc file to open a HTTP listen port. However, when I tried to use the proxy to visit some hidden services, "Saying: HTTP/1.0 405 Method Not Allowed" appeared in the notice log. It seemed that I can not use Tor as a HTTP proxy.