From Trac into Gitlab for Tor

by gaba | November 20, 2020

Tor has been using Trac until June 2020, when we moved to our self-hosted instance of Gitlab administered by the Tor sysadmin team. We reached some limitations with Trac as well as were concern on some of the plugins we depended on not being mantained. The challenges on doing this migration sooner were and are related to the capacity that we have to adapt a new ticketing system to our needs.
 
We're hoping Gitlab will be a good fit because:
  • Gitlab will allow us to collect our different engineering tools into a single application: Git repository handling, Wiki, Issue tracking, Code reviews, and project management tooling.
  • Gitlab is well-maintained, while Trac plugins are not well maintained and Trac itself hasn't seen a release for over a year (since 2019).
  • Gitlab will allow us to build a more modern approach to handling Continuous Integration for our different projects. This is going to happen after the ticket and wiki migration.
 
We spent several months fixing and testing problems on data migration, from formatting issues to where the information of trac goes to live to in Gitlab. We tested the Gitlab instance with a few projects until we jumped into migrating all data from Trac. You can read about the use cases for a bug tracker at Tor in this ticket. 
 
To accomplish the migration, the Gitlab migration group wrote a number of tools to make the migration happen. These tools were split into two parts: one part for fetching all the state from Trac and a number of tool from turning the Trac state into Gitlab issues and wiki content in various ways. The migration group worked with the engineering teams in the organisation on issues such as splitting up Trac "flat ticket namespace" into the different groups and projects that we wanted to have on Gitlab. This allowed the individual teams at Tor to decide the organisation they wanted to have on Gitlab and allow them to build a mapping that fits better with the project model of Gitlab, where each project have an issue tracker, a wiki, and a repository attached.
 
We specified a specific date for the migration to happen where all of Tor's engineering teams were asked to find different means of doing their work than using Trac while Trac was put in read-only mode. During this period the migration group worked together on the actual migration, verifying that all data was properly migrated to a point where things looked satisfying, and then finally we announced that Gitlab was what we would move to next. We did the transition period with start on Friday afternoon over the weekend to ensure that only a minimum amount of disruption would be caused by this.
 
The period after the migration did involve a bit of support handling from the different teams, but we ar amazed at how quickly everybody picked up the new work flows and we believe that Gitlab have made it easier for engineers to make choices around their respective projects now without needing help from the Gitlab admin team.
 
We are not migrating away from Gitolite and Jenkins just yet. This means those services are still fully operational and their equivalent features in GitLab are not supported (namely Git hosting and CI). Those services might eventually be migrated to GitLab.
 
The issues and wiki of the "Tor" project are migrated. There are no other projects in Trac.
Trac issues that remain are really legacy issues, others issues have been "moved" to the respective projects. All the tickets that were not moved to their respective projects have been closed in the first week of July 2020. Next year we will permanently shut Trac down and keep it archived in the Wayback machine.
 
To request a new account you have to fill the form in https://anonticket.torproject.org/ where we get the request and a few of us attend to them. Through the Outreachy Internship we are mentoring an intern that will help improve this application.
 
To be able to have all issues in one same board we created a main group "tpo" where all our projects live. The structure for the rest of the projects is:
 

Organization: host our main wiki, which links to documentation for all projects at TPO. It also hold issues that may not be related to any particular project but are organizational on TPO.

TPA: host any project related to the infrastructure administered by TPO

  • Gitlab : Any issue or documentation related to running the Gitlab instance.
  • Team: Any issue related to administering TP infrastructure

Core: host projects that are related to mantaining little-t tor

  • arti, tor specifications, shadow, trunnel, tor socks, fallback scripts, directory authorities, chutney, tor
  • Team: Any issue related to processes for the core team and the wiki.

Anti-Censorship: host projects that work on circumventing censorship with Tor

  • gettor, pluggable transports, rdsys, bridgedb, censorship analysis, bridgestrap, emma, state of censorship
  • Team: Any issue related to processes of the team and the wiki.

Network Health: it has all the projects related to monitoring the Tor network

  • doctor, exitmap, torflow, sbws, helper scripts
  • Team: Issues related to network health in general but not to specific projects.

Applications: everything at Tor that is a user facing product

  • tor browser projects, tor launcher, https everywhere
  • Team: issues and wiki related to processes of the team that work on user facing products

Metrics: everything related to collecting, analyzing and visualizing data from the Tor network

  • collector, metrics website, onionperf, weather ,utilities, analysis, exit scanner, exonerator
  • Team: issues and wiki related to the team

Community: is for all the projects that help people that help Tor. 

  • l10n, support, outreach, training
  • Team: anything about processes for the community

Web: all projects and code related to the websites that the Tor project mantains

  • support portal, community, torproject.org main website, donations, styleguide

UX: all projects related to user experience in all the software we develop at the Tor project

  • design, research, media
  • Team: anything about the people working on UX at Tor
 
You can read more about Tor's Gitlab instance in the documentation.
 
Edit: For any question not related to Gitlab or Trac please send a mail to frontdesk at torproject dot org. Thanks!

Comments

Please note that the comment area below has been archived.

Anonymous (not verified) said:

November 20, 2020

Permalink

Thank you for migrating the wiki. Please reduce the CSS padding and margins on tables so tables don't have to be scrolled as far horizontally. Please repair broken tables (example).

Please create a cypherpunks account so we, privacy-oriented Tor users, can revise wiki pages ourselves.

The issue is that to have a cypherpunk account that can revise wiki pages we would have to have it in the groups as a reporter at least. And that would give the account a lot more permissions that we are comfortable with. The wiki does not work so well in gitlab for us as we really need anonymous users to be able to contribute to it. This is something that will still be up for discussion on how to change it.

Anonymous (not verified) said:

November 23, 2020

Permalink

Have you explained this use-case to GitLab's developers to hear from them if it's possible in the current release? I read their documentation and haven't found any loopholes that might allow a situation like it. You can select a user role, but it appears not possible to fine-tune permissions or create custom user roles.

There are good solutions for wikis, though: MediaWiki, DokuWiki, PmWiki, ...

Do we need a valid email to sign up on Tor Project's GitLab?

Anonymous (not verified) said:

November 20, 2020

Permalink

Next year we will permanently shut Trac down and keep it archived in the Wayback machine.

Are you yourselves going to make sure every Trac URL is archived, either manually or by working with archive.org / Archive Team?

We will be working to make sure that everything is archived. Please report if you find anything that is not working as it should. Thanks!

Official Forum… (not verified) said:

November 20, 2020

Permalink

Are you ever planning on making an official web forum? You must've noticed the sharp decline of your user mailing lists. People have to go places like Reddit to get answers and it's just shameful. Even worse, 4chan (shudders).

Anonymous (not verified) said:

November 23, 2020

Permalink

>>> We are discussing the installation of a discourse
But does "Discourse" support anonymous comments?

>>> to act as an official forum and work as comments of this blog.
I would prefer to use something like commenting at "opennet.ru" - it looks like a custom implementation (AFAIR owner discussed some details at sub of "questions to administration"). Implementation is quite democratic (as no need registration) and require no JavaScript (also there is a capture at extra screen to avoid spam messages).

May be it is reasonable to use some custom implementation? - I would vote for implementation that supports
1) anonymous comments feature (no registration)
2) No JavaScript to post messages

Anonymous (not verified) said:

November 23, 2020

Permalink

Just to emphasize: email is completely unsuitable for Tor users. I am sure you yourself understand why and I wish Tor Project had easy options.

The situation can be fixed, and the incoming US administration just might* be willing to pay attention to calls for creating a privacy industry in the USA (as economic stimulus, as helping tech people unemployed by expected breakup of Big Tech companies, as responding to an unmet need by empowering ordinary citizens, etc). Innovation is needed and USG can provide some neccessary seed money.

*Rumors that Biden plans to put Eric Schmidt ('privacy is dead; get over it!") in his cabinet is not encouraging, to be sure.

Anonymous (not verified) said:

November 23, 2020

Permalink

I'm not going to collaborate on gitlab. It's not working without j/s. Old trac was able to edit without j/s

People don't have to go to Reddit or 4chan. People default to those because they reflexively stay inside their social knowledge bubble or don't search for specialized places or are loyal to a perception of convenience, etc. It's like why freer markets go toward oligarchs: Alphabet's Google, Amazon, Facebook, OS app stores, Microsoft, etc. But to get answers about Tor, there are options other than the mailing list:

https://www.torproject.org/contact/
https://support.torproject.org/get-in-touch/
https://blog.torproject.org/help-create-qa-site-tor (Stack Exchange)
https://trac.torproject.org/projects/tor/wiki/UserIssuesResearch

tracpunk (not verified) said:

November 23, 2020

Permalink

gitlab is missing onion addresses. trac had it. how can i contribute and report bug through onions safety?

Anonymous (not verified) said:

November 27, 2020

Permalink

Why Gitlab?

Have you ever tried Gitea (example)?

Gitlab is unusable without JavaScript. Other solutions like Gitea, Mediawiki, and Trac are usable without JS.

Anonymous (not verified) said:

December 10, 2020

Permalink

Why the hell is Trac.torproject.org suddenly redirecting to GitLab today?! The wiki on GitLab is NOT ready for use. Migration must be considered as "in progress" until they are accessibly identical. Do NOT cut off access to Trac at this point. GitLab does NOT have a proper landing page yet. The wiki pages on GitLab are NOT easy for users to find. Various wiki pages are BROKEN on GitLab. Cutting off access should be the LAST thing you do after they are accessibly identical. See also: https://blog.torproject.org/comment/289317#comment-289317