[gentoo-dev-announce] rsync->git mirror discontinued, repo mirror & CI updates

[Michał Górny]

Hello, everyone.

The official git migration inspired some real changes in the semi-
related Git mirror [1] and Repo mirror & CI [2] projects. In this mail,
I'd like to shortly summarize what changed and how to proceed forward.

TL;DR, short summary:

1. rsync->git mirror has been disabled and the project obsoleted.

2. Pull requests are now handled on official repo mirror [3].

3. User syncing can be done via cache-enabled mirror [4] maintained by
'Repository mirror & CI' project. The mirror has less lag than the old
one, and is updated every 20 minutes.

4. All user syncing mirrors [5] now retain history from git & svn,
and can be used with layman using alternative repositories.xml [6].

5. Travis CI is not used at the moment. However, repo mirror & CI runs
pkgcheck every 30 minutes and publishes the results at [11].


rsync->git mirror
=================

The rsync->git mirror has been officially disabled. Since the Gentoo
repository is now in git, there is no point in mirroring from rsync.
All the remaining Pull Requests were closed with an explanatory note
referring to the new official mirror.

I'd like to thank all the people who run and supported the mirror,
and to all our contributors. You've been doing a great job, and made
this effort beneficial to Gentoo. Hope to see you soon on the new
mirror :).


Pull requests on the official mirror
====================================

While the reference copy of our repository [7] is hosted by our
Infrastructure, we are also running an official GitHub mirror [3].
The mirror is updated by gitolite automatically on each push,
and therefore is always up-to-date and does not require any effort from
Gentoo developers.

The official mirror also accepts pull requests from our community.
Unlike with the rsync->git mirror, the commits can be merged directly
in the repository, making contributing via pull requests much closer to
being a Gentoo developer.

For users willing to contribute, I suggest reading our Git workflow
[8] documentation and either the key requirements listed in GLEP 63 [9],
or the instructions for generating matching keys [10]. If you can
provide a nicely fit commit with proper message and signature, you
can get all the kudos :).

For developers willing to help, proper reviews and assignments of pull
requests is always welcome. We'd like pull requests to provide both
the means to contribute easily and efficiently, and to help our users
improve their ebuild skills and join the developer community in
the future! So if you decide to fix minor issues with the contributions
yourself, please leave an explanatory comment so that the contributor
does not repeat the same mistake.

Please also remember to ping package maintainers. We usually do that
through @-highlights in comments. The developers who are direct package
maintainers can be referenced directly (@mgorny), while teams and
projects can be referenced via GitHub teams (@Gentoo/qt). Sadly, only
a handful of the teams is mirrored on GitHub right now and organization
admin rights are required to add new teams, so you'd have to ping one
of the 'Owners' (usually me) to mirror more as necessary.


User syncing mirror
===================

Being the development repository, the official git mirror lacks
metadata cache and other external resources and therefore is not
suitable for user syncing. For this goal, I recommend using
the syncing mirror [4] that has been provided by the Repository mirror
& CI project [2] for some time already.

This mirror combines the development repository history with
pre-generated caches. Currently it misses the external resources
(DTDs, GLEPs, news items, herds.xml) but there is an ongoing effort of
finding a clean way to merge them.

The mirror also preserves the original commit signatures, though the
merge commits and cache update commits are not signed at the moment.
For verification, I recommend verifying top-most original commit
signature and 'git diff'-ing HEAD to it to confirm that only caches were
modified/added.

It should be also noted that since the mirror is created from
the original repository rather than from poorly-timed rsync snapshots
of the repository, it has less lag.


Repository mirror updates
=========================

Given the necessity of improving Gentoo repository mirroring, I've
prepared and deployed a few improvements to the repository mirrors [5].

First of all, the mirroring scripts now output a repositories.xml [6]
file that can be used as a drop-in replacement for the official list,
replacing the upstream source locations with our mirrors. So if you'd
like layman to sync using mirrors with pre-generated metadata cache,
just use the following in layman.cfg:

  overlays  :
    https://api.gentoo.org/overlays/repositories.xml
    https://gentoo.github.io/repo-qa-check-results/repositories.xml

Secondly, the mirrors now preserve the commit history from the upstream
repository. The commits from the original repository are merged into
the mirror using recursive merges, therefore preserving copies of the
original commits. Metadata updates are committed on top of that.

The script is additionally prepared to handle upstream rewrites
(repository replacement, forced pushes) gracefully. In case of such
an event, a 'ours' strategy merge is performed to replace the HEAD with
the new tree, and caches are re-committed on top of that. This
guarantees that the tree at merge commit is 100% exact to the original
repository tree (the previous mirror tree is pretty much discarded) yet
users can perform a clean, fast-forward pull.

As noted above, the merges and metadata updates are not GPG-signed at
the moment. It is unclear whether automated signatures will be added
in the future since I can't guarantee the proper security for
the merges and metadata updates. Instead, it is safer to verify
signatures on the original commits and compare the merged result with
them (no files other than caches can be modified).

Thirdly, I've create a new pkgcore syncer using git-svn, and switched
subversion repository mirroring to it. As a result, the mirrors also
preserve history of subversion repositories. For no good reason,
probably ;).

And lastly, I've fixed and improve cache generation capabilities of
pkgcore. The resulting caches should be fully correct and compatible
with Portage now, and use.local.desc and pkg_desc_index are generated
as well. If you notice any differences or issues, please report.


Gentoo pkgcheck CI runs
=======================

None of the new repositories uses Travis-CI at the moment. The most
important reason is that it requires .travis.yml to be stored
in the repository while it really does not belong there. We will be
looking for a suitable replacement though.

The master branch of the repository itself is still subject to my
pkgcheck runs which are run every 30 minutes. The results are published
at [11], though there is no breakage reporting at the moment, or direct
access to historical data (though it's preserved in the git
repository).

Still, we need CI for pull requests. The pkgcheck runs could use
a proper UI, and possibly bisect to identify the commits introducing
breakage. In other words, there's a lot of space for improvement.


So once again, thanks a lot, everyone and hope the new services suit
you even more :).


[1]:https://wiki.gentoo.org/wiki/Project:Git_mirror
[2]:https://wiki.gentoo.org/wiki/Project:Repository_mirror_and_CI
[3]:https://github.com/gentoo/gentoo
[4]:https://github.com/gentoo-mirror/gentoo
[5]:https://github.com/gentoo-mirror
[6]:https://gentoo.github.io/repo-qa-check-results/repositories.xml
[7]:https://gitweb.gentoo.org/repo/gentoo.git
[8]:https://wiki.gentoo.org/wiki/Gentoo_git_workflow
[9]:https://wiki.gentoo.org/wiki/GLEP:63
[10]:https://wiki.gentoo.org/wiki/Project:Gentoo-keys/Generating_GLEP_63_based_OpenPGP_keys
[11]:http://gentoo.github.io/gentoo-qa-results/global.html

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>