Gentoo Archives: gentoo-dev-announce

From: "Michał Górny" <mgorny@g.o>
To: gentoo-dev-announce@l.g.o
Cc: gentoo-dev@l.g.o
Subject: [gentoo-dev-announce] rsync->git mirror discontinued, repo mirror & CI updates
Date: Thu, 13 Aug 2015 19:45:15
1 Hello, everyone.
3 The official git migration inspired some real changes in the semi-
4 related Git mirror [1] and Repo mirror & CI [2] projects. In this mail,
5 I'd like to shortly summarize what changed and how to proceed forward.
7 TL;DR, short summary:
9 1. rsync->git mirror has been disabled and the project obsoleted.
11 2. Pull requests are now handled on official repo mirror [3].
13 3. User syncing can be done via cache-enabled mirror [4] maintained by
14 'Repository mirror & CI' project. The mirror has less lag than the old
15 one, and is updated every 20 minutes.
17 4. All user syncing mirrors [5] now retain history from git & svn,
18 and can be used with layman using alternative repositories.xml [6].
20 5. Travis CI is not used at the moment. However, repo mirror & CI runs
21 pkgcheck every 30 minutes and publishes the results at [11].
24 rsync->git mirror
25 =================
27 The rsync->git mirror has been officially disabled. Since the Gentoo
28 repository is now in git, there is no point in mirroring from rsync.
29 All the remaining Pull Requests were closed with an explanatory note
30 referring to the new official mirror.
32 I'd like to thank all the people who run and supported the mirror,
33 and to all our contributors. You've been doing a great job, and made
34 this effort beneficial to Gentoo. Hope to see you soon on the new
35 mirror :).
38 Pull requests on the official mirror
39 ====================================
41 While the reference copy of our repository [7] is hosted by our
42 Infrastructure, we are also running an official GitHub mirror [3].
43 The mirror is updated by gitolite automatically on each push,
44 and therefore is always up-to-date and does not require any effort from
45 Gentoo developers.
47 The official mirror also accepts pull requests from our community.
48 Unlike with the rsync->git mirror, the commits can be merged directly
49 in the repository, making contributing via pull requests much closer to
50 being a Gentoo developer.
52 For users willing to contribute, I suggest reading our Git workflow
53 [8] documentation and either the key requirements listed in GLEP 63 [9],
54 or the instructions for generating matching keys [10]. If you can
55 provide a nicely fit commit with proper message and signature, you
56 can get all the kudos :).
58 For developers willing to help, proper reviews and assignments of pull
59 requests is always welcome. We'd like pull requests to provide both
60 the means to contribute easily and efficiently, and to help our users
61 improve their ebuild skills and join the developer community in
62 the future! So if you decide to fix minor issues with the contributions
63 yourself, please leave an explanatory comment so that the contributor
64 does not repeat the same mistake.
66 Please also remember to ping package maintainers. We usually do that
67 through @-highlights in comments. The developers who are direct package
68 maintainers can be referenced directly (@mgorny), while teams and
69 projects can be referenced via GitHub teams (@Gentoo/qt). Sadly, only
70 a handful of the teams is mirrored on GitHub right now and organization
71 admin rights are required to add new teams, so you'd have to ping one
72 of the 'Owners' (usually me) to mirror more as necessary.
75 User syncing mirror
76 ===================
78 Being the development repository, the official git mirror lacks
79 metadata cache and other external resources and therefore is not
80 suitable for user syncing. For this goal, I recommend using
81 the syncing mirror [4] that has been provided by the Repository mirror
82 & CI project [2] for some time already.
84 This mirror combines the development repository history with
85 pre-generated caches. Currently it misses the external resources
86 (DTDs, GLEPs, news items, herds.xml) but there is an ongoing effort of
87 finding a clean way to merge them.
89 The mirror also preserves the original commit signatures, though the
90 merge commits and cache update commits are not signed at the moment.
91 For verification, I recommend verifying top-most original commit
92 signature and 'git diff'-ing HEAD to it to confirm that only caches were
93 modified/added.
95 It should be also noted that since the mirror is created from
96 the original repository rather than from poorly-timed rsync snapshots
97 of the repository, it has less lag.
100 Repository mirror updates
101 =========================
103 Given the necessity of improving Gentoo repository mirroring, I've
104 prepared and deployed a few improvements to the repository mirrors [5].
106 First of all, the mirroring scripts now output a repositories.xml [6]
107 file that can be used as a drop-in replacement for the official list,
108 replacing the upstream source locations with our mirrors. So if you'd
109 like layman to sync using mirrors with pre-generated metadata cache,
110 just use the following in layman.cfg:
112 overlays :
116 Secondly, the mirrors now preserve the commit history from the upstream
117 repository. The commits from the original repository are merged into
118 the mirror using recursive merges, therefore preserving copies of the
119 original commits. Metadata updates are committed on top of that.
121 The script is additionally prepared to handle upstream rewrites
122 (repository replacement, forced pushes) gracefully. In case of such
123 an event, a 'ours' strategy merge is performed to replace the HEAD with
124 the new tree, and caches are re-committed on top of that. This
125 guarantees that the tree at merge commit is 100% exact to the original
126 repository tree (the previous mirror tree is pretty much discarded) yet
127 users can perform a clean, fast-forward pull.
129 As noted above, the merges and metadata updates are not GPG-signed at
130 the moment. It is unclear whether automated signatures will be added
131 in the future since I can't guarantee the proper security for
132 the merges and metadata updates. Instead, it is safer to verify
133 signatures on the original commits and compare the merged result with
134 them (no files other than caches can be modified).
136 Thirdly, I've create a new pkgcore syncer using git-svn, and switched
137 subversion repository mirroring to it. As a result, the mirrors also
138 preserve history of subversion repositories. For no good reason,
139 probably ;).
141 And lastly, I've fixed and improve cache generation capabilities of
142 pkgcore. The resulting caches should be fully correct and compatible
143 with Portage now, and use.local.desc and pkg_desc_index are generated
144 as well. If you notice any differences or issues, please report.
147 Gentoo pkgcheck CI runs
148 =======================
150 None of the new repositories uses Travis-CI at the moment. The most
151 important reason is that it requires .travis.yml to be stored
152 in the repository while it really does not belong there. We will be
153 looking for a suitable replacement though.
155 The master branch of the repository itself is still subject to my
156 pkgcheck runs which are run every 30 minutes. The results are published
157 at [11], though there is no breakage reporting at the moment, or direct
158 access to historical data (though it's preserved in the git
159 repository).
161 Still, we need CI for pull requests. The pkgcheck runs could use
162 a proper UI, and possibly bisect to identify the commits introducing
163 breakage. In other words, there's a lot of space for improvement.
166 So once again, thanks a lot, everyone and hope the new services suit
167 you even more :).
170 [1]:
171 [2]:
172 [3]:
173 [4]:
174 [5]:
175 [6]:
176 [7]:
177 [8]:
178 [9]:
179 [10]:
180 [11]:
182 --
183 Best regards,
184 Michał Górny
185 <>