1 |
Hello, everyone. |
2 |
|
3 |
The official git migration inspired some real changes in the semi- |
4 |
related Git mirror [1] and Repo mirror & CI [2] projects. In this mail, |
5 |
I'd like to shortly summarize what changed and how to proceed forward. |
6 |
|
7 |
TL;DR, short summary: |
8 |
|
9 |
1. rsync->git mirror has been disabled and the project obsoleted. |
10 |
|
11 |
2. Pull requests are now handled on official repo mirror [3]. |
12 |
|
13 |
3. User syncing can be done via cache-enabled mirror [4] maintained by |
14 |
'Repository mirror & CI' project. The mirror has less lag than the old |
15 |
one, and is updated every 20 minutes. |
16 |
|
17 |
4. All user syncing mirrors [5] now retain history from git & svn, |
18 |
and can be used with layman using alternative repositories.xml [6]. |
19 |
|
20 |
5. Travis CI is not used at the moment. However, repo mirror & CI runs |
21 |
pkgcheck every 30 minutes and publishes the results at [11]. |
22 |
|
23 |
|
24 |
rsync->git mirror |
25 |
================= |
26 |
|
27 |
The rsync->git mirror has been officially disabled. Since the Gentoo |
28 |
repository is now in git, there is no point in mirroring from rsync. |
29 |
All the remaining Pull Requests were closed with an explanatory note |
30 |
referring to the new official mirror. |
31 |
|
32 |
I'd like to thank all the people who run and supported the mirror, |
33 |
and to all our contributors. You've been doing a great job, and made |
34 |
this effort beneficial to Gentoo. Hope to see you soon on the new |
35 |
mirror :). |
36 |
|
37 |
|
38 |
Pull requests on the official mirror |
39 |
==================================== |
40 |
|
41 |
While the reference copy of our repository [7] is hosted by our |
42 |
Infrastructure, we are also running an official GitHub mirror [3]. |
43 |
The mirror is updated by gitolite automatically on each push, |
44 |
and therefore is always up-to-date and does not require any effort from |
45 |
Gentoo developers. |
46 |
|
47 |
The official mirror also accepts pull requests from our community. |
48 |
Unlike with the rsync->git mirror, the commits can be merged directly |
49 |
in the repository, making contributing via pull requests much closer to |
50 |
being a Gentoo developer. |
51 |
|
52 |
For users willing to contribute, I suggest reading our Git workflow |
53 |
[8] documentation and either the key requirements listed in GLEP 63 [9], |
54 |
or the instructions for generating matching keys [10]. If you can |
55 |
provide a nicely fit commit with proper message and signature, you |
56 |
can get all the kudos :). |
57 |
|
58 |
For developers willing to help, proper reviews and assignments of pull |
59 |
requests is always welcome. We'd like pull requests to provide both |
60 |
the means to contribute easily and efficiently, and to help our users |
61 |
improve their ebuild skills and join the developer community in |
62 |
the future! So if you decide to fix minor issues with the contributions |
63 |
yourself, please leave an explanatory comment so that the contributor |
64 |
does not repeat the same mistake. |
65 |
|
66 |
Please also remember to ping package maintainers. We usually do that |
67 |
through @-highlights in comments. The developers who are direct package |
68 |
maintainers can be referenced directly (@mgorny), while teams and |
69 |
projects can be referenced via GitHub teams (@Gentoo/qt). Sadly, only |
70 |
a handful of the teams is mirrored on GitHub right now and organization |
71 |
admin rights are required to add new teams, so you'd have to ping one |
72 |
of the 'Owners' (usually me) to mirror more as necessary. |
73 |
|
74 |
|
75 |
User syncing mirror |
76 |
=================== |
77 |
|
78 |
Being the development repository, the official git mirror lacks |
79 |
metadata cache and other external resources and therefore is not |
80 |
suitable for user syncing. For this goal, I recommend using |
81 |
the syncing mirror [4] that has been provided by the Repository mirror |
82 |
& CI project [2] for some time already. |
83 |
|
84 |
This mirror combines the development repository history with |
85 |
pre-generated caches. Currently it misses the external resources |
86 |
(DTDs, GLEPs, news items, herds.xml) but there is an ongoing effort of |
87 |
finding a clean way to merge them. |
88 |
|
89 |
The mirror also preserves the original commit signatures, though the |
90 |
merge commits and cache update commits are not signed at the moment. |
91 |
For verification, I recommend verifying top-most original commit |
92 |
signature and 'git diff'-ing HEAD to it to confirm that only caches were |
93 |
modified/added. |
94 |
|
95 |
It should be also noted that since the mirror is created from |
96 |
the original repository rather than from poorly-timed rsync snapshots |
97 |
of the repository, it has less lag. |
98 |
|
99 |
|
100 |
Repository mirror updates |
101 |
========================= |
102 |
|
103 |
Given the necessity of improving Gentoo repository mirroring, I've |
104 |
prepared and deployed a few improvements to the repository mirrors [5]. |
105 |
|
106 |
First of all, the mirroring scripts now output a repositories.xml [6] |
107 |
file that can be used as a drop-in replacement for the official list, |
108 |
replacing the upstream source locations with our mirrors. So if you'd |
109 |
like layman to sync using mirrors with pre-generated metadata cache, |
110 |
just use the following in layman.cfg: |
111 |
|
112 |
overlays : |
113 |
https://api.gentoo.org/overlays/repositories.xml |
114 |
https://gentoo.github.io/repo-qa-check-results/repositories.xml |
115 |
|
116 |
Secondly, the mirrors now preserve the commit history from the upstream |
117 |
repository. The commits from the original repository are merged into |
118 |
the mirror using recursive merges, therefore preserving copies of the |
119 |
original commits. Metadata updates are committed on top of that. |
120 |
|
121 |
The script is additionally prepared to handle upstream rewrites |
122 |
(repository replacement, forced pushes) gracefully. In case of such |
123 |
an event, a 'ours' strategy merge is performed to replace the HEAD with |
124 |
the new tree, and caches are re-committed on top of that. This |
125 |
guarantees that the tree at merge commit is 100% exact to the original |
126 |
repository tree (the previous mirror tree is pretty much discarded) yet |
127 |
users can perform a clean, fast-forward pull. |
128 |
|
129 |
As noted above, the merges and metadata updates are not GPG-signed at |
130 |
the moment. It is unclear whether automated signatures will be added |
131 |
in the future since I can't guarantee the proper security for |
132 |
the merges and metadata updates. Instead, it is safer to verify |
133 |
signatures on the original commits and compare the merged result with |
134 |
them (no files other than caches can be modified). |
135 |
|
136 |
Thirdly, I've create a new pkgcore syncer using git-svn, and switched |
137 |
subversion repository mirroring to it. As a result, the mirrors also |
138 |
preserve history of subversion repositories. For no good reason, |
139 |
probably ;). |
140 |
|
141 |
And lastly, I've fixed and improve cache generation capabilities of |
142 |
pkgcore. The resulting caches should be fully correct and compatible |
143 |
with Portage now, and use.local.desc and pkg_desc_index are generated |
144 |
as well. If you notice any differences or issues, please report. |
145 |
|
146 |
|
147 |
Gentoo pkgcheck CI runs |
148 |
======================= |
149 |
|
150 |
None of the new repositories uses Travis-CI at the moment. The most |
151 |
important reason is that it requires .travis.yml to be stored |
152 |
in the repository while it really does not belong there. We will be |
153 |
looking for a suitable replacement though. |
154 |
|
155 |
The master branch of the repository itself is still subject to my |
156 |
pkgcheck runs which are run every 30 minutes. The results are published |
157 |
at [11], though there is no breakage reporting at the moment, or direct |
158 |
access to historical data (though it's preserved in the git |
159 |
repository). |
160 |
|
161 |
Still, we need CI for pull requests. The pkgcheck runs could use |
162 |
a proper UI, and possibly bisect to identify the commits introducing |
163 |
breakage. In other words, there's a lot of space for improvement. |
164 |
|
165 |
|
166 |
So once again, thanks a lot, everyone and hope the new services suit |
167 |
you even more :). |
168 |
|
169 |
|
170 |
[1]:https://wiki.gentoo.org/wiki/Project:Git_mirror |
171 |
[2]:https://wiki.gentoo.org/wiki/Project:Repository_mirror_and_CI |
172 |
[3]:https://github.com/gentoo/gentoo |
173 |
[4]:https://github.com/gentoo-mirror/gentoo |
174 |
[5]:https://github.com/gentoo-mirror |
175 |
[6]:https://gentoo.github.io/repo-qa-check-results/repositories.xml |
176 |
[7]:https://gitweb.gentoo.org/repo/gentoo.git |
177 |
[8]:https://wiki.gentoo.org/wiki/Gentoo_git_workflow |
178 |
[9]:https://wiki.gentoo.org/wiki/GLEP:63 |
179 |
[10]:https://wiki.gentoo.org/wiki/Project:Gentoo-keys/Generating_GLEP_63_based_OpenPGP_keys |
180 |
[11]:http://gentoo.github.io/gentoo-qa-results/global.html |
181 |
|
182 |
-- |
183 |
Best regards, |
184 |
Michał Górny |
185 |
<http://dev.gentoo.org/~mgorny/> |