1 |
The two most common errors that occur in the rsync-gen process are |
2 |
missing DIST entries in the Manifest committed to git, and |
3 |
metadata.xml that fails to validate against the DTD. Two much less |
4 |
common errors that have occurred in the last 24 hours which prompted |
5 |
this email are FILESDIR items that are way too large (>60 KiB), and |
6 |
completely missing metadata.xml. All four of these errors are fatal |
7 |
to the rsync-gen process, and for good reason. |
8 |
|
9 |
Anything fatal to rsync-gen completely stops the git to rsync |
10 |
conversion, and the rsync tree remains at the state it was before the |
11 |
failure until the failure is resolved. If it isn't apparent, this |
12 |
completely blocks all updates, GLSA distribution, news item |
13 |
distribution, and anything else distributed by the rsync tree either |
14 |
now or in the future, until the problem is fixed. If you're curious |
15 |
about the current age of the rsync tree due to failures, you can |
16 |
retrieve the metadata/timestamp or metadata/timestamp.x (not |
17 |
timestamp.chk) file from rsync.gentoo.org and inspect its contents (or |
18 |
do a verbose listing of the file, and look at the mtime). The |
19 |
rsync.gentoo.org rotation only contains servers managed by infra, |
20 |
which sync against the private master rsync server very frequently, so |
21 |
you can be reasonably sure they're about as up to date as possible. |
22 |
Explanation of the various timestamp files: |
23 |
|
24 |
* timestamp: created by rsync-gen script in the staging directory |
25 |
immediately after md5-cache generation and manifest thickening is |
26 |
complete. |
27 |
* timestamp.x: created by cron job periodically (script comment says |
28 |
every 5 minutes, but I think it's less often than that) in the staging |
29 |
directory |
30 |
* timestamp.chk: created by cron job periodically (again, script |
31 |
comment says every 5 minutes, but I think it's every 15 minutes) in |
32 |
the *final* directory |
33 |
|
34 |
The final directory is what is served by the private master rsync |
35 |
server, so you can use timestamp.chk to see the age of your local |
36 |
mirror relative to the private master rsync server. The staging |
37 |
directory is transferred to the final directory as part of successful |
38 |
completion of the rsync-gen script (it's the last step), so if |
39 |
anything bails before that, it doesn't happen (thus, timestamp and |
40 |
timestamp.x are mostly equivalent in meaning). |
41 |
|
42 |
Why do these happen? The answer is very simple: you forgot to run |
43 |
repoman in the package directory before pushing your changes. |
44 |
Alternately, the metadata.xml failing to validate error can happen if |
45 |
your repoman is old. |
46 |
|
47 |
So what can you do about it? First, make sure repoman is installed |
48 |
and up to date. As of portage 2.3.0, repoman is a separate package |
49 |
(app-portage/repoman) with its own release schedule, so once you've |
50 |
updated portage past that version, you need to install repoman |
51 |
yourself. Repoman needs to be at least version 2.3.0 as well (which |
52 |
it will be if you've installed portage 2.3.0 or newer), because the |
53 |
metadata.xml validation check does not exist in any previous version, |
54 |
and as mentioned, that check is fatal to the rsync-gen process. |
55 |
Second, and I cannot stress this enough, *run repoman.* Before you |
56 |
push any changes to a package, run repoman manifest followed by at |
57 |
least repoman manifest-check. Repoman manifest will ensure that all |
58 |
your DIST entries are there, and repoman manifest-check will cover the |
59 |
other three issues I mentioned. Repoman's scan and full commands also |
60 |
include the checks done by manifest-check, so if you're running one of |
61 |
those already prior to pushing your changes, you don't need to do |
62 |
repoman manifest-check as well. Note: pkgcore's repoman-alike (I |
63 |
believe it's called pcheck) does not behave the same as repoman as of |
64 |
this writing, and some checks it does not consider fatal are fatal to |
65 |
repoman (eg the FILESDIR item larger than 60 KiB), so if you're |
66 |
relying on pcheck for your QA checks, you should also do a repoman |
67 |
manifest-check for good measure. This should go without saying, but |
68 |
if you modify an eclass in a way that affects SRC_URI for any package, |
69 |
you should go through the packages that use that eclass and run |
70 |
repoman manifest on them. |
71 |
|
72 |
Remember, QA includes YOU! (This also applies to all of you |
73 |
submitting pull requests or otherwise proxy maintaining packages, and |
74 |
all members of proxy-maint project; you should all be doing these QA |
75 |
steps before you submit or push changes.) |
76 |
|
77 |
Sincerely, |
78 |
|
79 |
The guy who gets emailed every half hour because you broke the rsync |
80 |
conversion until it's fixed, also known as Doug or dwfreed. |