1 |
Hello, |
2 |
|
3 |
I've reduced my work hours this week and used the extra time to finally |
4 |
finish setting up some scripts for the repository mirror & CI project. |
5 |
They're all now running in cron on the machine offered by Todd Goodman |
6 |
for the task, and uploading the results to github. |
7 |
|
8 |
|
9 |
Long story short, what we have now is: |
10 |
|
11 |
1. basic QA checks on repos (repo_name, masters, global-scope 'die' |
12 |
calls -- like ebuilds using EAPIs removed from eclass), uploaded to |
13 |
[1]. The results include a summary page for all repositories |
14 |
and a detailed log for each repository (full of cruft at the moment). |
15 |
|
16 |
2. Git mirrors of repositories supported by the scripts and passing |
17 |
basic QA checks, with metadata cache pregenerated. Hosted at [2]. |
18 |
|
19 |
3. Non-Travis runs of pkgcheck on gentoo. Now with plain HTML results |
20 |
(no AJAX!), distinction between warnings and errors (and colors that |
21 |
make sense) and quick links to look through all of them. Starts at [3]. |
22 |
|
23 |
4. Also, automated runs of gpyutils. Not that it's very related but |
24 |
since I had cron and gentoo repo handy, why not? Result files linked at |
25 |
the sub-project page [4]. |
26 |
|
27 |
The current TODO includes: |
28 |
|
29 |
A. Moving the result pages to Gentoo hosting. This requires Infra |
30 |
action per bugs [5,6]. |
31 |
|
32 |
B. Enabling the script to automatically file bugs for new basic QA |
33 |
violations (repo_name, masters). Requires Infra action per bug [7]. |
34 |
|
35 |
And before I get into details, the cron schedule for people willing to |
36 |
look at the results: |
37 |
|
38 |
1-2. repo check is run every hour, at HH:17. The full run usually takes |
39 |
around 5 minutes but major eclass changes can prolong it. |
40 |
|
41 |
3. gentoo pkgcheck is run every 30 minutes, at HH:04 and HH:34. It |
42 |
takes 7-8 minutes. |
43 |
|
44 |
4. gpyutils are run every 4 hours, at HH:47 UTC. It takes around 15 |
45 |
minutes. |
46 |
|
47 |
Please note that github has some website caching, so the results get |
48 |
updated few minutes later. |
49 |
|
50 |
|
51 |
Now, a few more boring details for people interested. |
52 |
|
53 |
The main CI scripts are run on all repositories listed |
54 |
in repositories.xml (in api.gentoo.org). Layman is used to parse |
55 |
the scripts; the syncing is done using pkgcore (if possible) or |
56 |
portage. |
57 |
|
58 |
Since pkgcore has partial and broken repos.conf support, all initial |
59 |
syncing is done using portage sync modules. This limits supported |
60 |
VCS-es to git and svn. Portage supposedly supports rsync too but |
61 |
the module works only with gentoo repository, so I unconditionally |
62 |
assume it's unsupported. |
63 |
|
64 |
So if your repository is marked as unsupported, you can either: |
65 |
|
66 |
a) switch to git, |
67 |
|
68 |
b) fix pkgcore to handle repos.conf properly (assuming it supports |
69 |
the vcs), |
70 |
|
71 |
c) add/fix portage sync module for the vcs. |
72 |
|
73 |
If a repository is considered supported, the script runs a few basic |
74 |
checks on it, and then regenerates the metadata cache. The checks |
75 |
include: |
76 |
|
77 |
- checking for set and correct repo_name (must be equal to the repo |
78 |
name on the list), |
79 |
|
80 |
- checking for set and correct masters= (must only list repositories |
81 |
available via the list), |
82 |
|
83 |
- checking for other metadata violations that make pkgcore unhappy ;), |
84 |
|
85 |
- checking if any failures happen during metadata cache regen (like |
86 |
'die' calls in global scope). |
87 |
|
88 |
This generates .txt logs for each repository and a summary.json file |
89 |
containing copy of repositories.xml metadata along with check results. |
90 |
The logs and summary are formatted as HTML, and uploaded. |
91 |
|
92 |
In the future, at this point another script would compare the new |
93 |
summary with bug database, and report new issues or close fixed ones. |
94 |
Assuming people won't block it because of 'potential bug spam'... |
95 |
|
96 |
Afterwards, another Python script updates GitHub repository list. New |
97 |
repositories without major QA issues are added to the mirror, while |
98 |
removed or unsupported repositories are removed. Repositories that |
99 |
start having QA issues afterwards are not removed from the list to avoid |
100 |
extra noise. |
101 |
|
102 |
For each mirrored repository, the local checkout (with generated |
103 |
metadata cache) is rsynced into git, and the update is pushed |
104 |
as a single commit. |
105 |
|
106 |
And I guess that would be all the gory details. I'm planning to publish |
107 |
the scripts as soon as I have time to clean them up a bit. |
108 |
|
109 |
|
110 |
[1]:http://gentoo.github.io/repo-qa-check-results/ |
111 |
[2]:https://github.com/gentoo-mirror |
112 |
[3]:http://gentoo.github.io/gentoo-qa-results/global.html |
113 |
[4]:https://wiki.gentoo.org/wiki/Project:GPyUtils |
114 |
[5]:https://bugs.gentoo.org/show_bug.cgi?id=552520 |
115 |
[6]:https://bugs.gentoo.org/show_bug.cgi?id=552524 |
116 |
[7]:https://bugs.gentoo.org/show_bug.cgi?id=552518 |
117 |
|
118 |
-- |
119 |
Best regards, |
120 |
Michał Górny |
121 |
<http://dev.gentoo.org/~mgorny/> |