| 1 |
Hello, |
| 2 |
|
| 3 |
I've reduced my work hours this week and used the extra time to finally |
| 4 |
finish setting up some scripts for the repository mirror & CI project. |
| 5 |
They're all now running in cron on the machine offered by Todd Goodman |
| 6 |
for the task, and uploading the results to github. |
| 7 |
|
| 8 |
|
| 9 |
Long story short, what we have now is: |
| 10 |
|
| 11 |
1. basic QA checks on repos (repo_name, masters, global-scope 'die' |
| 12 |
calls -- like ebuilds using EAPIs removed from eclass), uploaded to |
| 13 |
[1]. The results include a summary page for all repositories |
| 14 |
and a detailed log for each repository (full of cruft at the moment). |
| 15 |
|
| 16 |
2. Git mirrors of repositories supported by the scripts and passing |
| 17 |
basic QA checks, with metadata cache pregenerated. Hosted at [2]. |
| 18 |
|
| 19 |
3. Non-Travis runs of pkgcheck on gentoo. Now with plain HTML results |
| 20 |
(no AJAX!), distinction between warnings and errors (and colors that |
| 21 |
make sense) and quick links to look through all of them. Starts at [3]. |
| 22 |
|
| 23 |
4. Also, automated runs of gpyutils. Not that it's very related but |
| 24 |
since I had cron and gentoo repo handy, why not? Result files linked at |
| 25 |
the sub-project page [4]. |
| 26 |
|
| 27 |
The current TODO includes: |
| 28 |
|
| 29 |
A. Moving the result pages to Gentoo hosting. This requires Infra |
| 30 |
action per bugs [5,6]. |
| 31 |
|
| 32 |
B. Enabling the script to automatically file bugs for new basic QA |
| 33 |
violations (repo_name, masters). Requires Infra action per bug [7]. |
| 34 |
|
| 35 |
And before I get into details, the cron schedule for people willing to |
| 36 |
look at the results: |
| 37 |
|
| 38 |
1-2. repo check is run every hour, at HH:17. The full run usually takes |
| 39 |
around 5 minutes but major eclass changes can prolong it. |
| 40 |
|
| 41 |
3. gentoo pkgcheck is run every 30 minutes, at HH:04 and HH:34. It |
| 42 |
takes 7-8 minutes. |
| 43 |
|
| 44 |
4. gpyutils are run every 4 hours, at HH:47 UTC. It takes around 15 |
| 45 |
minutes. |
| 46 |
|
| 47 |
Please note that github has some website caching, so the results get |
| 48 |
updated few minutes later. |
| 49 |
|
| 50 |
|
| 51 |
Now, a few more boring details for people interested. |
| 52 |
|
| 53 |
The main CI scripts are run on all repositories listed |
| 54 |
in repositories.xml (in api.gentoo.org). Layman is used to parse |
| 55 |
the scripts; the syncing is done using pkgcore (if possible) or |
| 56 |
portage. |
| 57 |
|
| 58 |
Since pkgcore has partial and broken repos.conf support, all initial |
| 59 |
syncing is done using portage sync modules. This limits supported |
| 60 |
VCS-es to git and svn. Portage supposedly supports rsync too but |
| 61 |
the module works only with gentoo repository, so I unconditionally |
| 62 |
assume it's unsupported. |
| 63 |
|
| 64 |
So if your repository is marked as unsupported, you can either: |
| 65 |
|
| 66 |
a) switch to git, |
| 67 |
|
| 68 |
b) fix pkgcore to handle repos.conf properly (assuming it supports |
| 69 |
the vcs), |
| 70 |
|
| 71 |
c) add/fix portage sync module for the vcs. |
| 72 |
|
| 73 |
If a repository is considered supported, the script runs a few basic |
| 74 |
checks on it, and then regenerates the metadata cache. The checks |
| 75 |
include: |
| 76 |
|
| 77 |
- checking for set and correct repo_name (must be equal to the repo |
| 78 |
name on the list), |
| 79 |
|
| 80 |
- checking for set and correct masters= (must only list repositories |
| 81 |
available via the list), |
| 82 |
|
| 83 |
- checking for other metadata violations that make pkgcore unhappy ;), |
| 84 |
|
| 85 |
- checking if any failures happen during metadata cache regen (like |
| 86 |
'die' calls in global scope). |
| 87 |
|
| 88 |
This generates .txt logs for each repository and a summary.json file |
| 89 |
containing copy of repositories.xml metadata along with check results. |
| 90 |
The logs and summary are formatted as HTML, and uploaded. |
| 91 |
|
| 92 |
In the future, at this point another script would compare the new |
| 93 |
summary with bug database, and report new issues or close fixed ones. |
| 94 |
Assuming people won't block it because of 'potential bug spam'... |
| 95 |
|
| 96 |
Afterwards, another Python script updates GitHub repository list. New |
| 97 |
repositories without major QA issues are added to the mirror, while |
| 98 |
removed or unsupported repositories are removed. Repositories that |
| 99 |
start having QA issues afterwards are not removed from the list to avoid |
| 100 |
extra noise. |
| 101 |
|
| 102 |
For each mirrored repository, the local checkout (with generated |
| 103 |
metadata cache) is rsynced into git, and the update is pushed |
| 104 |
as a single commit. |
| 105 |
|
| 106 |
And I guess that would be all the gory details. I'm planning to publish |
| 107 |
the scripts as soon as I have time to clean them up a bit. |
| 108 |
|
| 109 |
|
| 110 |
[1]:http://gentoo.github.io/repo-qa-check-results/ |
| 111 |
[2]:https://github.com/gentoo-mirror |
| 112 |
[3]:http://gentoo.github.io/gentoo-qa-results/global.html |
| 113 |
[4]:https://wiki.gentoo.org/wiki/Project:GPyUtils |
| 114 |
[5]:https://bugs.gentoo.org/show_bug.cgi?id=552520 |
| 115 |
[6]:https://bugs.gentoo.org/show_bug.cgi?id=552524 |
| 116 |
[7]:https://bugs.gentoo.org/show_bug.cgi?id=552518 |
| 117 |
|
| 118 |
-- |
| 119 |
Best regards, |
| 120 |
Michał Górny |
| 121 |
<http://dev.gentoo.org/~mgorny/> |
Archives