| 1 |
Hello, everyone. |
| 2 |
|
| 3 |
TL;DR: I'm looking for some help to get pull-request CI running |
| 4 |
distributed, i.e. on multiple hosts. |
| 5 |
|
| 6 |
|
| 7 |
Quick history |
| 8 |
------------- |
| 9 |
|
| 10 |
The repo-mirror-ci project pretty much started as a simple cronjob |
| 11 |
running on hardware contributed to the purpose by Todd Goodman |
| 12 |
(considering how much this hardware has done for Gentoo, I'd suggest |
| 13 |
giving him a honorary citizenship of Gentoo or something like |
| 14 |
that ;-)). I can't say they were terrible but they weren't perfect |
| 15 |
either. Due to lack of time, I mostly focused on improving them |
| 16 |
moderately and adding new features. |
| 17 |
|
| 18 |
Today, everything is still running on the same hardware. While |
| 19 |
the hardware is real good, and I'm doing some effort to improve |
| 20 |
the performance of the tools, I think it's quite close to its limits. |
| 21 |
|
| 22 |
Right now, it is running three tasks every 20 minutes: updating repo |
| 23 |
mirrors, running pkgcheck on Gentoo and processing one pull request. |
| 24 |
However, if those tasks delay beyond those 20 minutes, the poor man's |
| 25 |
scheduling based on cron + lockfiles may cause some of the executions |
| 26 |
to be skipped. |
| 27 |
|
| 28 |
In other words, the best case is processing 3 pull requests / h. This |
| 29 |
is starting to be no longer sufficient, and causing significant delays |
| 30 |
during periods of high contributor activity -- while my goal would be to |
| 31 |
provide the results ASAP -- 10-20 minutes would be perfect. |
| 32 |
|
| 33 |
|
| 34 |
The goal and needs |
| 35 |
------------------ |
| 36 |
|
| 37 |
I think the best way forward would be to start distributing tasks |
| 38 |
between more systems. While I don't really have time to make whole |
| 39 |
repo-mirror-ci distributed, I think it should be feasible to start |
| 40 |
splitting pull request processing out of it. |
| 41 |
|
| 42 |
I would use some help to achieve that, esp. wrt distributed task |
| 43 |
scheduling. |
| 44 |
|
| 45 |
Can you think of any tools that could help me get the task done easily |
| 46 |
and with as little of reinventing the wheel as possible? Right now, I |
| 47 |
have just a lot of trivial shell script that checks pull request for |
| 48 |
changes, checks them out, runs 'pmaint regen', pkgcheck, publishes all |
| 49 |
kinds of results and statuses, and also compares QA results to |
| 50 |
determine new failures created by the PR [1]. |
| 51 |
|
| 52 |
As I see it, checking for changes and submitting the results should |
| 53 |
stay on the current host. However, it should be able to run all |
| 54 |
the actual work on slaves and retrieve the results from them. I would |
| 55 |
appreciate all the help with implementing that, plus possibly some |
| 56 |
degree of control to make it possible to update pkgcheck on slaves |
| 57 |
easily. |
| 58 |
|
| 59 |
Please let me know if you're interested in helping. Thanks. |
| 60 |
|
| 61 |
[1]:https://bitbucket.org/mgorny/repo-mirror-ci/src/master/pull-requests.job |
| 62 |
|
| 63 |
-- |
| 64 |
Best regards, |
| 65 |
Michał Górny |
Archives