| 1 |
On 11/19/19 8:34 AM, Pacho Ramos wrote: |
| 2 |
> El mar, 19-11-2019 a las 00:21 +0000, Sergei Trofimovich escribió: |
| 3 |
>> repoman slows down ~linearly with amount of profiles being scanned. |
| 4 |
>> In case of amd64 we have 28 stable profiles. |
| 5 |
>> |
| 6 |
>> To speed up processing and fit into time budged of various CIs we can |
| 7 |
>> split the work across different processes that handle different profiles. |
| 8 |
>> |
| 9 |
>> Example benchmark on ::haskell overlay: |
| 10 |
>> $ ./repoman full --include-arches=amd64 |
| 11 |
>> ~65 minutes |
| 12 |
>> $ ./repoman full --include-profiles=default/linux/amd64/17.0 |
| 13 |
>> ~4 minutes |
| 14 |
>> This allows for a crude sharding of work across processes and allows for |
| 15 |
>> cheap tree-wide scans for early failures. |
| 16 |
>> |
| 17 |
> |
| 18 |
> Just for knowing (as I guess there is a technical issue preventing that), why |
| 19 |
> repoman is not trying to check one profile per core in parallel by default by |
| 20 |
> itself? |
| 21 |
|
| 22 |
Some things to consider when implementing that: |
| 23 |
|
| 24 |
* Some metadata cache backends do not work across forks (sqlite). |
| 25 |
|
| 26 |
* You'll need a master process to aggregate serialized results from |
| 27 |
child processes. |
| 28 |
|
| 29 |
* Some work which is currently done only once would likely have to be |
| 30 |
repeated in parallel child processes, for example multiple processes |
| 31 |
would be simultaneously parsing the same metadata cache entries or even |
| 32 |
generating metadata for the same package simultaneously. |
| 33 |
|
| 34 |
> Thanks a lot for the info :) |
| 35 |
-- |
| 36 |
Thanks, |
| 37 |
Zac |
Archives