1 |
On 11/19/19 8:34 AM, Pacho Ramos wrote: |
2 |
> El mar, 19-11-2019 a las 00:21 +0000, Sergei Trofimovich escribió: |
3 |
>> repoman slows down ~linearly with amount of profiles being scanned. |
4 |
>> In case of amd64 we have 28 stable profiles. |
5 |
>> |
6 |
>> To speed up processing and fit into time budged of various CIs we can |
7 |
>> split the work across different processes that handle different profiles. |
8 |
>> |
9 |
>> Example benchmark on ::haskell overlay: |
10 |
>> $ ./repoman full --include-arches=amd64 |
11 |
>> ~65 minutes |
12 |
>> $ ./repoman full --include-profiles=default/linux/amd64/17.0 |
13 |
>> ~4 minutes |
14 |
>> This allows for a crude sharding of work across processes and allows for |
15 |
>> cheap tree-wide scans for early failures. |
16 |
>> |
17 |
> |
18 |
> Just for knowing (as I guess there is a technical issue preventing that), why |
19 |
> repoman is not trying to check one profile per core in parallel by default by |
20 |
> itself? |
21 |
|
22 |
Some things to consider when implementing that: |
23 |
|
24 |
* Some metadata cache backends do not work across forks (sqlite). |
25 |
|
26 |
* You'll need a master process to aggregate serialized results from |
27 |
child processes. |
28 |
|
29 |
* Some work which is currently done only once would likely have to be |
30 |
repeated in parallel child processes, for example multiple processes |
31 |
would be simultaneously parsing the same metadata cache entries or even |
32 |
generating metadata for the same package simultaneously. |
33 |
|
34 |
> Thanks a lot for the info :) |
35 |
-- |
36 |
Thanks, |
37 |
Zac |