1 |
Dnia June 28, 2020 3:00:00 AM UTC, Zac Medico <zmedico@g.o> napisał(a): |
2 |
>On 6/26/20 11:34 PM, Chun-Yu Shei wrote: |
3 |
>> Hi, |
4 |
>> |
5 |
>> I was recently interested in whether portage could be speed up, since |
6 |
>> dependency resolution can sometimes take a while on slower machines. |
7 |
>> After generating some flame graphs with cProfile and vmprof, I found |
8 |
>3 |
9 |
>> functions which seem to be called extremely frequently with the same |
10 |
>> arguments: catpkgsplit, use_reduce, and match_from_list. In the |
11 |
>first |
12 |
>> two cases, it was simple to cache the results in dicts, while |
13 |
>> match_from_list was a bit trickier, since it seems to be a |
14 |
>requirement |
15 |
>> that it return actual entries from the input "candidate_list". I |
16 |
>also |
17 |
>> ran into some test failures if I did the caching after the |
18 |
>> mydep.unevaluated_atom.use and mydep.repo checks towards the end of |
19 |
>the |
20 |
>> function, so the caching is only done up to just before that point. |
21 |
>> |
22 |
>> The catpkgsplit change seems to definitely be safe, and I'm pretty |
23 |
>sure |
24 |
>> the use_reduce one is too, since anything that could possibly change |
25 |
>the |
26 |
>> result is hashed. I'm a bit less certain about the match_from_list |
27 |
>one, |
28 |
>> although all tests are passing. |
29 |
>> |
30 |
>> With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world" |
31 |
>> speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup. |
32 |
>"emerge |
33 |
>> -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 sec |
34 |
>> (2.5% improvement). Since the upgrade case is far more common, this |
35 |
>> would really help in daily use, and it shaves about 30 seconds off |
36 |
>> the time you have to wait to get to the [Yes/No] prompt (from ~90s to |
37 |
>> 60s) on my old Sandy Bridge laptop when performing normal upgrades. |
38 |
>> |
39 |
>> Hopefully, at least some of these patches can be incorporated, and |
40 |
>please |
41 |
>> let me know if any changes are necessary. |
42 |
>> |
43 |
>> Thanks, |
44 |
>> Chun-Yu |
45 |
> |
46 |
>Using global variables for caches like these causes a form of memory |
47 |
>leak for use cases involving long-running processes that need to work |
48 |
>with many different repositories (and perhaps multiple versions of |
49 |
>those |
50 |
>repositories). |
51 |
> |
52 |
>There are at least a couple of different strategies that we can use to |
53 |
>avoid this form of memory leak: |
54 |
> |
55 |
>1) Limit the scope of the caches so that they have some sort of garbage |
56 |
>collection life cycle. For example, it would be natural for the |
57 |
>depgraph |
58 |
>class to have a local cache of use_reduce results, so that the cache |
59 |
>can |
60 |
>be garbage collected along with the depgraph. |
61 |
> |
62 |
>2) Eliminate redundant calls. For example, redundant calls to |
63 |
>catpkgslit |
64 |
>can be avoided by constructing more _pkg_str instances, since |
65 |
>catpkgsplit is able to return early when its argument happens to be a |
66 |
>_pkg_str instance. |
67 |
|
68 |
I think the weak stuff from the standard library might also be helpful. |
69 |
|
70 |
-- |
71 |
Best regards, |
72 |
Michał Górny |