1 |
Dnia June 28, 2020 3:42:33 AM UTC, Zac Medico <zmedico@g.o> napisał(a): |
2 |
>On 6/27/20 8:12 PM, Michał Górny wrote: |
3 |
>> Dnia June 28, 2020 3:00:00 AM UTC, Zac Medico <zmedico@g.o> |
4 |
>napisał(a): |
5 |
>>> On 6/26/20 11:34 PM, Chun-Yu Shei wrote: |
6 |
>>>> Hi, |
7 |
>>>> |
8 |
>>>> I was recently interested in whether portage could be speed up, |
9 |
>since |
10 |
>>>> dependency resolution can sometimes take a while on slower |
11 |
>machines. |
12 |
>>>> After generating some flame graphs with cProfile and vmprof, I |
13 |
>found |
14 |
>>> 3 |
15 |
>>>> functions which seem to be called extremely frequently with the |
16 |
>same |
17 |
>>>> arguments: catpkgsplit, use_reduce, and match_from_list. In the |
18 |
>>> first |
19 |
>>>> two cases, it was simple to cache the results in dicts, while |
20 |
>>>> match_from_list was a bit trickier, since it seems to be a |
21 |
>>> requirement |
22 |
>>>> that it return actual entries from the input "candidate_list". I |
23 |
>>> also |
24 |
>>>> ran into some test failures if I did the caching after the |
25 |
>>>> mydep.unevaluated_atom.use and mydep.repo checks towards the end of |
26 |
>>> the |
27 |
>>>> function, so the caching is only done up to just before that point. |
28 |
>>>> |
29 |
>>>> The catpkgsplit change seems to definitely be safe, and I'm pretty |
30 |
>>> sure |
31 |
>>>> the use_reduce one is too, since anything that could possibly |
32 |
>change |
33 |
>>> the |
34 |
>>>> result is hashed. I'm a bit less certain about the match_from_list |
35 |
>>> one, |
36 |
>>>> although all tests are passing. |
37 |
>>>> |
38 |
>>>> With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world" |
39 |
>>>> speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup. |
40 |
>>> "emerge |
41 |
>>>> -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 |
42 |
>sec |
43 |
>>>> (2.5% improvement). Since the upgrade case is far more common, |
44 |
>this |
45 |
>>>> would really help in daily use, and it shaves about 30 seconds off |
46 |
>>>> the time you have to wait to get to the [Yes/No] prompt (from ~90s |
47 |
>to |
48 |
>>>> 60s) on my old Sandy Bridge laptop when performing normal upgrades. |
49 |
>>>> |
50 |
>>>> Hopefully, at least some of these patches can be incorporated, and |
51 |
>>> please |
52 |
>>>> let me know if any changes are necessary. |
53 |
>>>> |
54 |
>>>> Thanks, |
55 |
>>>> Chun-Yu |
56 |
>>> |
57 |
>>> Using global variables for caches like these causes a form of memory |
58 |
>>> leak for use cases involving long-running processes that need to |
59 |
>work |
60 |
>>> with many different repositories (and perhaps multiple versions of |
61 |
>>> those |
62 |
>>> repositories). |
63 |
>>> |
64 |
>>> There are at least a couple of different strategies that we can use |
65 |
>to |
66 |
>>> avoid this form of memory leak: |
67 |
>>> |
68 |
>>> 1) Limit the scope of the caches so that they have some sort of |
69 |
>garbage |
70 |
>>> collection life cycle. For example, it would be natural for the |
71 |
>>> depgraph |
72 |
>>> class to have a local cache of use_reduce results, so that the cache |
73 |
>>> can |
74 |
>>> be garbage collected along with the depgraph. |
75 |
>>> |
76 |
>>> 2) Eliminate redundant calls. For example, redundant calls to |
77 |
>>> catpkgslit |
78 |
>>> can be avoided by constructing more _pkg_str instances, since |
79 |
>>> catpkgsplit is able to return early when its argument happens to be |
80 |
>a |
81 |
>>> _pkg_str instance. |
82 |
>> |
83 |
>> I think the weak stuff from the standard library might also be |
84 |
>helpful. |
85 |
>> |
86 |
>> -- |
87 |
>> Best regards, |
88 |
>> Michał Górny |
89 |
>> |
90 |
> |
91 |
>Hmm, maybe weak global caches are an option? |
92 |
|
93 |
It would probably be necessary to add hit/miss counter and compare results before and after. |
94 |
|
95 |
-- |
96 |
Best regards, |
97 |
Michał Górny |