1 |
On 26/08/2021 17:38, Marco Sirabella wrote: |
2 |
> - |
3 |
> |
4 |
> Hi Ed, |
5 |
> |
6 |
> I’ve taken a dabble at trying to track down portage’s bottlenecks (and have stopped for the time |
7 |
> being at solving them :/ ) |
8 |
> |
9 |
> Can anyone give me a leg up on how I could benchmark this further and look for the hotspot? |
10 |
> Perhaps someone understand the architecture of this point more intimately and could point at |
11 |
> whether there are opportunities to do some of the processing on mass, rather than per file? |
12 |
> |
13 |
> From my notes at the timem, it looks like yappi <https://pypi.org/project/yappi/> worked a bit |
14 |
> better than python’s built in cProfile for me because it properly dove into async calls. I used |
15 |
> snakeviz <https://jiffyclub.github.io/snakeviz/> for visualizing the profile results. |
16 |
> |
17 |
> I was taking a look at depclean, but I found similarly that a lot of duplicate process was being |
18 |
> done due to encapsulated abstractions not being able to communicate that the same thing was being |
19 |
> done multiple times eg removing each package processes a massive json structure for each package |
20 |
> removed, although I opted to work on the more-understandable unicode conversions. |
21 |
> |
22 |
> My stalled progress can be found here: #700 <https://github.com/gentoo/portage/pull/700>. Lost the |
23 |
> drive to continue for now unfortunately :< |
24 |
> |
25 |
> Good luck! Looking forward to your optimizations |
26 |
> |
27 |
> – Marco Sirabella |
28 |
> |
29 |
|
30 |
Hi All, thanks for the replies. Wow, Marco, that patch touches a lot of stuff...! |
31 |
|
32 |
OK, I will start by trying to get the profilers going and work from there... |
33 |
|
34 |
(Alec, to avoid replying separately: Thanks for your notes. Yes, I am not clear which of the |
35 |
install/merge phases specifically is the culprit, but it feels like something in that area is |
36 |
"slow", especially when run under qemu user mode. I think using qmerge won't work for my build |
37 |
script, but great idea to use it for benchmarking to narrow things down - thanks) |
38 |
|
39 |
Thanks all |
40 |
|
41 |
Ed W |