Gentoo Archives: gentoo-portage-dev

From: Ed W <lists@××××××××××.com>
To: gentoo-portage-dev@l.g.o
Subject: Re: [gentoo-portage-dev] Performance tuning and parallelisation
Date: Tue, 31 Aug 2021 08:00:18
Message-Id: a6c07377-3748-da61-cd82-7e27193934d1@wildgooses.com
In Reply to: Re: [gentoo-portage-dev] Performance tuning and parallelisation by Marco Sirabella
1 On 26/08/2021 17:38, Marco Sirabella wrote:
2 > -
3 >
4 > Hi Ed,
5 >
6 > I’ve taken a dabble at trying to track down portage’s bottlenecks (and have stopped for the time
7 > being at solving them :/ )
8 >
9 > Can anyone give me a leg up on how I could benchmark this further and look for the hotspot?
10 > Perhaps someone understand the architecture of this point more intimately and could point at
11 > whether there are opportunities to do some of the processing on mass, rather than per file?
12 >
13 > From my notes at the timem, it looks like yappi <https://pypi.org/project/yappi/> worked a bit
14 > better than python’s built in cProfile for me because it properly dove into async calls. I used
15 > snakeviz <https://jiffyclub.github.io/snakeviz/> for visualizing the profile results.
16 >
17 > I was taking a look at depclean, but I found similarly that a lot of duplicate process was being
18 > done due to encapsulated abstractions not being able to communicate that the same thing was being
19 > done multiple times eg removing each package processes a massive json structure for each package
20 > removed, although I opted to work on the more-understandable unicode conversions.
21 >
22 > My stalled progress can be found here: #700 <https://github.com/gentoo/portage/pull/700>. Lost the
23 > drive to continue for now unfortunately :<
24 >
25 > Good luck! Looking forward to your optimizations
26 >
27 > – Marco Sirabella
28 >
29
30 Hi All, thanks for the replies. Wow, Marco, that patch touches a lot of stuff...!
31
32 OK, I will start by trying to get the profilers going and work from there...
33
34 (Alec, to avoid replying separately: Thanks for your notes. Yes, I am not clear which of the
35 install/merge phases specifically is the culprit, but it feels like something in that area is
36 "slow", especially when run under qemu user mode. I think using qmerge won't work for my build
37 script, but great idea to use it for benchmarking to narrow things down - thanks)
38
39 Thanks all
40
41 Ed W