Re: [gentoo-user] Portage performance dropped considerably

[Tom Wijsman]

On Sun, 26 Jan 2014 16:35:43 +0200
Nikos Chantziaras <realnc@×××××.com> wrote:

> Anyone else noticed this yet? Some portage update seems to have made
> "emerge -uDN @world" perform about 10 times slower than before. It
> used to take seconds, now it takes about 4 minutes only to tell me
> that there's nothing to update. And it does that every time, even
> directly in succession and with the caches warm.
>
> Is it just me?

(TL;DR: It is a known problem and it will be improved.)

More people experience this, I've seen multiple reports in all the
communication channels I can think of so far; besides having been
bothered with it myself in the past, but now I'm fine with ~30 seconds.

=== What is going on and what is a cheap solution? ===

Often this is due to backtracking going mad; Portage keeps trying
again, and again, and ... until it has done it for a 10 extra times. And
if a single time perhaps takes like 30 seconds, doing it 10 times extra
gets you quickly to get to 6 minutes or so. Now imagine 30 times...

For this reason, I run --backtrack=0 myself; which finishes in half a
minute or so, but it needs some more experience to deal with the output
that comes after that as you'll need to fix up some things manually.
Not to forget --backtrack=0 has some side effects wrt package rebuilds;
I don't know the fine details, but was warned about this by others.

A solution in between would be to use --backtrack=0 and when that isn't
satisfied raise it a bit until satisfied; if you can get your
--backtrack=0 to work one or another time, you know you won't need
--backtrack=10 soon given that your Portage tree is pretty satisfied.
When you need to raise it, you will come around with --backtrack=5; and
in the case it doesn't you can raise it again, 3 minutes don't hurt.

So, it can be perceived as a trade-off between "I can quickly do this
myself" (given a good understanding of the output) and "Spent some time
doing it for me" (if you don't have the time to do it yourself).

=== Why is this becoming worse over time? ===

Simply put, it's a growth in complexity; we've moved to newer EAPIs
that bring newer features, we have packages improving and thus
providing more USE flags depending on more, we've got more overlays to
pull interesting packages from, as a consequence of the newer EAPIs
there are a lot of sub slot dependencies and the list goes on...

The more options Portage has to consider, the more time it will spend
on it; and if after going down a set of options Portage doesn't find a
satisfiable state, it has to backtrack with more and/or different
options to try again to find another satisfiable state. You can see
that the amount of options (= amount of overlays, packages, versions,
enabled USE flags, ...) takes a huge toll here (amongst other things).

To get an idea of what an @world emerge is doing by default, check:

https://gist.github.com/TomWij/65a61765ea03e6188a35

This is generated using the output additions from the attachment of:

http://article.gmane.org/gmane.linux.gentoo.user/272117

Those whom are interested can also find call tree graphs, although
these rather reveal ~2% improvements because huger improvements require
an actual understanding of how most code works together; they are here:

http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0.png
http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0-hot.png
http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-3.3-backtrack-0.png

(Generated with https://code.google.com/p/jrfonseca/wiki/Gprof2Dot)

From the output additions you can see that the slot operators (and thus
sub slots from the newer EAPIs) take up quite a few time. If you think
about what these slot operators are trying to solve, you'll remember
something along the lines of `revdep-rebuild` from a while ago. A lot
of time we've used to spend in that back in the days has moved towards
a cocktail of slot operators and preserved-rebuild.

So, part of the extra time we experience is just a move of time; but
that doesn't make it additional time (unless with lots of backtracking).

Tried to keep this layman explanation short; as I no longer intend to
make this more than a simple explanation, I used to share further
details in the past but I do no longer deem them as necessary today.

Why? Because Portage developer Sebastian Luther (few_) is working on
improving this; the results seem promising, so, let's await a Portage
release with that before we continue to look into this again.

=== What might be done to improve this? ===

Read this message (patch in second mail):

http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4231

The results look promising; it goes from 9:28 to 1:59, whereas
--backtrack=0 is 1:38. In other words, it gets quite close to the time
of --backtrack=0 without bringing its disadvantages; as it still does
the conflict resolving that backtrack does, but somewhat different.

(There are some other smaller patches that also spare out time; if you
are interested, you can find them by searching for him on the mailing
list. However, I think those other patches will be in a release soon.)

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@g.o
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D