Re: [gentoo-dev] Portage dependency solving algorithm

[Andrew Savchenko]

Hello,

On Mon, 17 Nov 2014 21:55:48 -0800 Zac Medico wrote:
> On 11/17/2014 09:47 PM, Andrew Savchenko wrote:
> > I use 2.2.14 on both hosts (and usually latest ~x86 portage is
> > there). I thought that running fixpackages should be enough to run
> > emerge with --dynamic-deps=n.
>
> It depends on how badly the installed deps have diverged from the
> corresponding ebuilds in the tree.

I tried fixpackages. It fixed some problems and looks like
dependencies resolution became faster. But not all problems are
fixed and I can't use --dynamic-deps n on both systems for now;
and emerge @changed-deps fails due to numerous conflicts, blocks,
unsatisfied deps (this is not surprising, since it doesn't try to
update all packages in tree).

By the way, is there any way to unroll conflict lists in portage
output? I mean if I have following:

  (dev-lang/ghc-7.6.3-r1:0/7.6.3::gentoo, installed) pulled in by
    >=dev-lang/ghc-6.8.2:0/7.6.3= required by (dev-haskell/random-1.0.1.1-r1:0/1.0.1.1::gentoo, installed)
			^^^^^^^^^
    (and 68 more with the same problem)

How can I see all list of these 68 packages? Sometimes this feature is
really desired, e.g. if I don't want to update all @world but need to
apply GLSA fix which leads to similar conflicts. I can't find any
switch in emerge manual.

> > When I'll manage to run emerge -DNupv @world without errors, I'll
> > send you stats for both runs with and without dynamic deps.
>
> Great, hopefully that will reveal some more good things to optimize.
>
> > By the way, do you need pstats files (e.g. for some extra data) or
> > pdf graphs are sufficient?
>
> The pdf graphs are typically enough for me, since they highlight the hot
> spots really well. I did not even bother with your pstats files.

OK. I managed to run emerge -DNupv @world on desktop without
conflicts. What was done:
1) fixpacgkages run
2) portage is updated to use your patch from bug 529660

At this point performance boost was really great: from ~35
minutes to ~19-20 minutes.

Afterward I tried emerge -DNupv @world with different python
versions:
     (2.7)  (~)2.7.8
     (3.3)  3.3.5-r1
     (3.4)  (~)3.4.2

Results are interesting (confidence level for error is 0.95, time
real value was used for calculations):
3.3 is 3% ± 5% faster than 2.7
3.4 is 20% ± 5% faster than 3.3
And with python:3.4 and steps above it takes now 15.5 minutes
instead of 35. Nice result :)

So there is no evidence that portage on 3.3 is faster than on 2.7,
but 3.4 is faster than 3.3 with very good confidence. Of course
this data is biased by -m cProfile overhead, but bias should
similar for each version. Just checked time to run command for
python:3.4 without profiling: it takes 11.5 minutes!

You may find generated pdf graphs together with system information
for each host here:
ftp://brunestud.info/gentoo/portage-v2.tar.xz

As for hitomi box, it is both slower and have much older packages,
so I'm still struggling to fix conflicts and other issues. Results
will be available later.

P.S. Note for those who would like to use gpro2dot: it should be
run with the same python interpreter active as was used during
pstats data collection, otherwise it will fail to process data.
I spent some time while figuring this out.

Best regards,
Andrew Savchenko