[gentoo-amd64] Re: kde-base components blocking each other

[Duncan]

Martin Herrman <martin@×××××××.nl> posted
40bb8d3b0906070049g26add8e4s78bfa9da8285beda@××××××××××.com, excerpted
below, on  Sun, 07 Jun 2009 09:49:28 +0200:

>> Note that newer versions of portage have /automatic/ blocker resolution
>> in many temporary cases, so depending on how old your portage is (of
>> course I'm on ~amd64 and have had 3.5.10 for months now, and don't know
>> if the feature is in stable portage yet), if at the --ask prompt, you
>> tell it to go ahead, it may well take care of everything for you
>> without you having to do anything else. =:^)
>
>
> in which version was this feature added? I just upgraded portage to
> 2.1.6.13 and could upgrade to 2.2.x, but that one is hard masked:
>
> http://www.gentoo-portage.com/sys-apps/portage

What happened there is that while 2.1.4 was current stable, 2.2 as the 
next presumed stable series, with several new features, was made ~arch.

However, some of those features matured faster than others.  Among them, 
parallel merges (see below) and automatic blocker resolution (again, see 
below).  Others such as sets (you'll see my mention of @system and @world 
below, tho I don't spend time on it or explain, see Zac's blog linked 
below for more), and FEATURES=preserved-libs (nice idea in theory but was 
breaking stuff due to originally poor implementation so I turned it off, 
I'm not sure the current status) didn't mature quite as fast.

So Zac and the other portage folks decided to release an in-between 
series starting with (IIRC) 2.1.5, that contained the more stable new 
features from 2.2 (parallel merges and auto-blocker-resolution), while 
omitting the ones that needed more work (sets, preserved-libs).  The idea 
was to have the new features that could go stable much faster.  However, 
that meant that they needed testers for the 2.1.5 series, that they 
didn't have, since the usual testing group, ~arch users, were alreadly on 
2.2.

So they hard-masked 2.2 so most ~arch users would fall back to the new 
2.1.5 series, thus testing it well enough so it could be stabilized.  
That's the status we have today.  2.2 is ~arch ready and in fact was 
~arch for a short period, until they decided they needed testers for the 
new middle ground 2.1.5 series.

But, there was a problem.  With 2.2 going ~arch, KDE moved KDE 4.2 (or 
was it still 4.1 at the time?), to that point either out of tree or hard-
masked, to ~arch as well.  The problem is that the official KDE-4 
migration guide referred to and continues to refer to sets, a 2.2-only 
feature at this point.  So anyone who was using or testing ~arch KDE 4.2 
and had followed the official migration guide using sets, had to either 
unmask portage-2.2 when it went back hard-masked, or fiddle around trying 
to figure out how to convert their previous set-merged kde-4.2 into a non-
set config.

The Gentoo/KDE-4 Migration Guide
http://www.gentoo.org/proj/en/desktop/kde/kde4-guide.xml

FWIW, while I'm still using KDE 3.5.10 as my "production" desktop, I've 
been testing 4.x since before the original release. (FWIW, KDE-4 still 
has problems in 4.2.x compared to later 3.5.x versions, at least for 
those with older video cards like the Radeon 9200 I'm still running, but 
it's /slowly/ getting there, and the later 4.2.x versions should be 
decently usable for many with newer video -- or smaller displays, I'm 
running 1920x2400, dual-stacked 1920x1200, and the Radeon 9200's OpenGL 
maxes at 2048x2048, thus the issue here, as KDE-4 really needs OpenGL to 
be worth the upgrade from KDE-3.)  Thus, I had and have 4.2.x installed 
as well -- using the upgrade guide and sets -- and had to unmask 
portage-2.2 when it was masked to encourage ~arch users to test the new 
2.1.5 series.

> I will now just give it a try: start to unmerge the blocked packages and
> update my system. We'll see :-)

Back to the blocked packages thing, versions, etc...

Note that with --ask (or --pretend), portage will still display the 
blocks.  It'll just manage them automatically if you say "y" at the --ask 
prompt (or redo the emerge without the --pretend).  That way, users can 
see any potential for temporary "dependency gaps" as I mentioned and can 
use their judgment and not try to do them right before that ever so 
important presentation or whatever.

I was unsure of the precise version, but I did a quick bug search and 
found a comment from Zac Medico (portage maintainer and release manager) 
indicating that for the 2.1 series, it was added in 2.1.5.

http://bugs.gentoo.org/show_bug.cgi?id=224487#c1

As he says there:

>> With automatic blocker resolution like this, it should
>> be relatively rare for a user to encounter a blocker
>> that requires any manual intervention.

That comment references the blog entry where he explains the idea, here 
(link wrap warning, but you can follow the link from the comment linked 
above):

http://blogs.gentoo.org/zmedico/2008/05/09/
blocking_package_file_collisions

Zac's answers to some of the comments are interesting/informative/helpful 
too.  Note that portage's blocker resolution has gotten slightly more 
complex since then.  In particular, in ebuilds where !<pkg-atom> 
originally indicated a blocker, that's still used if it's a normal 
blocker that it's OK for portage to try to resolve automatically, while 
!!<pkg-atom> is now used for complex blockers where portage must not try 
to resolve it automatically because something's likely to go wrong.  
Also, as noted in the blog, portage is more cautious about system 
packages and its own dependencies, where doing the wrong thing could 
leave the system temporarily unbootable or could kill portage itself, 
thus making recovery much more problematic.

Also likely of interest if you've not paid too much attention recently, 
and have a multi-cpu/multi-core system with a decent amount (say a gig a 
core) of memory is the multiple package merges in parallel stuff, see the 
"next" link at the blog.  Matter of fact, browsing around the whole blog 
and checking back for newer entries are both likely to be quite 
informative/helpful, as well.

http://blogs.gentoo.org/zmedico/2008/07/23/portage_parallel_builds

I've been using the parallel jobs feature for some time now, and have 
with experience developed some hints.  First, as I said, it doesn't make 
a lot of sense to use it for single cpu/core systems, or for those 
sufficiently memory constrained (say < 1-gig/core) that resources are 
already stretched, and may in fact increase merge times due to increased 
swap usage or the like.

Second, --keep-going is quite nice. =:^)  That way, when something fails, 
portage continues other presently going merge jobs and then regenerates 
its dependency graph without the failed package and tries again.  Thus, 
it installs as much as it can before failing with what it couldn't (and 
anything depending on what failed), leaving far less to have to merge 
later, after the issue with the failed package is resolved.

As for --jobs and --load-average, here's what I found.  First, you 
probably want to make these somewhat lower than the parallel MAKEOPTS 
settings.  (This assumes that you DO use both --load-average here and
-l in MAKEOPTS, if you just use --jobs and -j, adjust accordingly.)   
Doing so means the system will first try to run more parallel jobs on 
already running merges if it can, before the load drops to the point that 
it starts on additional package merges as well.  This helps limit memory 
and other resource usage, particularly for those (like me) that have 
PORTAGE_TMPDIR pointing at a tmpfs to speed emerges and minimize disk 
access.

Second, because the unpack and configure steps are by nature not very 
parallelized, what tends to happen is that in parallel mode, at first, 
portage will start working on one package, decide the load average is 
still low enough to do another... and another... and another... before 
any of them actually get to the more parallelized compile phase.  Also, 
at times, there will be a bottleneck package which portage must wait on 
because all further packages depend on it and must therefore wait until 
it's installed before they can be merged.  These two factors taken 
together tend to mean that in practice, the emerges clump up, portage 
will start several at a time before the load average gets high enough 
that it doesn't start any more, then (even with a higher MAKEOPTS setting 
than portage setting) when they hit the actual compile phase, it'll cap 
at the MAKEOPTS limit for quite some time and not start anything else 
until all or all but one of the packages are finished, then it'll start 
several more... etc.

Third, and this is what took the most getting used to here, since several 
packages are merging in parallel, it can't show the whole output for each 
one as it used to.  Instead, it shows a summary that shows when it starts 
working on a package, when it starts installing a package, how many it 
has completed, and how many total.  It's actually pretty nice, but it did 
take some getting used to.  Meanwhile, it's still logging the actual 
merges in the background, and if there's an error, it spits that info out 
and tells you where to look for the log.  Also, if you want to track the 
progress of an individual package, you can use the tradition Unix tail -f 
command in another terminal or whatever.

What I've done here is set it up so that my @system and @world update 
helper scriptlets (example: eaw for emerge -aNuD @world) always use the 
parallel merge options while ordinary single-package emerge helper 
scriptlets (example: ea for emerge -a1) don't do the parallel thing.  I 
do however have a parallel variant scriptlet (eam, the m for multiplexed) 
that does.  That way, I can see the full output when I'm merging 
individual packages, but for big jobs such as @world updates or if I'm 
upgrading KDE, I use the parallel emerges functionality.  I thus do NOT 
have portage's --jobs, etc options set in EMERGE_DEFAULT_OPTS, because I 
set it on the emerge command line as fed to it by the appropriate 
scriptlet.

One caveat!  Some packages require a LOT of memory for one particular 
job.  KDE-3's kmail (for monolithic package users I believe it's in 
kdepim) is one example, using over a gig of memory for a single job at 
one point in its compile, at least here.  There are others.  Also, while 
portage honors its --load-average option, not all packages honor the -l 
in makeopts.  At least parts of qt-4 are this way.  That can be bad when 
it's only a single package emerging at once, but it's made much worse if 
you've not allowed for it setting the portage parallel merge options and 
portage has several merges going at once, at the same time.  I've tended 
to handle these as I find them by setting a lower MAKEOPTS=-j option 
(while eliminating the -l they don't honor anyway) for them in the 
appropriate /etc/portage/env/cat-egory/pkg file, thus overruling the 
standard/global make.conf setting.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman