Re: [gentoo-dev] Thoughts about broken package handling

[Kent Fredric]

On 26 June 2011 14:59, Stuart Longland <redhatter@g.o> wrote:
> Hi all,
>
> I've been busy for the past month or two, busy updating some of my
> systems.  In particular, the Yeeloong I have, hasn't seen attention in a
> very long time.  Soon as I update one part however, I find some swath of
> packages break because of a soname change, anything Python-related stops
> working because of a move from Python 2.6 to 2.7, or Perl gets updated.
>

I have a system I use/developed which tries to improve consistency,
but it greatly increases the amount of compile/test runs that get done
in the process, but for my cases that's ok because I *want* things to
be rebuilt needlessly "just to make sure they can still be built".

<Massive rant follows>

I make some assumptions:

Take a timestamp.

Record all packages that need to be updated or reinstalled due to USE changes.

Assume, that there is a chance that any direct dependent of those
packages may become broken by this update, either at install time, or
during runtime ( ie: .so breakages, ABI breakages, $LANGUAGE breakages
etc )

Consider that all these packages were installed before that timestamp.

You can then mostly assume all packages installed after that timestamp
are built with the consideration of the changes that have occurred in
their dependencies.

For the most part, this principle appears to cover a very large range
of scenarios, and is somewhat like a proactive "revdep-rebuild" that
assumes that every new install/upgrade breaks everything that uses it.

To this end, I have conjured a very Naïve toolset which does what I want it to.

https://github.com/kentfredric/treebuilder

Note a lot of the paths are hardcoded in the source, and its only
really on Github for educational value.

Workflow proceeds as follows:  ( nb: I /do/ use paludis, so I'll be
using its terms for the sake of accuracy )

./mk_timestamp.sh  # this sets our timestamp values.

echo >| rebuild.txt   # start over the list of things that need rebuilding

Then I sync portage and overlays. ( I have a custom script that calls
cave sync as well as doing a few other tasks )

{

then I do a "deep update of system including new uses"  but not execute it.

cave resolve -c system

And I record each updated/reinstalled package line-by-line in rebuild.txt

Then perform it:

cave resolve -c system --continue-on-failure if-independent -x

}

and then repeat that with 'world'.

Then I run ./sync.sh , which works out all the "Still old" packages,
computes some simple regexps' and emits 'rebuilds.out', which is a
list of packages that "Might be broken"  which I'll reinstall "just in
case".

VERY Often this is completely needless, ie: -r bumps to kdelibs
triggers me rebuilding everything in KDE, -r bumps to perl causes me
rebuilding every perl module in existence, and everything that uses
perl ( including everything in KDE incidentally , as well as GHC and a
few other very large nasties ).

Once this list is complete, there are 2 approaches I generally take,

1. If the list is small enough, I'll pass the whole thing to cave/paludis.

  cave resolve -c -1 $( cat /root/rebuilder/rebuilds.out )

   and note record any significant changes ( ie: new uses/updates of
dependendents for things that are "orphans" for whatever reason )

   and then

  cave resolve -c -1 $( cat /root/rebuilder/rebuilds.out ) -x
--continue-on-failure if-independent

or

2. If this list looks a bit large, I'll pass the things to reinstall
randomly in groups

   dd if=/dev/urandom of=/tmp/rand count=1024 bs=1024  # generate
random file for 'shuf' to produce random but repeatable sort.

  cave resolve -c -1 $( shuf --random-source=/tmp/rand -n 200
/root/rebuilder/rebuilds.out )

   again, noting updates/etc

   cave resolve -c -1 $( shuf --random-source=/tmp/rand -n 200
/root/rebuilder/rebuilds.out )  -x --continue-on-failure
if-independent


After each build run, sync.sh is re-run, updating the list of things
which this code still considers "broken", and then I continue the
build/build/build/sync pattern until the build contains all items in
rebuilds.out, and they are all either failing or skipped.

At this point, all the files still listed in rebuilds.out are deemed
"somewhat broken".  This list is then concatentated with "brokens.out"
and the process is repeated until all results fail/skip.

then brokens.out and rebuilds.out are concatentated together and
replace broken.txt , which is a list of "things to check later".

at this point I consider that "merging things are as good as its going
to get", and the entire process starts over, update the timestamp,
sync portage.

Over time, broken.txt adapts itself, growing and shrinking as new
things start being broken due to dependencies, or being resolved due
to fixes entering portage.

Long story short, all of the above is mostly insanity, but its
reasonably successfull. And after all, I am an insane kinda guy =)

--
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz