Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

On Wed, Sep 27, 2006 at 02:24:41AM -0400, Mike Frysinger wrote:
> On Monday 25 September 2006 14:16, Brian Harring wrote:
> > Bad soname handling is just *part* of what BINCOMPAT could do; it's
> > not the sole reason for it's existance, as such it's not quite right
> > dismissing it just because it addresses a rarity the NEEDED approach
> > doesn't. :)
>
> i dismiss it as being a real advantage and/or argument for taking BINCOMPAT
> over an automated NEEDED scan
>
> as i said, if you have changed ABI without an ABI bump, then the upstream
> package maintainer is screwing everyone who uses the package, not just
> Gentoo ... so perhaps we should be talking to them to get the situation
> resolved for all consumers, not just Gentoo

Happens however; afaik, we also weren't monkey patching ssl in the 
past to preserve the abi either so it still is valid (although rare if 
upstream is behaving) scenario.


> > *IF* we actually had that in place it would enable detecting and
> > rebuilding c++ code whenever gcc pulls its next c++ abi change with
> > appropriate deps in place (iow, kill off the implicit system deps).
>
> scanelf already does this ... the only time you need to rebuild is when the
> ABI breaks ... aka libstdc++.so.5 -> libstdc++.so.6
>
> > If that were the case, why do we have libraries listed in DEPEND then?
>
> because i have a short memory
>
> > Bleh, this is getting back to exactly my point that it's unbounded
> > resolution.  To support this, every step of execution would require
> > scanning for dangling nodes to punt; aside from invalidating -p's
> > output it *still* is a collision-protect hit.
>
> when the package upgrade detects an ABI change you recalculate the packages
> that need to be rebuilt ...

Reiterating, this screws over any form of up front calculation; 
dialup users/per minute connections can't rely on emerge -f to 
actually fetch all involved sources, -p results ain't guranteed 
valid, parallelization must now lock at each threads merge on the off 
chance a recalc is forced.

This is a rather huge step *backwards*; it actually puts us at a 
similar level to autopackages 'resolver'.  The approach would have to 
solve it perfectly imo for it to even be realistically considered.

For your proposal to fly, it must address this somehow imo which 
isn't possible from where I'm sitting.


> dangling nodes get recorded in the new package and considering these old files
> are not detrimental to the health of the system, you can do such a cleanup
> once at the end of the emerge

It's not orphaning files, but your scheme doesn't work for any form of 
interruption; failed builds, user intervention, etc, they all can 
leave the lib stuck in the new contents.

Dealing with that possibility means the manager has to maintain on 
disk a list of the refcount of each dangling libs to decrement, 
unmerge has to modify said list, etc.

Further nastyness in short.


> > It also involves changing vdb nodes from "installed and usable" to
> > "installed/usable" or "lingering"
>
> no ... the old versions get merged into the new one as their existence is
> purely hidden

How exactly are they going to be hidden?  A new file?  If so, 
backwards compatibility for vdb for transitioning in such a support 
has to be addressed.


> > Tracking BINCOMPAT should *not* be that hard.
>
> it's one more thing to keep track of and considering all of the possibilities
> i outlined, a single maintainer can easily lose his sanity ... or you force
> wasted rebuilds on people when it's only needed in some circumstances

How exactly is this forcing wasted rebuilds?  You're stating that 
maintainers are going to bump it willy nilly.  As I said, it is a key 
that would be bumped *as needed*, and would only affected pkgs that 
specified that node as a binding dep (specially marked atom).

Seriously, maintainers ought to know *now* when they're forcing a 
revdep on folks systems, I'm not seeing the massive overhead from 
pushing that info into a var, nor am I seeing mass forced useless 
rebuilds.

What I *am* seeing is one cluster fuck done to resolution and some 
nasty side affects you're glossing over.  Invalidating -p's output 
opens up some nasty issues.

To be fair, one claimed con could be that it is one more knob 
maintainers have to fool with, although counter is that they should 
already know about the issue (see dev quiz about "what should be done 
when bumping a lib" for an example).


> > Hell, automate a tool to determine if it's a BINCOMPAT bump via NEEDED
> > data (folks should be compiling the pkg anyways), point is it's mainly
> > common sense for maintainenance of it.
>
> and when are you going to run that tool ?  when you bump the package ?

One would hope that when bumping a package, the dev actually attempts 
to *compile* the damn thing.  Theres your opportunity to run a check, 
hell integrate it as a feature into portage that does the check for 
soname change.  Gurantee that mod is far less intrusive, and far less 
error prone then what you're proposing.


> so now
> the maintainer has to test it on all the KEYWORDed arches with all the USE
> combos, or de-KEYWORD it and make the arch maintainers test it ?

Realistically, just the same as the NEEDED solution has holes, the 
maintaining dev can screw up and miss a BINCOMPAT bump.  Difference is 
that they can do something for BINCOMPAT; hell, QA checks can be 
automated to catch missing BINCOMPAT bumps.  

Meanwhile, use flag affects: use deps reduce this heavily (toggle a 
flag off, graph is incomplete and the resolver 'fixes' it), so that's 
actually solvable *without* bincompat.

KEYWORDed arches bit, bit unlikely that the underlying arch is 
actually going to screw with the soname, thus I'd want actual examples 
backing it up.

Besides, again, for keywording, the dev *should* be compiling it, so 
there is a way to catch it :P.  A revbump isn't going to break things 
unless the dev is introducing something intrusive, minor version bumps 
(1.1.0 to 1.1.1) shouldn't be again, introducing anything intrusive; 
regardless, dev should know the high level affects of their change

Inducing a revdep-rebuild is definitely one of those high level 
changes they should be aware of *prior* to the bump.


> or dig
> through the source code line by line and hope to catch all such cases by
> hand/eye ?

Bit of FUD here, although I spose I'll just point out that if so's 
change as massively as you're implying above, the affects on -p are 
that much worse.


> > Yes, if the solution can be automated without flinging poo into the
> > code, I'm for it; that said I know what the implementation is going to
> > have to look like, and it's *nasty*.
>
> i dont think either of our solutions are satisfactory for all presented cases;
> a hybrid model would be needed

bind the checks in as a QA measure, enabled via FEATURES=stricter, 
used to maintain a metadata var.

Keep in mind I'm not claiming that implementing the binding deps 
modification is all that simple for portage; what I'm pushing here is 
a general solution that can be abused for more then just soname, 
*while* not cluster fucking the resolver execution to hell and 
preserving bounded resolution.

Finally, anyone got a general list of revdep events for the tree?  
Literally, pkg x version y forced a rebuild.  revdep is annoying, but 
stats would be useful to actually evaluate the seperate proposals; 
related, getting stats for how often the various updaters were 
required to be ran for a particular pkgs upgrade would be useful in 
evaluating that particular gap NEEDED has.

~harring