Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

On Sat, Sep 23, 2006 at 11:31:12PM -0400, Mike Frysinger wrote:
> On Saturday 23 September 2006 10:30, Brian Harring wrote:
> > dlopen?
>
> we already said that this will need a new depend variable
>
> > How does this fix openssl horkage? (bad soname handling)
>
> it wont, but such things are broken regardless outside of Gentoo ... and
> trying to accommodate something that happens every three blue moons at the
> cost of developer time is not worth it

Bad soname handling is just *part* of what BINCOMPAT could do; it's 
not the sole reason for it's existance, as such it's not quite right 
dismissing it just because it addresses a rarity the NEEDED approach 
doesn't. :)

> > Also... what do we do for python/perl (*-updater scripts in general)
> > where a change in a pkg state means we have to rebuild the revdeps?
>
> my solution does not address this, but what you're proposing would over
> address this ... how do you know when a package needs to be rebuilt (a perl
> module) or rebuilding is a waste of time (a package installs perl scripts
> that execute `perl` and nothing else)

Original email stated that 'binding deps' would be required for that, 
marking deps in some way such that it indicates they're sensitive to 
changes in BINCOMPAT of the match.

*IF* we actually had that in place it would enable detecting and 
rebuilding c++ code whenever gcc pulls its next c++ abi change with 
appropriate deps in place (iow, kill off the implicit system deps).

Back to your example, if it's just a caller of it, it's not binding; 
now if it were a perl module that sticks its modules into the perl 
installation, yes, binding (it needs to rebuild to merge to the new 
location).


> > What you're suggesting works for strictly linkage; still think this
> > shouuld work for the general problem rather then just one subset.
>
> yes, i am addressing what i see to be the most critical issue and the easiest
> to break a user's system
>
> > Clarifying my statement; we don't break our DEPEND down into "this is
> > what is executed in building the package" (toolchain),
>
> DEPEND
>
> > vs "this is the
> > crap the binaries we build against need avail to be linked against",
> > literally what winds up as -l args.
>
> RDEPEND

If that were the case, why do we have libraries listed in DEPEND then?

DEPEND is (and always has been) "this is the crap I need merged to be 
able to build an install image of myself", RDEPEND is (and always has 
been) "this is the crap I need on the fs to actually run my 
binaries/libs" and PDEPEND is around to cover up portages resolver, 
but moreso the trees mostly whacked deps.

This is why eradicator, solar, and you were discussing splitting link 
depends out of DEPEND for saner CHOST/CTARGET support around a year 
back also.

Semantics at this point, but RDEPEND does *not* need to be merged to 
build a pkg, only DEPEND; it's never been any other way.


> > If punting the old lib (as I assumed), means we would potentially be
> > making certain DEPEND atoms unusable if they're required in an
> > execution context (rather then just winding up as a -l arg).
>
> no ... lemme give a perfect example.
>
> user has openssl-0.9.7j installed and they upgrade to openssl-0.9.8c ... the
> old version provided SONAME files libcrypto.so.0.9.7 and libssl.so.0.9.7
> while the new one provides SONAME files libcrypto.so.0.9.8 and
> libssl.so.0.9.8
>
> everything from openssl-0.9.7j is unmerged except for the two files
> libcrypto.so.0.9.7 and libssl.so.0.9.7.  openssl-0.9.8c, being a different
> ABI, does not provide these files thus there is no clash.
>
> portage keeps track of libcrypto.so.0.9.7 and libssl.so.0.9.7 and once no more
> packages have NEEDED entries for these, will silently clean them out
>
> openssl is odd in that it encodes .x.y.z version into the ABI ... if we use
> the more common example with say expat, older versions install
> libexpat.so.0.1, libexpat.so.0.2, libexpat.so.0.3, etc... but the ABI SONAME
> is still just libexpat.so.0.  when the next major version of expat comes out,
> the SONAME is bumped to libexpat.so.1 and portage needs to keep around the
> last libexpat.so.0
>
> > In that case, wouldn't mind a response to the "what about ctrl+c
> > during the run?"  The potential for orphaning there sucks; recording
> > the old library in the new version sucks also since it complicates the
> > merge process, that lib *must* be removed else it's a potential
> > collision-protect minefield.
>
> - portage merges new version 2 to $ROOT
> - system now has version 1 and version 2 in $ROOT
> - portage starts to clean out version 1 from $ROOT
> - do not fully trim version 1's vdb until version 2 has been updated with the
> new information
>
> so ctrl+c at any point and so what ?  you dont remove old files until new
> files are fully placed and you can resume at any point

Bleh, this is getting back to exactly my point that it's unbounded 
resolution.  To support this, every step of execution would require 
scanning for dangling nodes to punt; aside from invalidating -p's 
output it *still* is a collision-protect hit.

It also involves changing vdb nodes from "installed and usable" to 
"installed/usable" or "lingering", which makes things messier for
1) resolver
2) for --info
3) for has_version
4) for built_with_use
5) glsa scanning

There are more, but the point is that there isn't any concept of "half 
merged" packages for portage; introducing it gums things up pretty 
heavily, beyond the support NEEDED requires to actually mangle the 
half merged nodes.


> > Finally, even if the lib is temporarily left behind, this solution
> > doesn't gurantee the library actually would *work* still- it only can
> > work if the lib is standalone from the rest of the pkg and doesn't
> > rely on any external data from the pkg.
>
> we're talking about preserving the system long enough to rebuild things; we're
> not talking about keeping the system forever in a sane state.
>
> i would guess that this corner case is not the norm and thus we can ignore it
> as the situation is still a lot better than what we have now:
>  $ foo
>  foo: error while loading shared libraries: libbar.so.1: cannot open shared
> object file: No such file or directory
>
> awesome

So... NEEDED doesn't support bad soname handling, doesn't have a graph 
based gurantee that the remaining bits actually would work (bits 
above), and requires making merging *massively* slower (a NEEDED scan 
per vdb change effectively), finally emerge -p results no longer being 
accurate to what will be undertaken.

Ways to speed it up, but frankly NEEDED just plain sucks.  Its the 
route towards openembeddeds resolution, rather then up front 
calculation portage/rpm resolvers/apt all do.


> > Basically trying to point out that what you're proposing only works in
> > a subset of the cases revdep must deal with, and that revdep itself
> > doesn't deal with *all* situations as is; hence BINCOMPAT as a way to
> > try and shift it under maintainers control.
>
> revdep-rebuild doesnt take into consideration any of the issues you raised
>
> and forcing maintainers to always track BINCOMPAT is unwieldy ... a single
> package provides multiple SONAMEs ?  bitrot ?  SONAME is dynamic based upon
> architecture or USE flags ?

Tracking BINCOMPAT should *not* be that hard.  You hit a major 
version?  Bump the bincompat.  Not that fricking hard really; if 
dealing in a package that is pissy about it's so handling, well, you 
ought to know your inducing wide spread breakage for a rename already- 
so again, not that hard.

The only case that is slightly funky is if the so doesn't change name, 
but changes abi- BINCOMPAT at least enables notifying the manager that 
a rebuild is needed, NEEDED based approach is incapable of it.

Hell, automate a tool to determine if it's a BINCOMPAT bump via NEEDED 
data (folks should be compiling the pkg anyways), point is it's mainly 
common sense for maintainenance of it.


The NEEDED approach doesn't really fix much frankly; from a complexity 
standpoint, leaving it in revdep-rebuild is *far* better then trying 
to integrate the crap you're requesting, both from an implementation 
and from a long standing maintainence view (unbounded resolution is 
*really* not the direction you want to go).

The solution (imo) should be one that enables fixing the *class* of 
problem without making matters worse.

Yes, if the solution can be automated without flinging poo into the 
code, I'm for it; that said I know what the implementation is going to 
have to look like, and it's *nasty*.

~harring