Re: [gentoo-dev] new `usex` helper

[Brian Harring]

On Sun, Sep 18, 2011 at 10:16:46PM -0500, Donnie Berkholz wrote:
> On 04:22 Sun 18 Sep     , Brian Harring wrote:
> > On Sat, Sep 17, 2011 at 10:59:08PM -0500, Donnie Berkholz wrote:
> > > On 13:43 Fri 16 Sep     , Brian Harring wrote:
> > > > What I said from the getgo and you're missing is that pushing EAPI
> > > > implementation into the tree and ignoring EAPI, or having this notion
> > > > that every repository must automatically use gentoo-x86 (pushing the
> > > > format into the tree) is /wrong/;
> > >
> > > I'm not necessarily requiring that every repository must automatically
> > > use gentoo-x86 ??? just the ones that want to use features in an eclass
> > > distributed with gentoo-x86. It sounds to me like the logical
> > > consequence of what you're saying is that every useful function that's
> > > now or could eventually be in an eclass must instead be incorporated
> > > into EAPI. I guess I just don't see where you're drawing the line.
> >
> > Drawing it back to what spawned it; usex.  This isn't git.eclass, this
> > isn't svn.eclass, nor is it any of the other complex eclass
> > functionality.  It's a core function that benefits the rest and should
> > be in EAPI.
>
> OK, so the implication of what you're saying is that everything in
> eutils.eclass, base.eclass, toolchain-funcs.eclass, flag-o-matic.eclass,
> versionator.eclass, multilib.eclass, prefix.eclass, savedconfig.eclass,
> and maybe even linux-info.eclass and autotools{,-utils}.eclass should go
> into EAPI. All that stuff is pretty generally useful features.

Approximately... yes.  Some of that (base.eclass for example) is EAPI 
compatibility that can't be folded in (would require retroactively 
addding to an EAPI), others aren't yet stabilized (true multilib for 
example, seems to still be under discussion), etc.

You get the jist.

> > > What I'm suggesting is that we should add useful stuff to eclasses
> > > by default. If people want to use that stuff, they can stack on the
> > > gentoo-x86 repo and inherit the eclass. I don't know why EAPI needs
> > > to come into it at all.
> >
> > Stacking on gentoo-x86 means you get *everything* for that repository.
> > If all you want is a random function out of eutils, that's a *lot* of
> > uncontrolled change to have intermixed with your overlays eclasses
> > (even worse, if the eclasses truly overlay into gentoo-x86, that
> > introduces compatibility issues there too).
>
> Yeah, it seems like the ability to do partial stacking would be nice.
> Perhaps to do so, one wouldn't stack by default but would have a way to
> specify cross-repo dependencies (including eclasses) or file-specific
> UUIDs of some sort.

We can specify cross repository dependencies (this repo needs 
that repo) already via layout.conf; partial gets a bit messy, but may 
be useful at the user level.


> > > > aside from meaning that the format definition can now *vary*,
> > > > which is great for wasting dev time and screwing up compatibility,
> > > > it opens up tweaking/customizing that functionality- aka,
> > > > fragmentation/divergence.
> > >
> > > People doing that outside gentoo-x86 should do it the same way as
> > > ones within it, by bumping the eclass to a new unique name. Ideally
> > > one that's not just a numeric value so it won't conflict with ours,
> > > in the same way as EAPI naming.
> >
> > They should, but api compatibility of an eclass *can* be fluid- in the
> > past it was a locked API purely because of portage environment saving
> > issues.  That's been resolved, there isn't any strict requirement that
> > the eclass maintain API compat now.  You're trying to rely on people
> > doing best practices- for format building blocks
> > (use_with/usex/unpack/etc), that's not sane.
>
> Which still pisses me off. It's like a shared library changing its API
> without bumping SONAME.

I think you're viewing this wrong; if we're talking about openssl 
breaking API/ABI w/out bumping soname, sure.  It's one component that 
is distributed alone, but consumed by others.  There is no way to 
atomically deploy the new API at the same time as all of the consumers 
being updated for it.

That is /not/ the case of gentoo-x86; eclasses for us are equivalent 
to bundled libs.  Overlay stacking just makes it possible for a third 
party component to reach in and use what is effectively an internal 
lib.

This is even more true for repositories other than gentoo-x86; 
gentoo-x86 is just in the weird position of being both public and 
private.


> That makes me wonder whether there's some way we
> could "enforce" it, at least on the level of repoman or test suites to
> examine whether the public interface changed, perhaps by generating a
> cache of the interfaces and comparing to them.

Not without some doc/annotation of each function.  The fact functions 
are basically jump targets, w/ the function doing shift's/$@ to get at 
it's args makes this very, very hard to introspect and validate.  

Tests are the realistic option, or parsing an annotation defining the 
functions prototype, caching that, and comparing it across versions.

That's realistically a seperate discussion- one snagging betelgeuse 
(and that libbash gSoC student) might result in some benefit.


> > I suspect an easier way to wrap this thread up is if you just state
> > what you want for backwards compatibility- in a seperate thread you
> > already proposed basically locking out systems older than 6 months,
> > and this discussion (moreso the "eapi slows our development" subtext)
> > is along the same lines.
>
> Actually Zac said that, and I was trying to clarify if that's really
> what he was saying. 9-12 months is more my feeling, as it gets extremely
> difficult to maintain Gentoo systems without guru-level knowledge around
> there. (Spoken as someone who still gets support emails from a couple of
> previous positions.)
>
> While I would like there to be a "proper" way to express repo-level
> dependencies, properly announce deprecation and updates (e.g. to bash 4)
> in advance as a feature integrated with the PM, and so on, I don't think
> we should block our ability to move forward on these things.

We're not really blocked, frankly.  For repository format changes, 
while we lack versioning... it can be done.  There just hasn't been 
enough of a reason to do it, *yet* (it's been on my todo for a while 
since there is some stuff I want that requires it).

For deploying EAPI, we're able to do it pretty quickly if we wanted 
to; where we're boned is in profiles, and that will /not/ change.  
There is no magic solution to it- from a data structure standpoint, 
either we have to duplicate chunks of the profiles for backwards 
compatibility, or we have to keep it very low in EAPI deps.

That's just the fact of life for profiles; it's a central data struct, 
as such it's fairly bound to LCD requirements.


> The
> situation is essentially the same as it's always been, but our old
> solution is no longer considered good enough and we don't have a new one
> in place, so we're just sitting here.

Not really; the old solution was "deploy, then have to wait 6 months 
for small stuff, year for big stuff".

For format changes, we can move *much* faster- 2 months basically 
(month of portage sitting in unstable, allowing unstable ebuilds to 
use the EAPI, then portage stabled in the second month effectively 
stabling the EAPI).

For the profiles, we can move forward at the same rate if desired in 
terms of using the new functionality, although for avoiding lock out 
(preserving the EAPI upgrade pathway) duplication may be required or a 
slower rate of EAPI consumption.

Those times are accurate- that's what we *could* do.  EAPI versions 
being cut slower is a whole other issue (council having to sign off, 
bribing portage to add stuff, etc), and that issue should be addressed 
directly- any compat mechanism will run into the same issue there, so 
the mechanism isn't the issue.


> > how derivatives should be handled,
>
> With white gloves. More seriously, people making derivatives should have
> maximal freedom to experiment, and I'm guessing most of them don't want
> to deal with modifying 3 different PMs written at least partially in 3
> different languages. They'd rather instantaneously support things in the
> same language they use for everything else.

Full experimentation has typically required modifying the PM to extend 
functionality- you're focusing just on bash bits.  Bash bits are easy- 
you can hack that at the eclass level w/out an EAPI half the time.  
Frankly, that's not the stuff people get hung up on.

Realistically, the interesting stuff that people want have been more 
than just bash.

Simple example, you can't do USE deps, slot deps, repository deps, 
iuse defaults, etc, w/out modifying the PM.  That's just the way it 
is, as such you choose your preferred PM, get the custom eapi going, 
if it's more than a one off prototype you bribe others to support it, 
they do.

I don't see that changing; while folks can complain about it, there 
are hard technical reasons this is how it is- those issues have to be 
solved (no amount of complaining can solve 'em) ;)

~harring