Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies

[Brian Harring]

Pardon the belated response; responding to emails that are quick where 
possible, but lagging on -dev.  Missed this one however...

On Wed, Sep 19, 2012 at 09:16:02AM -0400, Ian Stakenvicius wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 19/09/12 09:09 AM, Michael Orlitzky wrote:
> > On 09/19/2012 06:59 AM, Duncan wrote:
> >> Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as
> >> excerpted:
> >>
> >>> On 16 September 2012 21:15, Brian Harring <ferringb@×××××.com>
> >>> wrote:
> >>
> >>>> So... basically, people are already doing this manually with
> >>>> their own intermediate vars.
> >>>
> >>> And this works fine, so it doesn't warrant a cosmetic change.
> >>
> >> @ferringb:
> >>
> >> yngwin has a point that I've not seen addressed.
> >>
> >> What /is/ wrong with the whole CDEPEND intermediate var idea?  It
> >> seems to work and /I/ don't know of any problems with it (and it
> >> would appear, neither does yngwin), yet you talk about it as if
> >> there's something wrong with it.
> >>
> >> And while we're at it, do DEPEND="$RDEPEND ..." style solutions
> >> have the same problems (or lack thereof)?
> >
> > The problem appears as we introduce more DEPEND variables (which is
> > what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
> > CDEPEND, and DDEPEND, and there's only some (i.e. not total)
> > sharing going on then the COMMON_DEPEND pattern starts to fall
> > apart. You potentially need,
> >
> > AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
> > ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
> > (COMMON_DEPEND)
> >
> > This obviously gets worse as more DEPEND vars are introduced.
> >
>
> Well not really, no -- the additional *DEPENDs that are being proposed
> (or at least mentioned) for new EAPI will either remove atoms from
> COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
> COMMON_DEPEND or other intermediate variable won't really be necessary
> for them.

It depends on the dep type actually, and how we're viewing those deps- 
if they must be complete or not.

Consider test depends for example.  Either we can specify it as "in 
addition to depend"- which sucks, because it's entirely possible for a 
DEPEND target to not be required for TDEPEND.

While that's a corner case, it's actually easy to address; just 
require TDEPEND to be comprehensive for the test phase.

Thus DEPEND and TDEPEND suddenly share a lot, and TDEPEND shares part 
of RDEPEND.

Moving on to a real world example... PDEPEND.  Currently a full graph 
resolution requires pulling RDEPEND and PDEPEND, collapsing them, and 
ensuring they're fullfilled for anything that is already installed 
(ie, PDEPEND is required after the transaction is completed).

In our current separated var setup, this as said, this requires the PM 
to track/handle it separately.  However, were PDEPEND to be 
complete/full- that would mean the PM could just render for dep:post 
and know "this is what is necessary outside of the transactional block 
of building/installing it".

So RDEPEND and PDEPEND actually share a *shitton*- they're the same 
contents in 97% of the tree.  Meaning dep:run,post? for 97% of the 
tree, w/ 3% needing to have an addition dep:post? section.

Offhand, comprehensive deps make things easier for devs- it gives them 
the ability to be crystal-freaking-clear as to what's needed at each 
stage; via them doing so, it means the resolver has a greater space to 
dig itself out of fucked up situations if necessary.

Additionally, it actually makes life easier for PM authors.  If we had 
TDEPEND (test), this is how things would go; note this is written for 
just an install, not a replacement (replace is similar, just noisier 
deflecting from my point):

1) ensure DEPEND is satisfied
2) run phases setup -> compile
3) ensure TDEPEND is satisfied;  Implicitly, DEPEND in the process (if 
there is a cycle induced by TDEPEND + DEPEND being required, we have 
no way out of it).
4) run install phase
5) ensure RDEPEND is satisfied.  No longer care about TDEPEND/DEPEND.
6) run pkg_preinst, merge, pkg_postinst.
7) a transaction is opened up, that must be completed before the PM 
exits; before that exit, RDEPEND must still be satisfied, as must 
PDEPEND.

With what I'm suggesting.
1) ensure dep:build is satisfied
2) run phases setup -> compile
3) Ensure dep:test is satisfied.  If cycle breaking is necessary, 
anything in dep:build but not dep:test can be pulled.
4) install phase
5) ensure dep:run is satisfied.  Again, if cycle breaking is 
necessary, punt whatever isn't in dep:run as necessary to break that 
cycle.
6) pkg_preinst, merge, pkg_postinst.
7) transactional block; ensure dep:post is satisfied before exiting 
said transaction.

That's for install; for the PM considering a pre-installed pkg, it 
converts from;
1) ensure RDEPEND and PDEPEND are satisfied

to
1) ensure dep:post is satisfied.

Tbh, I'm not sure I'm communicating these gains as well as I could be; 
at first glance, I'm pretty sure people will say "what's the 
difference?" because the gains/difference there are subtle.

From a PM level, I can tell you this shift to complete deps per stage 
*does* make things easier via simplifying the rules at each step 
(namely, render for this form, that's what's required for this point 
on), while giving full control to devs to be dead on accurate in their 
deps specified (which the PM can then exploit for better cycle 
breaking).

The point I'm trying to make here is that each dep phase 
should be authorative; in doing so, you start getting 
a lot of potential subsets (DEPEND is a subset of TDEPEND, TDEPEND 
isn't completely, but mostly a subset of RDEPEND as RDEPEND is a 
likely a superset of DEPEND; PDEPEND is a superset of RDEPEND).

So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in 
the ebuild.  Or you could just use a syntax form that allows you to 
directly inline that up front, rather than having to muck around w/ 
intermediate vars.


> Besides, this isn't actually a -problem- as there's nothing which
> really requires one to use such helpers; ebuild writers just, well,
> can.  :)

Getting to this point; yes, people could hack around it manually.  
Pretty sure that hasn't been in doubt.

The question is what do we *gain* from making them do so.  No one here 
has stated what we gain from COMMON_DEPENDS in comparison to 
DEPENDENCIES (where they can just inline it).

And as I'm trying to lay out, where dependencies *should* be going 
towards, this issue will get worse if we're doing it out of band from 
DEPENDENCIES.

~harring