Re: [gentoo-alt] branched prefix chaining

[Viktor Griph]

2009/7/8 Markus Duft <mduft@g.o>:
> Also I couldn't think of a real use case where this would be absolutely
> necessary. i admit that I'm slightly influenced by the use case here at
> our company: we have one "system prefix", building upon that we have a
> "framework prefix" and again, building upon that the "application
> prefix".
> the "system prefix" is shared upon all "framework prefixes". each
> "framework prefix" is used by multiple "application prefixed" each of
> them containing one final application. So we're perfect with a one
> dimensional chain. (the "one dimensional" is from the POV of each
> involved prefix, of course. looking at the whole picture we have this:
>
>      s1
>      |
>   +--+--+
>  f1  f2 f3
>   |  |  +--+
>   |  |  |  |
>  a1  a2 a3 a4
>
> so, none of the applications (a1-a4) know of each other. those prefixes
> are unrelated and each application has a one dimensional direct chain to
> s1).
>
> could you describe your current and desired setup a little closer? would
> be interesting... :)

My intention is to use portage for easy installation of software, not
supported by the IT department, at my university. The policy for such
software is that each should be installed in its own directory, and
that old versions should be kept for some time after an upgrade. To
satisfy this policy I hope to be able to use chained prefixes. The
idea is to build up a more or less minimal gentoo prefix installation
with prefix chaining enabled, and to that add a bunch of scripts that
wraps around portage for installing application in sush a way that
each and every dependency is installed in its own prefix, and
registered with the university software packaging system.

I also would like to make use of dependencies from the host system
provided by the OS (i.e. not manually installed by other users with
right to install software) to avoid building libraries that already
exist on the system. Finally te applications installed should be
usable from the host OS without having to set additinal variables,
which means that binaries has to be linked with -rpath to find the
right libraries on runtime.

So in more detail the set up will be (where $PHC is the root of the
phc (software installation point)

$PHC/gentoo-prefix -- gentoo prefix installation with portage, and set
up to recognize libraries on the host OS (via
/etc/portage/profile/package.provided)
$PHC/${P} - minimal prefix which inherits the prefix of all immediate
dependencies to the packages ${P} and contains ${P} emerged ans well
as some control files for the university software packaging system.

There will be other drectories in $PHC, possible ones with conflicting
names, and the portage wrapper I'm writing will have to have a
strategy for that as well. I'm incline to let the prtage wrapper be
aware of any prefixes created with it, and trhe ebauilds installed in
them to allow it to setup prefix chains including already built
dependencies of earlier versions than the ones in the current portage
tree, but I'm not sure if I'll bother since it mmay be good to build
the latest version of the dependencies anyway.


>
>>
>> I'm tempted to try to change this myself, but before I do I'd like to
>> get some input on why it was made the way it is. A few things I've
>> noticed about the chaining prefixes as they are now:
>> 1) the prefixes are linked in with no particular order in portage, but
>> follows the chain in /etc/profile. I'm not sure if the portage order
>> makes any difference as I've very little experience with this yet, but
>> I think that those prefixes should be included in order as well.
>
> what do you mean by "linked in with no particular order in portage"?
> since the chaining is a strictly one dimensional chain (at least
> ATM :)), the ordering is implicit. the top most prefix (it may or may
> not contain it's own portage btw.) reads it's config, finds a readonly
> prefix, reads that one's config, finds another one, etc. this way the
> whole chain of configurations is built up during portage startup. hope
> that was clear...?

But aren't the search paths set in the PORTAGE_READONLY_EPREFIXS
variable joined from a map, in which case the order the keys are
entered to the map is not guaranteed to be the order the keys are
joined? So while the prefixes are added to the map in order, the order
is forgotten immediately, or am I missing something?

>
>>
>> My idea on how branching prefixes could be done is similar to how the
>> chaining prefixes work now, but with a little extra logic required to
>> not include the same prefix more than once.
>
> so you want something like this?:
>
>         a
>         |
>      +--+--+
>      b  c  d
>      +--+  |
>         e  |
>         +--+
>         f
>
> where f inherits from *all* prefixes?

Basically yes, but also, in the f config file it could include a
reference to b as well. I'm not yet sure of what would be the best
search order for joining paths, but I'm thinking standard left to rigt
search order, so that the first prefix satisfied is searched first,
and after thinking about it for a wile, I believe that a depth first
search is in order, so that if f specifies prefixes "d e" and e "c b"
it would first search d then a then e then c and finally b.

> i guess there is a reason why
> modern OO languages forbid multiple inheritance - it's a headache! :)
> And still those languages are quite popular, and seemingly don't lack
> any obvious big features. one of them is java, so lack of multiple
> inheritance can't really hurt, can it?

While I could probably do without multiple inheritnce, (for instance a
very deep chain) it would add extra difficulties, in the event that a
package were to be removed, and also increas the risk of linking to
the wrong version, since a chain could have multiple version of the
same package, as if a new package were installed, it would end up at
the top of the chain, and the old package would still be inside the
chain, whereas with branching prefixes I would never have more than
one version of the same package in the chain (with possible exception
of packages in the base prefix installation, or packages provided by
the host OS via package.provided)

Or I could cain each package only with the base prefix, and setup
paths manually to other child prefixes, and add their packages to
package.provided, but that feels more hackerish than having proper
branched prefixes.

>
>> I think that changing the current inclusion to a breath first tree
>> traversal with marking of already visited prefixes should work for the
>> portage tree. When it comes to the environment I really don't see why
>> it is made from /etc/profile recursively currently. Wouldn't it make
>> more sence to have env-update simply source the profile.env of the
>> parent prefix, and poin it with the env.d at the current level, and
>> possible include a utility for recursively doing env-update in several
>> prefix-chains if needed. At least that's the way I think would work
>> the best when it comes to multiple readonly prefixes. Is there any
>> reason why this wouldn't work? (Other than the fact that this would be
>> a depth first inclusion instead of a breath first inclusion as I
>> suggested for portage, if the order matters.)
>
> phew. IIRC it took me a few days to get that right in my head only,
> without implementing it (implementation was done in a few minutes
> actually) - thinking back to the problems i faced, the only one that
> jumps to my mind right now was that a patched env-update would not
> suffice because any of the parent prefix can change without us getting
> any note of it. (however of course there are more problems of that kind,
> like the parent prefix not knowing us, so he won't prevent unmerging
> libraries the child prefix needs).

Well, for my usecase it would be sufficient to have to run env-update
again if I change a parent prefix. That is what envc-update is for
anyway, take new changes into account, and it could be made enough
torun env-update in one prefix, if env-update is made to do full
recursion (as opposed to just ask the parents, which might not have
ran env-update yet).

> also, changing the type of inheritance (RDEPEND, DEPEND, PDEPEND) would
> require an env-update call - one thing that for sure is forgotten at
> least half of the time.

well, I just say that I think it is cleaner than putting it in
/etc/profile, which may be altered in the diferent prefixes, and
possible could be made to break the prefix chain. If it were in
/etc/profile.env on the other hand that would be OK, but I'd still be
in favor of the environment not changing unless I manually run
env-update.

/Viktor