Re: [gentoo-dev] RFC - Gentoo on the Lab

[Brian Harring]

Lot of text left inline, pardon, just scroll and deal with it :P

On Tue, Aug 23, 2005 at 12:28:08PM -0400, Kristian Benoit wrote:
> Here is my recent communication with Pieter:
>
> On Sat, 2005-08-13 at 21:59 +0200, Pieter Van den Abeele wrote:
> > On 13 Aug 2005, at 19:16, Kristian Benoit wrote:
> >
> > > I've heard that you might be the last to know something about
> > > portage-ng. What's the actual status,
> >
> > Here's what I've done so far:
> >
> > I've build upon Andreas' Zeller idea of using Smolka's feature logic
> > for software configuration management. Zeller's approach was ok for
> > determining consistency/inconsistency of software configurations. In
> > his phd thesis he described the algorithm involved and discussed time
> > complexity (which goes to NP if you allow for quantification). My
> > impression after implementing his idea was that for automated
> > construction there were a few things that were lacking, and some
> > things were incorrect. I revisited Zellers feature logic, and ended
> > up with a clean implementation of a declarative language which has
> > some very desirable properties for automated software construction.
> > I'd say my approach is closer to the spirit of Smolka's feature logic
> > than Zellers approach. My non-destructive feature unification has a
> > worst case time complexity of O(n x m) where n is the length of the
> > feature term describing your system, and m is the length of the
> > feature term describing the component to be added. In practice
> > performance is very good. An emerge --vp --emptytree kde with the
> > regular portage takes about 55 seconds on my Open Desktop Workstation
> > and produces a list of 200 packages. A similar action using my
> > implementation:
> >
> > =====================================================================
> > Total time: 14.55 seconds
> > =====================================================================
> > Predicate                       Box Entries =    Calls+Redos     Time
> > =====================================================================
> > vunify/4                            341,900 =  341,900+0        72.6%
> > $garbage_collect/1                      326 =      326+0        13.6%
> > lists:append/3                      684,320 =  684,320+0         4.0%
> > genterm/2                               520 =      520+0         3.9%
> > hunify/4                                520 =      520+0         3.3%
> > is/2                                342,420 =  342,420+0         1.8%
> >  >/2                                 342,160 =  342,160+0         0.8%
> > buildsystem/2                             1 =        1+0         0.0%
> > val/3                                 5,200 =    5,200+0         0.0%
> > unify/3                                 260 =      260+0         0.0%
> >
> > % 9,531,861 inferences, 13.96 CPU in 14.55 seconds (96% CPU, 682798
> > Lips)
Stating it as nicely as possible, without knowing what that's doing, 
stats can be construed several ways; faster backend access (both vdb 
and ebuild/cache), dodging/caching virtuals, etc; language 
implementation is a point of curiousity also.

Faster implementation doesn't surprise me- stable portage is fricking 
*absolutely* retarded about caching, parsing of deps and cpvs, loading 
of profile, etc.  Either way, interest remains in seeing it :)

> > The search space is kept as small as possible because I've introduced
> > lazy feature evaluation and both multi valued and open features.
> > Those concepts are missing in Zellers approach. Comparing current
> > portage and my implementation performance-wise is difficult. In
> > general mine is faster, but current portage doesn't use sql either.
> > What is for sure is that mine allows you to express various
> > constraints. You can prevent a dependency from being built with a
> > specific use flag. My implementation automatically solves 'blockers'.
> > Circular dependencies are also solved correctly. For instance, if you
> > want to install unixodbc with the qt use flag enabled and you want to
> > install qt wit the unixodbc use flag enabled, my portage knows that
> > it needs to:
> >
> > emerge unixodbc without qt
> > emerge qt with unixodbc
> > remerge unixodbc with qt support
> >
> > So it makes revdep-rebuild unnecessary basically.
revdep-rebuild is still necessary, regardless of use deps or full 
state graphs- anyone doubting that should take a look at the breakage 
that has occured whenever mysql/ssl have decided to change their 
soname while maintaining compatibility.
Need a bincompat metadata attribute to kill off revdep-rebuild.


> > Once I was done implementing this new declarative language in which
> > for instance ebuild concepts can be easily expressed (but also rpm,
> > deb, fink, darwinports).

This sounds a lot like my restriction subsystem in the rewrite (code's 
available to anyone after it).

With the restriction setup, anyone who wants it could just plugin a 
different repo/pkg plugin, and have deps based on sonames fex; granted 
anyone doing it would need some form of a binary repository, but the 
agnostic aspect of representing matching criteria is the key to it.

> > I implemented a knowledge base in which
> > those feature terms can be stored/looked up efficiently. I used an
> > odbc bridge design pattern and have tested the thing with postgresql/
> > mysql and sqlite. I've also implemented a DCG parser which parses
> > ebuilds (no more having to start a bash process for each ebuild like
> > current portage does)

Strictly a stable slow down, and relevant when pulling metadata; 2.1 
(ebd patchset) is a daeonized ebuild processor effectively.  Shaves a 
good 30% off of runtime, while being proper (sans bugs).

Alternative implementations welcome of course, but ebd is pretty much 
something whipped out that addresses long standing bugs without 
requiring a full rewrite of parsing (interested parties can do it 
another day, I care not to after filter-env).
~harring