Re: [gentoo-project] Making Gentoo more fun

[Daniel Robbins]

On Sat, Feb 17, 2018 at 2:29 AM, Tiziano Müller <dev-zero@g.o> wrote:

> Hi Daniel
>
> I haven't thought enough yet about the actual proposal to comment on it,
> but this systems seems to be similar to what openSuse has: projects
> taking care of a set of packages in a bleeding edge manner, occasional
> snapshots of those projects (or single packages?) make it into a final
> distro, which afterwards gets backports only. Or am I wrong?
> There it seems to work quiet well so far, except for some packages
> shared between all the projects (like the compiler), which are then
> lagging behind. How do you avoid this or similar issues of
> cross-dependencies between the kits? Especially with the possibility of
> changing dependencies with USE flags?
>

Well, Funtoo used to lag far behind Gentoo when it came to glibc and
compiler, due to being a much smaller project. But we have found that this
new system actually allows us to accelerate our adoption of newer things,
since we spend less time triaging breakages. Rather than focus so much
effort on bumping packages in the main tree, we have the freedom to make
more aggressive changes in the non-prime kits as we get them ready. We have
a stability rating system for our kits, so we can tag one as 'development
only' and hack away at it. So the development effort happens in kits that
users are not (yet) using directly. But it can happen more efficiently.

As far as USE flags, we have found a number of elegant solutions. Before
explaining, let me strip away some of the magic. Kits are of course
essentially overlays. And the different versions of kits are git branches.
Our core kit contains all the core system stuff, and a complete set of
eclasses, as well as the profiles. Other kits contain their collection of
catpkgs, as well as a snapshot of eclasses they use (taken from the same
snapshot to help ensure compatibility). We can add newer versions of
eclasses as needed, if necessary. Then we have "nokit", which is our
default kit of leftover catpkgs that we have decided not to create kits for
yet.

Here is one technique we use to support the flexibility of users changing
kit branches. We can move profile stuff related to gnome into the gnome-kit
itself, so that changing your gnome-kit also allows for the gnome profile
to automagically have new profile settings. We can do this by having the
gnome profile in core-kit reference gnome-kit:foo/bar/gnome-profile.

Next trick. As you know, there are python ebuilds everywhere, in all kits.
And users may want to switch their python-kit. But then, do all the
python-use settings become broken? Actually, no, there is an elegant
solution to this. Before kits, I had a script that would automatically set
the correct python-use settings so that everything would work -- assuming
you were using python-3.4, that is. But let's say you want to use
python-3.6. Does that meant that all the python-use settings in the tree
are now broken for you? Nope.Thanks to Funtoo's multi-profile system, my
"system profile" looks like this:

core-kit:funtoo/1.0/linux-gnu/arch/x86-64bit
core-kit:funtoo/1.0/linux-gnu/build/current
core-kit:funtoo/1.0/linux-gnu/arch/x86-64bit/subarch/intel64-haswell
core-kit:funtoo/1.0/linux-gnu/flavor/desktop
core-kit:funtoo/1.0/linux-gnu/mix-ins/gnome
nokit:funtoo/kits/python-kit/3.6-prime
haskell-kit:funtoo/kits/python-kit/3.6-prime
media-kit:funtoo/kits/python-kit/3.6-prime
xfce-kit:funtoo/kits/python-kit/3.6-prime
xorg-kit:funtoo/kits/python-kit/3.6-prime
perl-kit:funtoo/kits/python-kit/3.6-prime
gnome-kit:funtoo/kits/python-kit/3.6-prime
games-kit:funtoo/kits/python-kit/3.6-prime
core-hw-kit:funtoo/kits/python-kit/3.6-prime
desktop-kit:funtoo/kits/python-kit/3.6-prime
ruby-kit:funtoo/kits/python-kit/3.6-prime
security-kit:funtoo/kits/python-kit/3.6-prime
java-kit:funtoo/kits/python-kit/3.6-prime
php-kit:funtoo/kits/python-kit/3.6-prime
core-kit:funtoo/kits/python-kit/3.6-prime
ml-lang-kit:funtoo/kits/python-kit/3.6-prime
kde-kit:funtoo/kits/python-kit/3.6-prime
lisp-scheme-kit:funtoo/kits/python-kit/3.6-prime
editors-kit:funtoo/kits/python-kit/3.6-prime
text-kit:funtoo/kits/python-kit/3.6-prime
dev-kit:funtoo/kits/python-kit/3.6-prime
lang-kit:funtoo/kits/python-kit/3.6-prime
science-kit:funtoo/kits/python-kit/3.6-prime
net-kit:funtoo/kits/python-kit/3.6-prime
python-kit:funtoo/kits/python-kit/3.6-prime

It looks complicated but is actually all quite simple. Every kit has an
embedded python-kit/3.6-prime and python-kit/3.4-prime profile. These
profiles contain the proper python-use settings for the python-using
ebuilds in that particular kit branch. So if you switch from python-kit
3.4-prime to 3.6-prime, ego, our configuration tool, automatically adjusts
your profile to point to all those 3.6-prime profiles in each kit. Now your
python-use settings are perfect.

So there are a number of elegant solutions to manage these things. We also
have full automation of our kit generation. We actually use two primary git
trees to generate our kits -- one is the gentoo tree, of course, and the
other is a tree called kit-fixups. Our merge scripts select catpkgs from
the gentoo tree using package lists, which can also do things like select
all catpkgs from a category that use a specific eclass, and other neat
tricks to efficiently select catpkgs, and then we add our forks and fix-ups
on top of this. By updating the kit package lists, we have the ability to
move a catpkg from kit A to kit B totally transparently. The next time a
user syncs, it is gone from kit A and in kit B and causes no issues for
Portage or users. This is due to the design of our sync system which
ensures that you are using kit commits that are designed to work together.

Bigger picture -- yes, there will always be interdependencies between kits
and these need to be managed carefully. But, we have the ability to create
new kits as needed, and move catpkgs to and fro as needed, and this does
not impact users at all -- it's totally transparent -- so this helps a lot.
And it is easier to track interdependencies between 10 trees than have one
huge monolithic tree where the interdependencies are effectively not
tracked at all (at least not formally.) At least you can start the process
of mapping the interdependencies between the kits and understanding how one
kit connects to another kit and what the potential impacts of changing
things are. I think that's actually an important thing to begin doing --
start breaking down the dependency problem into smaller, more manageable
chunks.

I hope that gives you a bit of insight into the system. This news post is
somewhat out-of-date but gives you some perspective about my reasoning for
moving to a kits system:
https://www.funtoo.org/News:New_Ports-2017_tree_and_Kits (the parts that
are italicized have the key points)

Anyway, although I am happy to talk about the kits system in Funtoo, what I
am hoping is that having a more flexible system in Gentoo could provide the
ability for more junior developers to work in a kit that doesn't directly
impact users, but still do important work. And then have a workflow that
gradually promotes this kit to being a 'prime' kit when it has been
thoroughly tested. If we improve quality in Gentoo, and have processes that
support this, it will give newer developers a more pleasant environment to
ramp up and also provide the quality assurance systems that senior
developers really want to see moving forward, so they can relax a bit. The
goal is for the processes and technology to help support positive
collaboration and, yes, 'fun'.

Best,

Daniel