Re: [gentoo-project] Making Gentoo more fun

[Alec Warner]

On Sat, Feb 17, 2018 at 6:34 AM, Daniel Robbins <drobbins@××××××.org> wrote:

> On Sat, Feb 17, 2018 at 2:29 AM, Tiziano Müller <dev-zero@g.o>
> wrote:
>
>> Hi Daniel
>>
>> I haven't thought enough yet about the actual proposal to comment on it,
>> but this systems seems to be similar to what openSuse has: projects
>> taking care of a set of packages in a bleeding edge manner, occasional
>> snapshots of those projects (or single packages?) make it into a final
>> distro, which afterwards gets backports only. Or am I wrong?
>> There it seems to work quiet well so far, except for some packages
>> shared between all the projects (like the compiler), which are then
>> lagging behind. How do you avoid this or similar issues of
>> cross-dependencies between the kits? Especially with the possibility of
>> changing dependencies with USE flags?
>>
>
> Well, Funtoo used to lag far behind Gentoo when it came to glibc and
> compiler, due to being a much smaller project. But we have found that this
> new system actually allows us to accelerate our adoption of newer things,
> since we spend less time triaging breakages. Rather than focus so much
> effort on bumping packages in the main tree, we have the freedom to make
> more aggressive changes in the non-prime kits as we get them ready. We have
> a stability rating system for our kits, so we can tag one as 'development
> only' and hack away at it. So the development effort happens in kits that
> users are not (yet) using directly. But it can happen more efficiently.
>
> As far as USE flags, we have found a number of elegant solutions. Before
> explaining, let me strip away some of the magic. Kits are of course
> essentially overlays. And the different versions of kits are git branches.
> Our core kit contains all the core system stuff, and a complete set of
> eclasses, as well as the profiles. Other kits contain their collection of
> catpkgs, as well as a snapshot of eclasses they use (taken from the same
> snapshot to help ensure compatibility). We can add newer versions of
> eclasses as needed, if necessary. Then we have "nokit", which is our
> default kit of leftover catpkgs that we have decided not to create kits for
> yet.
>
> Here is one technique we use to support the flexibility of users changing
> kit branches. We can move profile stuff related to gnome into the gnome-kit
> itself, so that changing your gnome-kit also allows for the gnome profile
> to automagically have new profile settings. We can do this by having the
> gnome profile in core-kit reference gnome-kit:foo/bar/gnome-profile.
>
> Next trick. As you know, there are python ebuilds everywhere, in all kits.
> And users may want to switch their python-kit. But then, do all the
> python-use settings become broken? Actually, no, there is an elegant
> solution to this. Before kits, I had a script that would automatically set
> the correct python-use settings so that everything would work -- assuming
> you were using python-3.4, that is. But let's say you want to use
> python-3.6. Does that meant that all the python-use settings in the tree
> are now broken for you? Nope.Thanks to Funtoo's multi-profile system, my
> "system profile" looks like this:
>
> core-kit:funtoo/1.0/linux-gnu/arch/x86-64bit
> core-kit:funtoo/1.0/linux-gnu/build/current
> core-kit:funtoo/1.0/linux-gnu/arch/x86-64bit/subarch/intel64-haswell
> core-kit:funtoo/1.0/linux-gnu/flavor/desktop
> core-kit:funtoo/1.0/linux-gnu/mix-ins/gnome
> nokit:funtoo/kits/python-kit/3.6-prime
> haskell-kit:funtoo/kits/python-kit/3.6-prime
> media-kit:funtoo/kits/python-kit/3.6-prime
> xfce-kit:funtoo/kits/python-kit/3.6-prime
> xorg-kit:funtoo/kits/python-kit/3.6-prime
> perl-kit:funtoo/kits/python-kit/3.6-prime
> gnome-kit:funtoo/kits/python-kit/3.6-prime
> games-kit:funtoo/kits/python-kit/3.6-prime
> core-hw-kit:funtoo/kits/python-kit/3.6-prime
> desktop-kit:funtoo/kits/python-kit/3.6-prime
> ruby-kit:funtoo/kits/python-kit/3.6-prime
> security-kit:funtoo/kits/python-kit/3.6-prime
> java-kit:funtoo/kits/python-kit/3.6-prime
> php-kit:funtoo/kits/python-kit/3.6-prime
> core-kit:funtoo/kits/python-kit/3.6-prime
> ml-lang-kit:funtoo/kits/python-kit/3.6-prime
> kde-kit:funtoo/kits/python-kit/3.6-prime
> lisp-scheme-kit:funtoo/kits/python-kit/3.6-prime
> editors-kit:funtoo/kits/python-kit/3.6-prime
> text-kit:funtoo/kits/python-kit/3.6-prime
> dev-kit:funtoo/kits/python-kit/3.6-prime
> lang-kit:funtoo/kits/python-kit/3.6-prime
> science-kit:funtoo/kits/python-kit/3.6-prime
> net-kit:funtoo/kits/python-kit/3.6-prime
> python-kit:funtoo/kits/python-kit/3.6-prime
>
> It looks complicated but is actually all quite simple. Every kit has an
> embedded python-kit/3.6-prime and python-kit/3.4-prime profile. These
> profiles contain the proper python-use settings for the python-using
> ebuilds in that particular kit branch. So if you switch from python-kit
> 3.4-prime to 3.6-prime, ego, our configuration tool, automatically adjusts
> your profile to point to all those 3.6-prime profiles in each kit. Now your
> python-use settings are perfect.
>
> So there are a number of elegant solutions to manage these things. We also
> have full automation of our kit generation. We actually use two primary git
> trees to generate our kits -- one is the gentoo tree, of course, and the
> other is a tree called kit-fixups. Our merge scripts select catpkgs from
> the gentoo tree using package lists, which can also do things like select
> all catpkgs from a category that use a specific eclass, and other neat
> tricks to efficiently select catpkgs, and then we add our forks and fix-ups
> on top of this. By updating the kit package lists, we have the ability to
> move a catpkg from kit A to kit B totally transparently. The next time a
> user syncs, it is gone from kit A and in kit B and causes no issues for
> Portage or users. This is due to the design of our sync system which
> ensures that you are using kit commits that are designed to work together.
>
> Bigger picture -- yes, there will always be interdependencies between kits
> and these need to be managed carefully. But, we have the ability to create
> new kits as needed, and move catpkgs to and fro as needed, and this does
> not impact users at all -- it's totally transparent -- so this helps a lot.
> And it is easier to track interdependencies between 10 trees than have one
> huge monolithic tree where the interdependencies are effectively not
> tracked at all (at least not formally.) At least you can start the process
> of mapping the interdependencies between the kits and understanding how one
> kit connects to another kit and what the potential impacts of changing
> things are. I think that's actually an important thing to begin doing --
> start breaking down the dependency problem into smaller, more manageable
> chunks.
>
> I hope that gives you a bit of insight into the system. This news post is
> somewhat out-of-date but gives you some perspective about my reasoning for
> moving to a kits system: https://www.funtoo.org/News:New_Ports-2017_tree_
> and_Kits (the parts that are italicized have the key points)
>
> Anyway, although I am happy to talk about the kits system in Funtoo, what
> I am hoping is that having a more flexible system in Gentoo could provide
> the ability for more junior developers to work in a kit that doesn't
> directly impact users, but still do important work. And then have a
> workflow that gradually promotes this kit to being a 'prime' kit when it
> has been thoroughly tested. If we improve quality in Gentoo, and have
> processes that support this, it will give newer developers a more pleasant
> environment to ramp up and also provide the quality assurance systems that
> senior developers really want to see moving forward, so they can relax a
> bit. The goal is for the processes and technology to help support positive
> collaboration and, yes, 'fun'.
>

I like the idea (having myself thought about a 'core' set of packages with
overlays) so I'm excited to see your implementation. I would avoid any
senior / junior designations (I think it blunts your message). Senior
people can cause screwups and break the tree and not care 'enough', just as
junior developers can. I think this is more a difference in goals for the
project than any specific 'experience level'.

I think that many developers are just here for the metadistribution level
and do not prioritize things like "QA" or "sane upgrades" or other things
that an *actual* distribution cares about, but perhaps a meta distribution
cares about less. This isn't meant as a judgement, but simply differences
in what people care about or are willing to put effort into.

Ultimately I think the model where people can work on their 'kits' and a
team who cares about 'stability' can assemble kits to produce a product is
a good model. I'd like to see more data and time regarding the Funtoo
implementation; particularly around kit lifecycles.

-A


>
> Best,
>
> Daniel
>
>