[gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

Hello


Please reply to gentoo-dev in case ML daemon changes Reply-To.


### TL; DR ###

By introducing feature patches which menu options are disabled by
default to genpatches, we can deduplicate *-sources maintainers as well
as large groups of users work. By introducing a distribution section
in the menuconfig, we can provide options that enable minimal Gentoo
requirements by default (DEVTMPFS, making Gentoo systems bootable since
an udev release a long time ago) and other distribution stuff.


### Proper distribution integration of kernel *-sources, patches ... ###

Gentoo is a distribution; but what is a distribution that doesn't
properly integrate what it provides, but instead expects its users to
do so, needlessly duplicating work amongst its maintainers and users.

Let's say I want genpatches, aufs and TuxOnIce; closest candidates:

 - sys-kernel/aufs-sources: genpatches, aufs
 - sys-kernel/pf-sources: genpatches, CK, BFQ, BFQ, TuxOnIce, UKSM
 - sys-kernel/tuxonice-sources: genpatches, TuxOnIce

What do I do? Take one (eg. aufs) and apply the other (eg. TuxOnIce)?
What if I want to add another common patchset to it? Hardened perhaps?

You can see, some of these sources (like pf-sources) already attempt to
do so; with pf-sources in mind, why do we even have ck-sources,
tuxonice-sources in the Portage tree? Just to duplicate work?

I think we should trim down on the amount of *-sources and combine
multiple patches into genpatches. Take an example of geek-sources
which does all this without a problem; I don't really like the approach
with USE flags used there, as the menuconfig can just cover that.

https://github.com/init6/init_6/wiki/geek-sources

What does a patch introducing new features really do? Or rather, what
should it do when we add it? Let me summarize:

 1) The features should be disabled by default.
 2) These feature should depend on a non-vanilla / experimental option.
 3) The patch should not affect the build by default.
 4) The user can optionally enable the feature.

So, in genpatches, since 3.9.7, BFQ was added to try this out (except
2.). Ensured it to be disabled by default, did not affect the build for
anyone that does not enable it and the users have been enabling and
using it on their own. So, does it differentiate more from vanilla? No.

This helps users as well as maintainers to not have to apply BFQ on
their own, they simply have to tick a config option and they are set.
If all patches that introduce new features are added in this fashion,
it spares large groups of users from having to do this on their own; we
can also deduplicate the work in the Portage tree this way.


### ... and configuration. ###

This problem is not only visible for patches, but also in the config.

Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
telling users to enable it in some places, in the handbook it's a single
line you must read, on the Wiki it's kind of missing unless you are
luckily on the right page, on the Quick Install book it is missing too.

So, we are currently providing a configuration that expects anyone to
enable CONFIG_DEVTMPFS. But you have to remember that it need to make
sure you read about it or enable it from the udev upgrade a while ago
if you decide to start from a fresh config or are installing without
that handbook you kind of have memorized.

Searching for CONFIG_DEVTMPFS in the forums and #gentoo logs shows that
this is quite often suggested as a fix and quite often actually fixes
an user's boot. Why duplicate telling users to do that if we can do it?

There are a small set of other variables in this nature, mostly *TMPFS*.

This is why I think it would be handy to add a Gentoo section to the
kernel, along the lines as described by Linus.

https://lkml.org/lkml/2012/7/13/369

The same goes for asking the user to enable configuration options in the
kernel, why can't we just tell him to enable all option that toggles
support for the user. For example; in the Gentoo section, there would be
a "Init System Support" under which you can toggle an option to support
the minimal requirements for some init system.

Feedback is very welcome.

P.S.: Not everything in this mail has been acked by the kernel lead;
only some thoughts, I was suggested to take this to ML for discussion.
The usage of the word 'we' in this mail is therefore hypothetical.


### F.A.Q. ###

Q: I don't want feature X? Please don't add the patch!

A: It's optional, don't enable it in your menu config.

Q: What about my stable server? I really don't want to run this stuff!

A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL
   which would be disabled by default, therefore if you keep this option
   the way it is on your stable server; it won't affect you.

   In other words, genpatches stay as stable as before; unless you
   explicitly toggle options that intentionally make it unstable.

Q: Genpatches used to be minimal, would it gain weight?

A: If you don't enable those options, it would be as minimal as before;
   this gives the user the choice between minimal and fat.

Q: What about patches that fail to build / run?

A: For testing kernels (~), we would do our best to make them work but
   there could be occasions where we will have to cut them; this is no
   different from how the kernel herd has been handling this before, we
   have already dropped fbcondecor in the past when it was broken and
   the current new branch 3.10 does not support deblob yet.

   For stable kernels, which are near the EOL of a branch; if the
   feature isn't still there yet then it means that its author is
   simply no longer working on it, this is no different from when the
   patch would be in another *-sources or manually applied by the user.

   If you are unaware of how upstream releases work, just note that
   the major patch between 3.9.8 and 3.10.0 introduces a lot more
   changes are applied than the minor patch between 3.9 and 3.9.8;
   therefore, we prefer to stabilize 3.9.8 or a later 3.9 kernel than
   stabilizing 3.10.0. You can see this pattern in history; the
   previous three have been 3.6.11, 3.7.10 and 3.8.13.

Q: Can't you do something about those build and runtime failures?
   I like to run testing, but I don't want breaks half of the time.

A: To cover this, live ebuilds help, 3.10.9999 and 3.9999 and 9999;
   the earlier ones tracking the stable queues, whereas the last one
   tracking the upstream rc kernel. This way we can notice what fails
   in advance and make it working by the time it reaches testing (~).

   That being said, it is to be expected that things are not always
   fine very early in a branch; it's why those kernels are never stable.

Q: What about kernel bugs, how would you know the user enabled them?

A: We expect the .config to be provided, which we can run a sanity
   check on; it's much more handy if we are aware of these patches than
   that we have to figure out the user applies it on their own.

Q: Are documentation / infrastructure / eclass changes necessary?

A: The kernel project documentation has to be updated to reflect this;
   no infrastructure or eclass changes are needed, since this is all
   done through patches in genpatches. (Sub directories supported afaik)

Q: Does this affect kernel stabilization or QA?

A: No, experimental features (which would include these optional
   features) have historically never been looked at by the arch teams;
   if you can remember, there used to be an CONFIG_EXPERIMENTAL option
   in the kernel to cover this. For example, you can find yourself
   running some pretty unstable code if you use 3.8.13 and enable
   Nouveau reclocking and power management; there are some options in
   the menuconfig that follow this nature.

   As there would be less *-sources as a consequence; less has to be
   taken into consideration for stabilization, this is why most
   *-sources are currently unstable, because we don't have the
   resources to be stabilizing them all.

   For QA, I don't really see a problem.

Q: So, would this make vanilla-sources the new gentoo-sources?

A: No, vanilla-sources undergoes some changes to more closely reflect
   the upstream kernels; as a consequence, it would no longer have
   stable kernels and older versions will drop more often. You can read
   this request and the resulting discussion at the following link.

   http://thread.gmane.org/gmane.linux.gentoo.kernel/697

   The kernel lead summarized a part of this.

   http://thread.gmane.org/gmane.linux.gentoo.kernel/697/focus=730

Q: Is this a one man army comedy show?

A: One man army comedy shows work, see geek-sources; if you want to
   look at some examples of what a one man army can do, the following
   link is a good read to go through.

   http://www.bennorthrop.com/Essays/2013/pair-programming-my-personal-nightmare.php

   But joking aside; no, we are not.

   We are at least with two on the kernel herd and a third developer is
   likely to join in the near future; on top of that, other people are
   welcome and I think it would be nice to see maintainers from
   *-sources jump in to help along. genpatches isn't hard to maintain.

   Since we're no longer a one man army, we can do more.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@g.o
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D