Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Tomáš Pružina]

This reminds me of: http://imgs.xkcd.com/comics/standards.png
It sure sounds nicely, however I would not want to be the guy who maintains
the whole mess of (often) incompatible patchsets.
Given the fact that some patches lag 1-3 stable versions behind Linuses
tree (grsec used by hardened for example),
this might be better idea on paper than in reality.

If you feel up for this, go for it. Be ready to meet resistance though.

// I personally maintain my own tree, based on stable from gregkh and few
patches

On Mon, Jul 1, 2013 at 4:41 PM, Tom Wijsman <TomWij@g.o> wrote:

> Hello
>
>
> Please reply to gentoo-dev in case ML daemon changes Reply-To.
>
>
> ### TL; DR ###
>
> By introducing feature patches which menu options are disabled by
> default to genpatches, we can deduplicate *-sources maintainers as well
> as large groups of users work. By introducing a distribution section
> in the menuconfig, we can provide options that enable minimal Gentoo
> requirements by default (DEVTMPFS, making Gentoo systems bootable since
> an udev release a long time ago) and other distribution stuff.
>
>
> ### Proper distribution integration of kernel *-sources, patches ... ###
>
> Gentoo is a distribution; but what is a distribution that doesn't
> properly integrate what it provides, but instead expects its users to
> do so, needlessly duplicating work amongst its maintainers and users.
>
> Let's say I want genpatches, aufs and TuxOnIce; closest candidates:
>
>  - sys-kernel/aufs-sources: genpatches, aufs
>  - sys-kernel/pf-sources: genpatches, CK, BFQ, BFQ, TuxOnIce, UKSM
>  - sys-kernel/tuxonice-sources: genpatches, TuxOnIce
>
> What do I do? Take one (eg. aufs) and apply the other (eg. TuxOnIce)?
> What if I want to add another common patchset to it? Hardened perhaps?
>
> You can see, some of these sources (like pf-sources) already attempt to
> do so; with pf-sources in mind, why do we even have ck-sources,
> tuxonice-sources in the Portage tree? Just to duplicate work?
>
> I think we should trim down on the amount of *-sources and combine
> multiple patches into genpatches. Take an example of geek-sources
> which does all this without a problem; I don't really like the approach
> with USE flags used there, as the menuconfig can just cover that.
>
> https://github.com/init6/init_6/wiki/geek-sources
>
> What does a patch introducing new features really do? Or rather, what
> should it do when we add it? Let me summarize:
>
>  1) The features should be disabled by default.
>  2) These feature should depend on a non-vanilla / experimental option.
>  3) The patch should not affect the build by default.
>  4) The user can optionally enable the feature.
>
> So, in genpatches, since 3.9.7, BFQ was added to try this out (except
> 2.). Ensured it to be disabled by default, did not affect the build for
> anyone that does not enable it and the users have been enabling and
> using it on their own. So, does it differentiate more from vanilla? No.
>
> This helps users as well as maintainers to not have to apply BFQ on
> their own, they simply have to tick a config option and they are set.
> If all patches that introduce new features are added in this fashion,
> it spares large groups of users from having to do this on their own; we
> can also deduplicate the work in the Portage tree this way.
>
>
> ### ... and configuration. ###
>
> This problem is not only visible for patches, but also in the config.
>
> Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
> telling users to enable it in some places, in the handbook it's a single
> line you must read, on the Wiki it's kind of missing unless you are
> luckily on the right page, on the Quick Install book it is missing too.
>
> So, we are currently providing a configuration that expects anyone to
> enable CONFIG_DEVTMPFS. But you have to remember that it need to make
> sure you read about it or enable it from the udev upgrade a while ago
> if you decide to start from a fresh config or are installing without
> that handbook you kind of have memorized.
>
> Searching for CONFIG_DEVTMPFS in the forums and #gentoo logs shows that
> this is quite often suggested as a fix and quite often actually fixes
> an user's boot. Why duplicate telling users to do that if we can do it?
>
> There are a small set of other variables in this nature, mostly *TMPFS*.
>
> This is why I think it would be handy to add a Gentoo section to the
> kernel, along the lines as described by Linus.
>
> https://lkml.org/lkml/2012/7/13/369
>
> The same goes for asking the user to enable configuration options in the
> kernel, why can't we just tell him to enable all option that toggles
> support for the user. For example; in the Gentoo section, there would be
> a "Init System Support" under which you can toggle an option to support
> the minimal requirements for some init system.
>
> Feedback is very welcome.
>
> P.S.: Not everything in this mail has been acked by the kernel lead;
> only some thoughts, I was suggested to take this to ML for discussion.
> The usage of the word 'we' in this mail is therefore hypothetical.
>
>
> ### F.A.Q. ###
>
> Q: I don't want feature X? Please don't add the patch!
>
> A: It's optional, don't enable it in your menu config.
>
> Q: What about my stable server? I really don't want to run this stuff!
>
> A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL
>    which would be disabled by default, therefore if you keep this option
>    the way it is on your stable server; it won't affect you.
>
>    In other words, genpatches stay as stable as before; unless you
>    explicitly toggle options that intentionally make it unstable.
>
> Q: Genpatches used to be minimal, would it gain weight?
>
> A: If you don't enable those options, it would be as minimal as before;
>    this gives the user the choice between minimal and fat.
>
> Q: What about patches that fail to build / run?
>
> A: For testing kernels (~), we would do our best to make them work but
>    there could be occasions where we will have to cut them; this is no
>    different from how the kernel herd has been handling this before, we
>    have already dropped fbcondecor in the past when it was broken and
>    the current new branch 3.10 does not support deblob yet.
>
>    For stable kernels, which are near the EOL of a branch; if the
>    feature isn't still there yet then it means that its author is
>    simply no longer working on it, this is no different from when the
>    patch would be in another *-sources or manually applied by the user.
>
>    If you are unaware of how upstream releases work, just note that
>    the major patch between 3.9.8 and 3.10.0 introduces a lot more
>    changes are applied than the minor patch between 3.9 and 3.9.8;
>    therefore, we prefer to stabilize 3.9.8 or a later 3.9 kernel than
>    stabilizing 3.10.0. You can see this pattern in history; the
>    previous three have been 3.6.11, 3.7.10 and 3.8.13.
>
> Q: Can't you do something about those build and runtime failures?
>    I like to run testing, but I don't want breaks half of the time.
>
> A: To cover this, live ebuilds help, 3.10.9999 and 3.9999 and 9999;
>    the earlier ones tracking the stable queues, whereas the last one
>    tracking the upstream rc kernel. This way we can notice what fails
>    in advance and make it working by the time it reaches testing (~).
>
>    That being said, it is to be expected that things are not always
>    fine very early in a branch; it's why those kernels are never stable.
>
> Q: What about kernel bugs, how would you know the user enabled them?
>
> A: We expect the .config to be provided, which we can run a sanity
>    check on; it's much more handy if we are aware of these patches than
>    that we have to figure out the user applies it on their own.
>
> Q: Are documentation / infrastructure / eclass changes necessary?
>
> A: The kernel project documentation has to be updated to reflect this;
>    no infrastructure or eclass changes are needed, since this is all
>    done through patches in genpatches. (Sub directories supported afaik)
>
> Q: Does this affect kernel stabilization or QA?
>
> A: No, experimental features (which would include these optional
>    features) have historically never been looked at by the arch teams;
>    if you can remember, there used to be an CONFIG_EXPERIMENTAL option
>    in the kernel to cover this. For example, you can find yourself
>    running some pretty unstable code if you use 3.8.13 and enable
>    Nouveau reclocking and power management; there are some options in
>    the menuconfig that follow this nature.
>
>    As there would be less *-sources as a consequence; less has to be
>    taken into consideration for stabilization, this is why most
>    *-sources are currently unstable, because we don't have the
>    resources to be stabilizing them all.
>
>    For QA, I don't really see a problem.
>
> Q: So, would this make vanilla-sources the new gentoo-sources?
>
> A: No, vanilla-sources undergoes some changes to more closely reflect
>    the upstream kernels; as a consequence, it would no longer have
>    stable kernels and older versions will drop more often. You can read
>    this request and the resulting discussion at the following link.
>
>    http://thread.gmane.org/gmane.linux.gentoo.kernel/697
>
>    The kernel lead summarized a part of this.
>
>    http://thread.gmane.org/gmane.linux.gentoo.kernel/697/focus=730
>
> Q: Is this a one man army comedy show?
>
> A: One man army comedy shows work, see geek-sources; if you want to
>    look at some examples of what a one man army can do, the following
>    link is a good read to go through.
>
>
> http://www.bennorthrop.com/Essays/2013/pair-programming-my-personal-nightmare.php
>
>    But joking aside; no, we are not.
>
>    We are at least with two on the kernel herd and a third developer is
>    likely to join in the near future; on top of that, other people are
>    welcome and I think it would be nice to see maintainers from
>    *-sources jump in to help along. genpatches isn't hard to maintain.
>
>    Since we're no longer a one man army, we can do more.
>
> --
> With kind regards,
>
> Tom Wijsman (TomWij)
> Gentoo Developer
>
> E-mail address  : TomWij@g.o
> GPG Public Key  : 6D34E57D
> GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
>