Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Rick \\\"Zero_Chaos\\\" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/01/2013 10:41 AM, Tom Wijsman wrote:
> Hello
>
>
> Please reply to gentoo-dev in case ML daemon changes Reply-To.
>
>
> ### TL; DR ###
>
> By introducing feature patches which menu options are disabled by
> default to genpatches, we can deduplicate *-sources maintainers as well
> as large groups of users work. By introducing a distribution section
> in the menuconfig, we can provide options that enable minimal Gentoo
> requirements by default (DEVTMPFS, making Gentoo systems bootable since
> an udev release a long time ago) and other distribution stuff.
>
>
> ### Proper distribution integration of kernel *-sources, patches ... ###
>
> Gentoo is a distribution; but what is a distribution that doesn't
> properly integrate what it provides, but instead expects its users to
> do so, needlessly duplicating work amongst its maintainers and users.
>
> Let's say I want genpatches, aufs and TuxOnIce; closest candidates:
>
>  - sys-kernel/aufs-sources: genpatches, aufs
>  - sys-kernel/pf-sources: genpatches, CK, BFQ, BFQ, TuxOnIce, UKSM
>  - sys-kernel/tuxonice-sources: genpatches, TuxOnIce
>
> What do I do? Take one (eg. aufs) and apply the other (eg. TuxOnIce)?
> What if I want to add another common patchset to it? Hardened perhaps?

Some patches are reasonably easy to combine, such as genpatches and
aufs.  Some patches are difficult to combine, such as hardened and *.
When you combine hardened patches and aufs (for example) you need extra
patches.  I would be THRILLED to see the number of sources cut down, but
hardened-sources must be it's own thing (that said, I'll personally
maintain the aufs patches for hardened if they wanted to add a USE=aufs
flag).
>
> You can see, some of these sources (like pf-sources) already attempt to
> do so; with pf-sources in mind, why do we even have ck-sources,
> tuxonice-sources in the Portage tree? Just to duplicate work?
>
> I think we should trim down on the amount of *-sources and combine
> multiple patches into genpatches. Take an example of geek-sources
> which does all this without a problem; I don't really like the approach
> with USE flags used there, as the menuconfig can just cover that.
>
> https://github.com/init6/init_6/wiki/geek-sources
>
> What does a patch introducing new features really do? Or rather, what
> should it do when we add it? Let me summarize:
>
>  1) The features should be disabled by default.
>  2) These feature should depend on a non-vanilla / experimental option.
If users want a vanilla kernel, they want vanilla-sources.  Nothing
about that should change.  I don't feel that it would be honest to add a
vanilla use flag to gentoo-sources as in no reality are those vanilla.

Thanks,
Zero

>  3) The patch should not affect the build by default.
>  4) The user can optionally enable the feature.
>
> So, in genpatches, since 3.9.7, BFQ was added to try this out (except
> 2.). Ensured it to be disabled by default, did not affect the build for
> anyone that does not enable it and the users have been enabling and
> using it on their own. So, does it differentiate more from vanilla? No.
>
> This helps users as well as maintainers to not have to apply BFQ on
> their own, they simply have to tick a config option and they are set.
> If all patches that introduce new features are added in this fashion,
> it spares large groups of users from having to do this on their own; we
> can also deduplicate the work in the Portage tree this way.
>
>
> ### ... and configuration. ###
>
> This problem is not only visible for patches, but also in the config.
>
> Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
> telling users to enable it in some places, in the handbook it's a single
> line you must read, on the Wiki it's kind of missing unless you are
> luckily on the right page, on the Quick Install book it is missing too.
>
> So, we are currently providing a configuration that expects anyone to
> enable CONFIG_DEVTMPFS. But you have to remember that it need to make
> sure you read about it or enable it from the udev upgrade a while ago
> if you decide to start from a fresh config or are installing without
> that handbook you kind of have memorized.
>
> Searching for CONFIG_DEVTMPFS in the forums and #gentoo logs shows that
> this is quite often suggested as a fix and quite often actually fixes
> an user's boot. Why duplicate telling users to do that if we can do it?
>
> There are a small set of other variables in this nature, mostly *TMPFS*.
>
> This is why I think it would be handy to add a Gentoo section to the
> kernel, along the lines as described by Linus.
>
> https://lkml.org/lkml/2012/7/13/369
>
> The same goes for asking the user to enable configuration options in the
> kernel, why can't we just tell him to enable all option that toggles
> support for the user. For example; in the Gentoo section, there would be
> a "Init System Support" under which you can toggle an option to support
> the minimal requirements for some init system.
>
> Feedback is very welcome.
>
> P.S.: Not everything in this mail has been acked by the kernel lead;
> only some thoughts, I was suggested to take this to ML for discussion.
> The usage of the word 'we' in this mail is therefore hypothetical.
>
>
> ### F.A.Q. ###
>
> Q: I don't want feature X? Please don't add the patch!
>
> A: It's optional, don't enable it in your menu config.
>
> Q: What about my stable server? I really don't want to run this stuff!
>
> A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL
>    which would be disabled by default, therefore if you keep this option
>    the way it is on your stable server; it won't affect you.
>
>    In other words, genpatches stay as stable as before; unless you
>    explicitly toggle options that intentionally make it unstable.
>
> Q: Genpatches used to be minimal, would it gain weight?
>
> A: If you don't enable those options, it would be as minimal as before;
>    this gives the user the choice between minimal and fat.
>
> Q: What about patches that fail to build / run?
>
> A: For testing kernels (~), we would do our best to make them work but
>    there could be occasions where we will have to cut them; this is no
>    different from how the kernel herd has been handling this before, we
>    have already dropped fbcondecor in the past when it was broken and
>    the current new branch 3.10 does not support deblob yet.
>
>    For stable kernels, which are near the EOL of a branch; if the
>    feature isn't still there yet then it means that its author is
>    simply no longer working on it, this is no different from when the
>    patch would be in another *-sources or manually applied by the user.
>
>    If you are unaware of how upstream releases work, just note that
>    the major patch between 3.9.8 and 3.10.0 introduces a lot more
>    changes are applied than the minor patch between 3.9 and 3.9.8;
>    therefore, we prefer to stabilize 3.9.8 or a later 3.9 kernel than
>    stabilizing 3.10.0. You can see this pattern in history; the
>    previous three have been 3.6.11, 3.7.10 and 3.8.13.
>
> Q: Can't you do something about those build and runtime failures?
>    I like to run testing, but I don't want breaks half of the time.
>
> A: To cover this, live ebuilds help, 3.10.9999 and 3.9999 and 9999;
>    the earlier ones tracking the stable queues, whereas the last one
>    tracking the upstream rc kernel. This way we can notice what fails
>    in advance and make it working by the time it reaches testing (~).
>
>    That being said, it is to be expected that things are not always
>    fine very early in a branch; it's why those kernels are never stable.
>
> Q: What about kernel bugs, how would you know the user enabled them?
>
> A: We expect the .config to be provided, which we can run a sanity
>    check on; it's much more handy if we are aware of these patches than
>    that we have to figure out the user applies it on their own.
>
> Q: Are documentation / infrastructure / eclass changes necessary?
>
> A: The kernel project documentation has to be updated to reflect this;
>    no infrastructure or eclass changes are needed, since this is all
>    done through patches in genpatches. (Sub directories supported afaik)
>
> Q: Does this affect kernel stabilization or QA?
>
> A: No, experimental features (which would include these optional
>    features) have historically never been looked at by the arch teams;
>    if you can remember, there used to be an CONFIG_EXPERIMENTAL option
>    in the kernel to cover this. For example, you can find yourself
>    running some pretty unstable code if you use 3.8.13 and enable
>    Nouveau reclocking and power management; there are some options in
>    the menuconfig that follow this nature.
>
>    As there would be less *-sources as a consequence; less has to be
>    taken into consideration for stabilization, this is why most
>    *-sources are currently unstable, because we don't have the
>    resources to be stabilizing them all.
>
>    For QA, I don't really see a problem.
>
> Q: So, would this make vanilla-sources the new gentoo-sources?
>
> A: No, vanilla-sources undergoes some changes to more closely reflect
>    the upstream kernels; as a consequence, it would no longer have
>    stable kernels and older versions will drop more often. You can read
>    this request and the resulting discussion at the following link.
>
>    http://thread.gmane.org/gmane.linux.gentoo.kernel/697
>
>    The kernel lead summarized a part of this.
>
>    http://thread.gmane.org/gmane.linux.gentoo.kernel/697/focus=730
>
> Q: Is this a one man army comedy show?
>
> A: One man army comedy shows work, see geek-sources; if you want to
>    look at some examples of what a one man army can do, the following
>    link is a good read to go through.
>
>    http://www.bennorthrop.com/Essays/2013/pair-programming-my-personal-nightmare.php
>
>    But joking aside; no, we are not.
>
>    We are at least with two on the kernel herd and a third developer is
>    likely to join in the near future; on top of that, other people are
>    welcome and I think it would be nice to see maintainers from
>    *-sources jump in to help along. genpatches isn't hard to maintain.
>
>    Since we're no longer a one man army, we can do more.
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJR0aw5AAoJEKXdFCfdEflKc+AQAKGa7v+beI5x6oBFoMCri4xM
pVX3jNdjtqidNg9Kyg5yMtoUl2WW6jJp8nIgd+gLeJ+8aNg1G/L94Jc4iRIHw8TH
Rv1oTafAa7Xc76ZxOvNHJMQNjTV+DtTsGuAKD0KAeEOJ6ld+wwS55D26NGXYAe0i
J1wPR7ndDHscBQQCGS+S4qAgHvfLaVq6wu7oQaGi9CTBTT1AtlmhkbBTH2qHTBJX
ZV98+hqiLBto6A42FmmZbqEooFWMGtcoJzlZNYCZduQ6uODPy30vt55dhh2OKsZW
5FCmUNdiekY8KRFWlYfcVFPAsrKVCutba/PEZunm9WwZ1Z9AoVz8WzBBLQU6b30P
vAzkUeTfET+y/NnKcuEO7j7CRXXm3+IjbNRPJ8qh5KiEiq1QfyxiOvZBQe08PhdP
i2Hxau6RhGeHVedyv9quImxBwNTdBgdJjuLlEP6cPBRMUb9HNqa0YNdsZ8CYu8I3
6rbL3pbUJmzhUUsbgV5nFdIZOlKWHOICzOuwKnR8T9OklWhI12gZUEePfYvk2Rl1
v5pYsse25O2FduNu24bwDI8qnUvtfEsqLpr9y5z0e94BX0FUR1wITb8Zp4KsKzo3
+R9nzQFqe9cF544Pt1Bp1ljpXEfXQl05aYEWEBFttkLSxHxMNtv2lHFh4C3Q9NNm
s4P30hArFvTeCo4OIfOd
=+mZ8
-----END PGP SIGNATURE-----