[gentoo-soc] GSoC 2017: Atom Gentoo

[gentoo-soc] GSoC 2017: Atom Gentoo

[Huimin Zhang]

[-- Attachment #1: Type: text/plain, Size: 2360 bytes --]

Hi,

I am interested in doing the Atom Gentoo project. I’ve looked through the
past versions of the project idea (micro Gentoo, Stateless Minimal Gentoo)
as well, and this is how I think things could work:

Atom Gentoo will have a build script that will generate a disk image,
complete with bootloader, to be cloned to disks for deployment. I suspect
BIOS is a lot more prevalent than UEFI for servers and virtualization so a
BIOS bootloader will be default. The default kernel config should be
tailored for virtualization.

Each installation of Atom Gentoo consists of two separate partitions for
the root filesystem. Only one will be in use at a time, and the other will
be used for upgrades. If an upgrade fails, the partially upgraded partition
should be reset to be a copy of the other partition. If an upgrade
succeeds, the unupgraded partition should copy the contents of the upgraded
partition. A stateful partition, possibly on an external device, can be
overlayed on top.

The logic for choosing which partition to boot and maintaining the two
partitions will be handled in the init of initramfs, and the functionality
of generating such an initramfs could be patched into genkernel.

For updates, Atom Gentoo can rsync from a server that was defined at build
time. By using rsync the updates are diff-only instead of fetching the full
image each time. This can run automatically in the background as a cronjob,
updating the root partition not currently in use.

I have some questions as well (is there more background on the project?):


Should Atom Gentoo be minimal to the point of excluding portage and not
being self-sufficient? Since Atom Gentoo will receive full atomic OS
updates, it doesn’t really need to update its own packages, and perhaps
package management as a whole can be offloaded? This makes a lot of sense
for mass deployment.

For building Atom Gentoo, should the starting point be one of the Gentoo
Stage 3 tarballs or even earlier? I am leaning towards “even earlier”,
which will be necessary if portage is cut out of Atom Gentoo.

Should Atom Gentoo use a specific stdlib? I think uClibc and musl fit this
project well, but I haven’t worked with them before.

Also, since I expect to be writing a lot of Bash, does Gentoo have its own
Bash style guide?


Best,
Min

[-- Attachment #2: Type: text/html, Size: 6316 bytes --]

Re: [gentoo-soc] GSoC 2017: Atom Gentoo

[Robin H. Johnson]

On Thu, Mar 30, 2017 at 02:57:21AM -0700, Huimin Zhang wrote:
> Atom Gentoo will have a build script that will generate a disk image,
> complete with bootloader, to be cloned to disks for deployment. I suspect
> BIOS is a lot more prevalent than UEFI for servers and virtualization so a
> BIOS bootloader will be default. The default kernel config should be
> tailored for virtualization.
The image-building part will probably already be very similar to the
OpenStack diskimage-builder tool, you can possibly benefit from that.

> Each installation of Atom Gentoo consists of two separate partitions for
> the root filesystem. Only one will be in use at a time, and the other will
> be used for upgrades. If an upgrade fails, the partially upgraded partition
> should be reset to be a copy of the other partition. If an upgrade
> succeeds, the unupgraded partition should copy the contents of the upgraded
> partition. 
Why sticking to writable filesystems here? This could be treated via
block-level replication, with squashfs or read-only optimized
filesystems, and verified via dm-verity or similar, providing options
for verified boot.

I presume you were thinking of something like OverlayFS for the stateful
layer, but that could also be used for one or more application R/O
layers.

These ideas aren't new, look at how the OS is handled in Android &
ChromeOS, you'll see many of them together. Even Docker's container
build model is very similar to this.

> The logic for choosing which partition to boot and maintaining the two
> partitions will be handled in the init of initramfs, and the functionality
> of generating such an initramfs could be patched into genkernel.
As the genkernel author, there have been debates about replacing the
initramfs-generation portion with Dracut (which requires porting of some
of the existing initramfs functionality to Dracut) [this is probably a
GSOC project candidate as well, but it'll be a lot of work].

> For updates, Atom Gentoo can rsync from a server that was defined at build
> time. By using rsync the updates are diff-only instead of fetching the full
> image each time. This can run automatically in the background as a cronjob,
> updating the root partition not currently in use.
Since the start & final state can be known, you can pre-generate the
delta (see rsync --write-batch etc), or go even more efficent by looking
at block layer transfers (including bindiff of squashfs, zfs/btrfs send
& other models).

> Should Atom Gentoo be minimal to the point of excluding portage and not
> being self-sufficient? Since Atom Gentoo will receive full atomic OS
> updates, it doesn’t really need to update its own packages, and perhaps
> package management as a whole can be offloaded? This makes a lot of sense
> for mass deployment.
Putting the packages in a different layer could be valuable.

> For building Atom Gentoo, should the starting point be one of the Gentoo
> Stage 3 tarballs or even earlier? I am leaning towards “even earlier”,
> which will be necessary if portage is cut out of Atom Gentoo.
Move it to the packages layer maybe, or a "dev" layer.

> Should Atom Gentoo use a specific stdlib? I think uClibc and musl fit this
> project well, but I haven’t worked with them before.
Having multiple options shouldn't be a hurdle here ;-).

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Re: [gentoo-soc] GSoC 2017: Atom Gentoo

[Rich Freeman]

On Thu, Mar 30, 2017 at 2:54 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Thu, Mar 30, 2017 at 02:57:21AM -0700, Huimin Zhang wrote:
>
>> The logic for choosing which partition to boot and maintaining the two
>> partitions will be handled in the init of initramfs, and the functionality
>> of generating such an initramfs could be patched into genkernel.
> As the genkernel author, there have been debates about replacing the
> initramfs-generation portion with Dracut (which requires porting of some
> of the existing initramfs functionality to Dracut) [this is probably a
> GSOC project candidate as well, but it'll be a lot of work].

I'm not sure what genkernel functionality dracut is actually missing,
but you could certainly use dracut even without porting any additional
functionality to it.  Since dracut is highly modular it is probably a
good starting point.

If anybody does port genkernel functionality to dracut I suggest doing
it in coordination with upstream so that this isn't some kind of
Gentoo-specific patch (or more likely a set of modules).

-- 
Rich

Re: [gentoo-soc] GSoC 2017: Atom Gentoo

[Robin H. Johnson]

On Thu, Mar 30, 2017 at 08:39:55PM -0400, Rich Freeman wrote:
> > As the genkernel author, there have been debates about replacing the
> > initramfs-generation portion with Dracut (which requires porting of some
> > of the existing initramfs functionality to Dracut) [this is probably a
> > GSOC project candidate as well, but it'll be a lot of work].
> I'm not sure what genkernel functionality dracut is actually missing,
> but you could certainly use dracut even without porting any additional
> functionality to it.  Since dracut is highly modular it is probably a
> good starting point.
> 
> If anybody does port genkernel functionality to dracut I suggest doing
> it in coordination with upstream so that this isn't some kind of
> Gentoo-specific patch (or more likely a set of modules).
The objective here would be trying for 1:1 compatibility layer with
genkernel initramfs arguments & behavior, such that the upgrade can
detect "are all kernel options presently used supported by the new
version, or would the user have to modify their kernel boot params to
boot safely"

Some of the behavior might just be thin wrapping or reuse of dracut
behavior, eg:
- btrfs
- dmcrypt
- iscsi
- lvm
- mdadm
- multipath
- ssh (dropbear)
- zfs
- dmcrypt
- luks
- dmraid
- ataraid

Other parts might be unique and need to be ported still, eg:
- gentoo livedvd
- docache
- aufs
- do/no$HW
- swsusp
- tuxonice
- unionfs
- overlayfs
- fbsplash (I think there are two different implementations even)

Some functionality WILL be on the chopping block, because it's probably
sufficiently underutilized at this point to not be worth porting, eg:
- MIPS CD EFS partitions
- older compressed ISO formats
- non-libata IDE devices

As a GSOC project, this would probably start by making sure there is a
comprehensive list of genkernel initramfs functionality, and matching
parts to existing dracut functionality, and then prioritizing the
remaining pieces for those that need wrappers vs from-scratch vs
obsolete code.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Re: [gentoo-soc] GSoC 2017: Atom Gentoo

[Rich Freeman]

On Fri, Mar 31, 2017 at 2:58 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Thu, Mar 30, 2017 at 08:39:55PM -0400, Rich Freeman wrote:
>> > As the genkernel author, there have been debates about replacing the
>> > initramfs-generation portion with Dracut (which requires porting of some
>> > of the existing initramfs functionality to Dracut) [this is probably a
>> > GSOC project candidate as well, but it'll be a lot of work].
>> I'm not sure what genkernel functionality dracut is actually missing,
>> but you could certainly use dracut even without porting any additional
>> functionality to it.  Since dracut is highly modular it is probably a
>> good starting point.
>>
>> If anybody does port genkernel functionality to dracut I suggest doing
>> it in coordination with upstream so that this isn't some kind of
>> Gentoo-specific patch (or more likely a set of modules).
> The objective here would be trying for 1:1 compatibility layer with
> genkernel initramfs arguments & behavior, such that the upgrade can
> detect "are all kernel options presently used supported by the new
> version, or would the user have to modify their kernel boot params to
> boot safely"

Is there really any value in having argument/behavior-level
compatibility vs simply feature-level compatibility?  How many Gentoo
tools are actually integrated with genkernel today?  It seems like it
would make far more sense to just port those to dracut.

>
> Some of the behavior might just be thin wrapping or reuse of dracut
> behavior, eg:
> - btrfs
> - dmcrypt
> - iscsi
> - lvm
> - mdadm
> - multipath
> - ssh (dropbear)
> - zfs
> - dmcrypt
> - luks
> - dmraid
> - ataraid
>

If this stuff already works, why bother wrapping it?  Do we really
need alternative syntax on a popular cross-distro tool?

> Other parts might be unique and need to be ported still, eg:
> - gentoo livedvd
> - docache
> - aufs
> - do/no$HW
> - swsusp
> - tuxonice
> - unionfs
> - overlayfs
> - fbsplash (I think there are two different implementations even)

There is nothing wrong with creating dracut modules for these things,
and I'm sure that upstream would accept some of them.  However, I
don't really see any of this as a pre-req for Gentoo Atom.

>
> As a GSOC project, this would probably start by making sure there is a
> comprehensive list of genkernel initramfs functionality, and matching
> parts to existing dracut functionality, and then prioritizing the
> remaining pieces for those that need wrappers vs from-scratch vs
> obsolete code.
>

This might make sense as a GSOC project, but it seems more like a
dracut/Redhat/Fedora project than a Gentoo one.

-- 
Rich

Re: [gentoo-soc] genkernel+dracut GSOC project thread

[Robin H. Johnson]

On Fri, Mar 31, 2017 at 01:54:46PM -0400, Rich Freeman wrote:
> Is there really any value in having argument/behavior-level
> compatibility vs simply feature-level compatibility?  How many Gentoo
> tools are actually integrated with genkernel today?  It seems like it
> would make far more sense to just port those to dracut.
The argument-level functionality is specifically so that users can
simply upgrade genkernel WITHOUT having to modify their grub
configuration.

> > Some of the behavior might just be thin wrapping or reuse of dracut
> > behavior, eg:
...
> If this stuff already works, why bother wrapping it?  Do we really
> need alternative syntax on a popular cross-distro tool?
I'd say the great majority of the users with genkernel will be totally
fine with the defaults, and won't need to do anything.

It'll enable the right dracut modules/options, and print a deprecation
warning to encourage users to move (with a deadline for the old
functionality turning off).

The compat layer would NOT be suited for upstream inclusion.

> > Other parts might be unique and need to be ported still, eg:
...
> There is nothing wrong with creating dracut modules for these things,
> and I'm sure that upstream would accept some of them.  However, I
> don't really see any of this as a pre-req for Gentoo Atom.
As I said when I answered about Genkernel-Dracut, this is orthagonal to
Gentoo-Atom and enough work for it's OWN GSoC project. Just some of the
behavior of atom was going to have overlap with how genkernel initramfs
works and is further developed (the fallback root esp, I think dracut
has similar already).

> > As a GSOC project, this would probably start by making sure there is a
> > comprehensive list of genkernel initramfs functionality, and matching
> > parts to existing dracut functionality, and then prioritizing the
> > remaining pieces for those that need wrappers vs from-scratch vs
> > obsolete code.
> This might make sense as a GSOC project, but it seems more like a
> dracut/Redhat/Fedora project than a Gentoo one.
It's an overlap of several pieces:
- improve genkernel to correctly call dracut for generating the
  initramfs (genkernel still does the kernel config/build)
- improve dracut for all genkernel initramfs functionality not present
  in dracut (yay for drop in modules).
- improve livecd-tools (this is the package that gives the gentoo
  bootable install media lots of the magic)

To any potential GSoC students reading this thread about genkernel: I
will consider being a mentor for a very strong proposal, you're going to
have to show lots of research on the parts of genkernel initramfs/dracut
I haven't mentioned yet, possibly start with showing one each of the
ported pieces (the overlay filesystems maybe) and several thin wrapper
pieces (eg nolvm).

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Re: [gentoo-soc] genkernel+dracut GSOC project thread

[Rich Freeman]

On Fri, Mar 31, 2017 at 3:40 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Fri, Mar 31, 2017 at 01:54:46PM -0400, Rich Freeman wrote:
>> Is there really any value in having argument/behavior-level
>> compatibility vs simply feature-level compatibility?  How many Gentoo
>> tools are actually integrated with genkernel today?  It seems like it
>> would make far more sense to just port those to dracut.
> The argument-level functionality is specifically so that users can
> simply upgrade genkernel WITHOUT having to modify their grub
> configuration.

I don't really get the point honestly.

You seem to be suggesting a migration path of genkernel->dracut(with
deprecation warnings)->dracut(native).

Why not just add deprecation warnings to genkernel instead (no
development required) and let people migrate to dracut native as they
wish?  Either way they need to do the work to migrate, but this way
lets them start sooner and doesn't require somebody to re-invent the
wheel.

Adding support for additional filesystems/etc is of course useful
all-around and I'm sure upstream would accept that.  Of course, it
stands to reason that most of those modules don't exist because there
isn't much demand for them.  Still, it probably isn't a lot of work in
many cases to support them.

In any case, I won't prolong this thread any longer since if somebody
wants to make dracut look like genkernel they can of course do so.  It
just seems a bit contrary to the general principle of sticking to
upstream.  Sure, back in the day every distro rolled their own
initramfs and genkernel made perfect sense in that context.  As more
standardized alternatives come along I think it makes more sense for
Gentoo to adapt to them rather than trying to make those look like
Gentoo.  When gumiboot came along nobody suggested making a
Gentoo-specific patch so that it would take grub config files, and so
on.

But, I've beat that horse to death now...

-- 
Rich

Re: [gentoo-soc] genkernel+dracut GSOC project thread

[Robin H. Johnson]

On Fri, Mar 31, 2017 at 04:26:56PM -0400, Rich Freeman wrote:
> I don't really get the point honestly.
...
> Why not just add deprecation warnings to genkernel instead (no
> development required) and let people migrate to dracut native as they
> wish?  Either way they need to do the work to migrate, but this way
> lets them start sooner and doesn't require somebody to re-invent the
> wheel.
Genkernel does two separate but related tasks, and supports it on a lot
of different architectures (more architectures than dracut as well).
- configure & build & install kernel+modules
- build & install initramfs
Depending on your system, the outputs are either seperate or interleaved
in different ways.

It's a question of what the upgrade path becomes:

Options:
(current) 0. User runs genkernel, Genkernel builds the kernel & initramfs
1. User runs genkernel, it builds the kernel, wraps dracut to build initramfs
2. User runs Genkernel, it builds the kernel; User runs dracut manually to build initramfs
3. User $other builds the kernel; User runs dracut manually.

You're advocated we move to #2 or #3, whereas I want #1 to be usable,
because there are cases where #2/#3 can't work.

It's esp. crucial for cases where the initramfs is built into the
kernel: 1. build modules; 2. build initramfs w/ modules; 3. link
initramfs into kernel.
There also used to be systems that had the initramfs concatenated on the
end of the kernel, rather than linked in as binary data.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136