Re: [gentoo-dev] separate /usr without initramfs

[Joshua Kinard]

On 10/25/2019 14:14, William Hubbs wrote:
> Hey all,
>
> I have been advised to bring this topic back to the list before taking
> any action, so here it is.
>
> First, I need to clarify what I'm *NOT* talking about.
>
> This discussion has nothing to do with whether or not you have the
> split-usr use flag turned on; all of us officially have that on because
> /bin, /lib* and /sbin are directories in the official Gentoo setup. In
> other words, I am *not* talking about forcing the /usr merge.
>
> Unfortunately, the concept of separate usr has gotten wrapped up in the
> split-usr use flag and doesn't have to be.  For the record, I mean something
> very specific when I say "separate usr". I am talking about the situation
> where /usr is a mount point separate from /, so in this thread, let's stick
> to "separate usr" for that situation. I am *not* even saying that using
> separate usr is wrong or unsupported. You can even run separate usr with
> split-usr turned off if you would like to do so.
>
> Now for the use case I want to talk about, and that is using separate
> /usr without using an initramfs to boot your system and pre-mount /usr.
>
> If you do this, many things are broken, and this is why the binary
> distros all use an initramfs if you do this. This configuration is also
> unsupported officially in Gentoo [1] [2], and it is not shown as the
> example setup in our handbook.
>
> I want to hear from people who have / and /usr on separate partitions
> and who are not using an initramfs.
>
> If you are in this group, I have a very specific question. Why aren't
> you using an initramfs?
>
> Thanks,
>
> William


I use this approach.  /usr is its own partition (usually /dev/sda6 in my
buildouts), and it mounts a few seconds after init is started.  I've been
doing it this way on multiple architectures (x86, amd64, sparc64, mips)
since pretty much 2003, when I first started experimenting w/ Gentoo and
later went on to become a developer.  None of my existing Gentoo installs
have *ever* needed an initramfs to boot with /usr on a separate partition.
Yes, there's a small bit in OpenRC that whines apparently, but I have not
seen any ill effects.

My hardware layout is fairly static.  amd64 box uses hardware RAID5
(LSI/3Ware 9750), and LILO as bootloader, and the SGI machines use MD-RAID
in either RAID1 or RAID5 mode, and they all netboot (manually).  Maybe it's
because I stick to very simplistic builds, that I've avoided problems.  ::
shrug ::

Why do I not like an initramfs, though?  Well, for one, it complicates the
kernel compiles (and it makes them bigger, something which is an issue on
the old SGI systems at times).  Two, it's another layer that I have to
maintain.  Three, it violates, in my mind, the simplicity of keeping the
kernel and userland separated (e.g., kernel does kernel-y things, userland
does userland-y things).

Put simply, the kernel's single purpose, as nothing more than a
hyper-complex while loop, is to get the hardware up into a usable state and
then hand off to userland, then sit and service userland's needs as called
upon.  The kernel should have all of the subsystems loaded into it necessary
to accomplish this task.  The fact that the userland, in the current
ill-conceived case, cannot get itself up and running simply because /usr is
on a yet-to-be-mounted partition is not a concern of the kernel's.  Thus,
the loading of an initramfs into the kernel to solve this issue is, in
principle, a violation (and a Cthulhu-awful hack).

Maybe I'm just a old codger who refuses to accept change.  I'm fine with
that description.  I like things to remain somewhat simple, and my view of
Linux, both kernel and userland, over the last few years is one of growing
dismay due to the constant introduction of subsystem layer atop subsystem
layer for very little gain.  How much longer until we need a kernel to boot
the kernel to mount the userland that mounts the userland (yo dawg)?

And there's that tiny, tiny issue of some kiddos from another, commercial,
distribution that just up and declared one day that the idea of
/usr-on-another-partition was "broken", and they then used their market
position to effectively force all other distributions to bow to their
belief.  I took, and still to this day, take, great umbrage at that.  I'll
probably never get over it.  Surely, if it works for me on multiple systems
*today*, it can't be broken, right?  It certainly was *not* broken back
then.  So why change it?  Why completely reinstall all of my systems because
non-Gentoo developers said something I did was wrong?  No, I don't play that
game.

Oh, and for a technical reason, I like setting mount flags on partitions
like /usr, /var, and /tmp for security.  It's really as simple as that.  All
of the fluff above is just the philosophical pudding that I use to justify
my obstinance.

---

That all said, I've fallen madly in love with ZFS.  I've got two network
appliances and an old laptop running FreeBSD 12.0-RELEASE with ZFS, and the
things I can do with ZFS eliminate the need to have multiple physical
partitions on the disks in those systems (outside of FreeBSD's needed boot
and swap partitions).  I get separate datasets mounted at /usr, /var, /tmp,
and quite a few others, and each can have its own mount properties set that
also achieve the security aims I cherish.  I can even replicate VM-like
snapshot capabilities where I can break part of the install with some dumb
idea (like testing 12.1-RC2 out last weekend), then switch boot environments
back to the working one and the system comes back up as if nothing happened.

So lately, I've been messing around w/ ZFS On Linux using a crappy old ~2012
2.5" 5400rpm drive in my dev box, attached to the LSI 9750.  And with ZFS,
that little drive *screams*.  It's virtually faster than the RAID5 array on
the other three 9750 ports running XFS.  If that drive was a heavy metal
singer, it'd take over Bruce Dickinson's day job.

As such, I am contemplating switching to ZFS Root soon on that machine, once
I work out some minor details (like can I still use LILO, because GRUB is
meh).  If that happens, well, that's one machine that won't ever have to
worry about the /usr-on-a-sep-partition mess again (even though the juvenile
logic behind the no-separate-/usr movement is completely wrong, but I
digress in a troll-like fashion).

Of course, this leaves my poor SGI systems in a questionable state.  ZFS is
completely reliant on checksums, and ARC needs gobs upon gobs of RAM.  So
that right there means ZFS might not be viable on those old systems (at
least w/ SHA checksumming -- fletcher4 might be fine).  And that's assuming
ZFS even compiles for a mips64 target and runs without turning a disk into a
large pool of entropic bit slop.

Well, we'll just cross that bridge when we get there...

-- 
Joshua Kinard
Gentoo/MIPS
kumba@g.o
rsa6144/5C63F4E3F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic