Re: Should /usr be merged with /? (Was: Re: [gentoo-user] Re: Anyone switched to eudev yet?)

[Bruce Hill]

On Sat, Dec 29, 2012 at 01:16:34AM +0800, Mark David Dumlao wrote:
> TLDR: FHS is unrealistic about its promises. if we move our binaries /
> libraries to /usr and work it to make sure /usr is mounted, we will
> better serve FHS goals and also happen to fix some systemic, but
> silent bugs.
>
>
> On Fri, Dec 28, 2012 at 12:20 PM, Michael Mol <mikemol@×××××.com> wrote:
> > On Thu, Dec 27, 2012 at 5:37 PM, Mark David Dumlao <madumlao@×××××.com>
> > wrote:
> >> Second, going back to problem solving in general - just because you
> >> can put down in words what you think the problem is, doesn't mean
> >> you've mapped out an accurate or even consistent statement of the
> >> problem. There really are cases where it's not enough to just give
> >> general airy abstractions and rules of thumb to map out a problem,
> >> where it isn't obvious that you're running into edge cases until you
> >> really look at it deeply, and yes, the / and /usr split is one of
> >> them.
> >
> > So let's look at it deeply, since nobody else will. I'm game. This is the
> > most detailed technical discussion of the problem anyone's cared to
> > actually have, as far as I've been able to observe.
>
> For the purposes of clarity I'm going to compare two competing
> standards, which I will be identifying as follows:
>
> s1) "FHS", or "plain FHS", based on FHS2.3, as identified in
> http://www.pathname.com/fhs/pub/fhs-2.3.html
> s2) "merged FHS", or "merged standard", based on FHS2.3 as above, but
> with the caveat that all binaries and libraries are placed in /usr
> instead of being split between /usr and /, as described by
> http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge
>
> It will be helpful to examine how each system reacts to strange cases
> that challenge FHS.
>
> I think some of the following considerations are helpful in
> determining which one works better. Whichever one is emphasized
> conspicuously depends on which systems you're interested in
> maintaining, how many people are using them, your personal taste,
> sense of justice, etc. Perhaps you could add some of your own.
> g1) Primary FHS purpose: software/users can predict location of
> installed files and directories
> g2) make distro maintainers' job easier
> g3) make sysads' job easier
> g4) it does not directly conflict with general practice
>
> It is my contention that in all goals, merged FHS is better than plain
> FHS. Secondly, it is also my contention that plain FHS with a separate
> /usr does not give enough information to reliably satisfy its own
> primary goal (g1). Back to the cases below.
>
>
>
> =========================================
> === FUNDAMENTAL PROBLEM: / and /usr desync ===
> =========================================
> Thesis: FHS promise of /usr being sharable is not really deliverable
> unless it contains the libraries in /.
>
> >>>> And the "we have a standard" part is effectively not true anymore, on
> >>>> the matter of the / and /usr split. That is - what the specification
> >>>> says should happen is not happening, on a massive scale, because it
> >>>> turns out that it's not that trivial to determine which binaries go in
> >>>> / and which go in /usr.
> >>>
> >>> Give me an example, and I'll describe a reasonably detailed solution.
> >>> It would be my pleasure.
> >>
> >> The most fundamental and relevant one for us Gentoo users is this:
> >> - how can /usr be sharable among different hosts if it depends on
> >> libraries in /?
> >>
> >> """
> >> http://www.pathname.com/fhs/pub/fhs-2.3.html#THEUSRHIERARCHY
> >>
> >> Purpose
> >>
> >> /usr is the second major section of the filesystem. /usr is shareable,
> >> read-only data. That means that /usr should be shareable between
> >> various FHS-compliant hosts and must not be written to. Any
> >> information that is host-specific or varies with time is stored
> >> elsewhere.
> >> """
> >>
> >> Many distros place fundamental libraries that many programs in /usr
> >> depend on in /lib. Especially bad for Gentoo - libraries in /lib may
> >> be recompiled as same-version variants if you want to change the USE
> >> flags, resulting in clients that don't synchronously recompile their
> >> own libraries in /lib to both silently and noisily fail.
> >>
> >> In other words, many programs in /usr in practice are functionally
> >> inseparable from the libraries in /, conflicting with the notion that
> >> they were properly shared in the first place.
> >
> > There are certain implicit assumptions made in the spec that are important.
> >
> > First, it's assumed the binaries are compatible with all the hosts. It's
> > assumed you're not sharing s360 binaries with x86 hosts, or sparc binaries
> > with ppc hosts.
> >
> > From there, it's reasonable to assume that the authors of the spec assume
> > the administrator to be smart enough to not do things like:
> > * Mix compiler versions
> > * Mix program compile options
> > * Place a dependency on a binary that's going to be missing.
> >
> > The spec is very, very much a "do what you want within these guidelines;
> > don't shoot yourself in the foot" thing, it's very much not a declarative
> > bikeshedding of everything related to it.
>
> Unfortunately, FHS actually does explicitly specify the meaning of shareable.
>
> http://www.pathname.com/fhs/pub/fhs-2.3.html#THEFILESYSTEM
>
> """"Shareable" files are those that can be stored on one host and used
> on others. "Unshareable" files are those that are not shareable. For
> example, the files in user home directories are shareable whereas
> device lock files are not."""
>
> Here I take the meaning "stored on one host and used on others", in
> the case of an executable, to mean "stored on one host and
> successfully executed on others".
>
> I think it is plainly obvious that binaries are not usable on other
> hosts unless they come with exact libraries they were compiled for. As
> far as the user is concerned, the library and the binary are two parts
> of the same thing, except that the library is stored in such a way
> that memory / disk space is minimized because it can be loaded
> multiple times.
>
> Following the above reasoning, to properly satisfy plain FHS, basic
> libraries such as readline, ncurses, gz, that various programs depend
> on should at minimum have _copies_ in /usr. But even with that
> satisfied, the client of the /usr share would still need to be
> configured to use those copies for the programs in /usr, possibly
> independently from its own programs in /.
>
> You say that the spec assumes that the sysad / distro will not "place
> a dependency on a binary that's going to be missing". Placing the
> library in /usr as well is the only way guarantee that for arbitrary
> /usr clients.
>
> So compare the following shared NFS /usr setups:
>
> plain FHS
> - some libraries are in /lib, with copies in /usr.
> - clients using the /usr share need to be configured to use the
> libraries in /usr over the ones in / (specifically only for the
> libraries in /usr)
> - distro maintainer has to determine whether library belongs in / or
> /usr. If it belongs in /, a copy will also be placed in /usr.
> - chance that sysad can accidentally unsync a client's / with /usr,
> resulting in arbitrary unpredictable failures (hard to tell which
> programs will silently segfault!)
>
> merged FHS
> - only one copy of the libraries, all in /usr
> - clients using the /usr share automatically use the libraries in /usr
> without reconfiguration
> - distro maintainer installs all libraries to one location only
> - sysad cannot desync / and /usr.
>
> > If I were doing a shared-/usr-over-NFS setup, here's how I'd do it:
> >
> > 1) Have two NFS shares for /usr, A and B.
> > 2) Have all my clients configured to draw from A.
> > 3) Perform my upgrade on B.
> > 4) Prepare my upgraded client image matching the data on B. Upgraded client
> > image would mount B for /usr.
> > 5) Replace my clients targeting A with clients targeting B.
> >
> > And tick-tock between A and B. Or proceed A->B->C, removing old shares once
> > they're no longer needed.
>
> First, the reason you need this update procedure is that you're
> compensating for the point of failure you're introducing with plain
> FHS: manually having to sync / and /usr. Which is functionally just
> one step of sharing /usr anyways. In other words, the fact of / and
> /usr being separated is what's giving you all this extra work.
>
> Second, since you're doing an image upgrade instead of a system
> upgrade, you're also complicating your process by having to either
> redo your host-specific files in /etc or being dependent on additional
> programs for reconfiguring each client based on mac address. Further
> opportunities for accidentally shooting yourself in the foot.
>
> >
> > And in any case, I wouldn't share /usr between images with different roles.
> > That way lies madness.
>
> It might be a weird setup, but it isn't any more invalid than wanting
> special performance characteristics out of having /usr separated from
> /.
>
> Consider, for instance, that a classroom of thin clients all shares
> the same architecture and compile farm, with some clients wanting to
> boot to a "photoshop" class and other clients wanting to boot to a
> "office productivity class". Students of each class would have
> effectively different roles that might be set in the / image, but
> there's no good reason for their software to have to be compiled
> twice.
>
>
>
> ==============================================
> === Where do udev rules belong? And the direction of udev ===
> ==============================================
> Thesis: udev/systemd recommends the /usr merge because it fixes
> subtle, silent bugs that nobody else wants to fix. We can have udev /
> systemd try to work around these bugs, but right now merging /usr is
> an available option.
>
> >>>> * some teasers:
> >>>> [1] udev rules themselves being a case in point. I mean, do the
> >>>> requisite binaries belong in /?
> >>>
> >>> Udev is a dispatcher. Actually, in substance, it's a piece of the
> >>> kernel that resides in userland; it exists because it was decided back
> >>> around the time of devfs that what devfs was doing is something that
> >>> ought to be outside the kernel. In reality, it's effectively been a
> >>> userland kernel-support process its entire life.
> >>>
> >>> What should probably happen is that udev should be fixed to defer
> >>> hotplug events until a rules file is able to sucessfully handle it.
> >>
> >> This is an interesting idea for a fix for the udev subproblem. However
> >> again note that the / and /usr split/merge is a bigger thing than the
> >> issues that just happen to manifest in udev. Perhaps right now they're
> >> giving up on it because they'd rather not waste time fixing a sub
> >> problem when fixing the / and /usr split fixes udev with less effort.
> >
> > When we went from x86 to x86_64, we didn't consider it the processor's fault
> > for badly-written packages misbehaving. As we went from 8-bit codepages to
> > UTF-8, we didn't consider it UTF-8's fault for badly-written packages
> > misbehaving. As we go from IPv4 to IPv6, most don't consider it IPv6's fault
> > when packages start misbehaving.
> >
> > We say the package needs to be fixed.
>
> """...when fixing the / and /usr split fixes udev with less effort."""
>
> >
> > So what on earth makes this different?
>
> I think an analogy with programming languages makes this clear. There
> are a lot of people that use terrible PHP security practices, and I'm
> probably one of them on bad days. That doesn't mean that PHP is buggy.
>
> Yes, the delivered software, in the end, still needs to be fixed.
>
> Now the PHP devs have the ability to "fix" those bugs by twiddling the
> default settings / behavior of PHP. Those things "fix" the bug by
> changing PHP or its settings. OR they can add language features that,
> if used correctly, will fix the bugs. Ditto udev. The udev guys can
> twiddle the default settings / behavior of it to "fix" the bugs in the
> udev rules. Or they can add udev logic that makes the bugs go away.
>
> Something _is_ special when you're talking about language features.
> How much software out there is still using deprecated features in PHP
> 4.x / 3.x? Heck, deprecated features in HTML4?
>
> It is a fact of life that "waiting for other people to implement a fix
> on their broken software", in general, doesn't work. If there's a big
> enough majority following some behavior, it's better to change the
> default behavior to fix the bug.
>
> Which the merged /usr already does. Yes, it'd be nice to have extra
> udev logic for detecting that rules can't execute yet. It isn't
> needed, though, if you merge / and /usr. So hats off to eudev for
> perhaps their most obvious coding target.
>
> > While I understand why *systemd*
> > should care about badly-written packages with cross-mount-point
> > dependencies, I don't understand why udev should, or why udev's direction so
> > be so *completely* subsumed by systemd's direction.
>
> Because it fixes the same bugs.
>
> >>
> >> On the other hand you can just edit the system so that the _default
> >> setting_ makes it unnecessary to specify dependencies for like 99.xx%
> >> of programs out there.
> >
> > And why is this a thing that needs to be done except on an as-needed basis?
>
> BECAUSE IT FIXES BUGS.
>
> >>>  systemd does
> >>> something interesting with its "After" clause; that makes perfect
> >>> sense. And that's why I asked Canek why one couldn't write a systemd
> >>> service file to treat /usr as a service
> >>
> >> Again, it's not enough to have just a passing understanding of the
> >> problem and talk airy abstractions about solutions. You really have to
> >> have an understanding of the problem space.
> >>
> >> systemd, for similar reasons to udev, is installed to /usr by default.
> >
> > Please, explain these reasons.
>
> systemd (or some of the stuff it starts) depends on some stuff in
> /usr. This makes their placement in /usr technically FHS bugs (for
> systemd systems) that the upstreams won't fix.
> - locales
> - some udev rules
> - udisks
> - SMART
> - PCI database
> - USB database
>
> (Actually some of those seem to kinda _really_ need to be in /
> following plain FHS)
>
> That it happens to work is just either systemd or the relevant
> subsystems happening to fail silently and gracefully.
>
> Placing systemd with all the other stuff fixes those silent failures
> in the same way that placing udev with all the other stuff fixes bugs
> in udev rules.
>
> Reference: (Warning: Lennartspeak)
> http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/1337
> http://article.gmane.org/gmane.comp.sysutils.systemd.devel/1350
> http://article.gmane.org/gmane.comp.sysutils.systemd.devel/1426
> http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken
>
> >
> > Almost everything useful the kernel does depends on the presence of an init.
> > Does that seriously challenge the assumption that init should be separate
> > from the kernel?
>
> Only if the kernel's design goal was to do those useful things itself
> (it isn't). One of systemd's design goals is to minimise boilerplate
> code in its unit files.
>
>
>
> =============================================
> === Hypothetical: does client software belong in / or /usr? ===
> =============================================
> Thesis: we cannot predict which programs are needed for mounting
> arbitrary filesystems, even though mounting "other filesystems" is
> required in the root filesystem in plain FHS.
>
> >>>> [2] fuse-based filesystems allow an administrator the crazy
> >>>> possibility of, for example, demanding that /home be an ssh
> >>>> connection. Should the ssh client belong in /? ftp? substitute any
> >>>> arbitrary client program.
> >>>
> >>> System dependent binaries and libraries aren't commonly placed in
> >>> /home. Your better argument would have been fuse-mounted /usr...in
> >>> which case it would be the administrator's responsibility to ensure
> >>> said arbitrary client program is present in /bin, and its libraries in
> >>> /lib.
> >>
> >> It's misleading to think that /home being in ssh is an issue, because
> >> the point is that the purpose of the root filesystem is:
> >
> > Wha? *You're* the one who brought up /home in the first place.
> >
> >>
> >> "To boot a system, enough must be present on the root partition to
> >> mount other filesystems. This includes utilities, configuration, boot
> >> loader information, and other essential start-up data."
> >>
> >> In other words, if /home is one of the "other filesystems" tools for
> >> mounting it should, according to FHS, be in the root filesystem.
> >
> > Why would /home be "one of the 'other filesystems' tools for mounting"? You
> > made a jump here I seriously don't follow. Are we talking credentials? That
> > kind of thing that's usually kept under /etc.
> >
>
> You are misreading the spec. FHS says that the root filesystem needs
> to be able to check and mount "other filesystems". This doesn't mean
> "the /usr filesystem or any place where executables lie", but "all
> other filesystems", including /home. If /home were mounted as a
> separate filesystem, then the root filesystem needs to contain
> whatever binaries are necessary to mount it, whatever filesystem type
> it is.
>
> It is following this logic that mount.fuse is expected to be found in /sbin.
>
> It is _contrary_ to the spec that any files mount.fuse depends on is
> found outside of the root filesystem.
>
> As you've misunderstood the spec, I'll be dropping your comments on
> this scenario being invalid.
>
> > I'm frankly unfamiliar with emacs-daemon, but (from what you say), it's
> > either broken, or was never, ever intended to be run as an init service.
> >
>
> I'll just drop this as it's not essential to the argument
> http://www.emacswiki.org/emacs/EmacsAsDaemon
>
>
> >>
> >> Now you say that it's alright, in this case, the sysad makes ssh
> >> available at /bin. But this undermines the primary FHS rationale: for
> >> binaries to be in _predictable_ locations for both the sysad AND the
> >> software packager.
> >
> > The sysad controls the package manager. Regardless of the distribution,
> > package placement should _always_ be done via the package manager. This is
> > why every package management system that exists is accompanied with a tool
> > for importing other package management systems' packages. Including binary
> > and source tarballs.
> >
>
> Following this, for any distro to correctly FHS, there needs to be a
> package manager switch to copy arbitrary packages (and dependent
> libraries) from /usr to /. As of yet not implemented.
>
> >> Compare: if all executables were in /usr FHS would have no problem
> >> locating the client binaries, because they're all in the same place.
> >
> > And defeats existing functional systems without valid purpose.
>
> What does it mean to "defeat existing functional systems"? Creating a
> single merged FHS system on your network will make your other machines
> explode?
>
> There is an upgrade path from plain FHS to merged FHS (copy and
> symlink). I'd know, because I've done it on my box.
>
>
> So compare the following setups with strange and unpredictable new
> FUSE filesystems:
>
> plain FHS
> - without knowledge of all fuse filesystems installed on the machine,
> the sysad cannot determine if a binary should be in / or /usr
> - binaries moved to / have the chance of confusing arbitrary sysads /
> maintainance scripts that hardcode path /usr/bin
> - package managers need a new unimplemented feature that tells
> programs to be moved to /bin
> - programs moved to / will also need _copies_ of requisite libraries
> in /, independent of libraries in /usr
>
> merged FHS
> - only one copy of the programs and libraries, all in /usr
> - sysads / maintainance scripts will always guess correctly by
> guessing /usr/bin/foo.
> - no need to change any package managers
>
>
>
> =============================================
> === Hypothetical: does server software belong in / or /usr? ===
> =============================================
> Thesis: Just as you cannot determine which client software is needed
> for mounting arbitrary filesystems, neither can you determine which
> server software will be the dependency of some filesystem.
>
> >>>> [3] a fuse-based filesystem depends on a local network service being
> >>>> started. For example, someone writes a crazy fuse mysql browser that
> >>>> also is coincidentally mounted at boot time. Should the mysqld service
> >>>> belong in /? ldap? substitute any arbitrary server program.
> >>>
> >>> And if an administrator decides to do this, it's his responsibility to
> >>> make sure mysqld is located in /bin, its libraries are in /lib...and
> >>> he's got to find some place other than /var for his database! By this
> >>> point, you've gone so far into reducto ad absurdum I honestly can't
> >>> imagine anyone apart from someone who has absolutely _no_ idea what
> >>> they're doing landing themselves in that situation.
> >>>
> >>
> >> More or less same as above. Since the admin is now manually moving
> >> things around, the spec _fails to achieve its goal_.
> >
> > The presumption is that the admin would use the tools available to him to
> > achieve his goals in a consistent fashion. We consider it an error for
> > packages to be manually installed into /usr/ as opposed to /usr/local. Why
> > would you think I wouldn't consider it an error for a package to be manually
> > installed into /?
> >
> > *Anything* an admin does without the knowledge of his package manager is his
> > own fault if it blows up in his face. This is why he is provided with a
> > package manager in the first place.
> >
>
> As above, the necessary action is unimplemented in any package
> managers I know of.
>
> As you've blown yourself into a tangent avoiding the fundamental
> question, this one goes more or less the same as the one with client
> software. But I find that you're very myopic in what you are going to
> allow in your Unix. Why, in a Plan9-like system it wouldn't be
> uncommon for daemons to serve up a number of filesystems, and if
> someone crazy enough makes a toy environment with FUSE it's not our
> place to stop them from following the standard.
>
> =====================================
> === Hypothetical: Should sysad tools be in /? ===
> =====================================
> Thesis: Because the root user's home directory is intended for the
> root user to place arbitrary emergency restore tools and information
> in, it is possible for the root user to depend on tools that depend on
> /usr being mounted. This creates an FHS violation for all those tools
> that does not exist in merged FHS.
>
> >>>> [4] /root (which is why it's separated from /home) contains docs and
> >>>> custom utilities used by root user for recovery. Unfortunately,
> >>>> there's a lot of perl scripts there specifically for doing filesystem
> >>>> checks / reports. Should perl be in in /? substitute any scripting
> >>>> language.
> >>>
> >>> You quoted FHS. I'll quote it back to you:
> >>>
> >>> "/usr, /opt, and /var are designed such that they may be located on
> >>> other partitions or filesystems."
> >>>
> >>> /root is ridiculously out of the question. /home isn't defended by the
> >>> spec, but it's commonly enough separated that it's difficult to
> >>> imagine someone making that error twice.
> >>
> >> /root is the home directory of the root user. If it's not available
> >> there, it defaults to the root directory (/). The point being the root
> >> user has its own storage that defaults to the root directory,
> >> independent of /home or whatnot.
> >>
> >> http://www.pathname.com/fhs/pub/fhs-2.3.html#FTN.AEN1037
> >>
> >> One good reason why it's separated from /home is because the root user
> >> may download or store his own stuff there relevant to fixing that
> >> machine. For example, post-install notes are often placed in /root,
> >> which detail which packages were selected upon installation. Or while
> >> performing lengthy system maintainance, the sysad may write down their
> >> upgrade notes, or snip relevant tcpdumps, or what-have-you and store
> >> them in the /root directory while the /home directory is unavailable.
> >
> > Of course. This is why /root as a separate filesystem is ridiculously out of
> > the question.
>
> I _didn't_ say /root was going to be a separate filesystem. I was
> _only_ describing /root for the purposes of clarifying why I expect it
> to have the following contents:
>
> >> It is very conceivable for the /root directory to contain perl scripts
> >> or whatever the sysad has downloaded in his adventures to grep through
> >> his logs and find out what the heck is going on in his system and/or
> >> repair it.
> >>
> >> Should perl be in / or /usr?
> >
> > Now that is a good question, if only because Perl traditionally _loathes_
> > being in /bin, for its own philosophical reasons.
> >
> > Now, as a practical matter? WTF are the scripts written in Perl? Or in
> > anything other than sh? If they're intended for emergency use, they've got
> > some pretty fat dependencies, and should probably be launched from a full
> > rescue environment instead.
>
> The fact that Perl has fat dependencies doesn't mean that it
> shouldn't, according to the spec, be placed in /. The fact is, in this
> case plain FHS suggests that Perl be in /. It's that ambiguous about
> it. This strongly conflicts with the history of Perl and general
> practice, in a way that suggests that we'd rather break plain FHS than
> follow the general practice (suggesting the plain FHS is in the
> wrong).
>
> > If you need a Cadillac for bootstrapping and emergency maintenance, sure.
> > But this is one of those scenarios that experience and good sense teaches
> > you to avoid.
>
> Doesn't help the fact that it still violates plain FHS.
>
>
>
> ====================================================
> === Hypothetical: Can we tell portage to install to / instead of /usr? ===
> ====================================================
> >>> With a system such as portage, it should be entirely possible (with
> >>> few code changes) to configure installation targets (/ vs /usr) on a
> >>> per-package basis, and have that trickle down the dependency chain.
> >>
> >> Yes, that should be. In fact I think that's the cleanest way to push
> >> through so far. Just add a USE=prefix-root to udev.
> >
> > Make it default. The change of default to /usr is what's ridiculously stupid
> > about the current scenario.
> >
> > And, frankly, if this is as much an issue as it's described to be, then
> > something beyond USE flags is necessary. A different per-package tunable is
> > needed.
>
> I don't think this is going to be trivial to implement, though. Any
> package moved to / is going to also need copies of any library
> dependencies to /. I only wonder how linking / revdep-rebuild is going
> to work.
>
>
>
> --
> This email is:    [ ] actionable   [x] fyi        [ ] social
> Response needed:  [ ] yes          [x] up to you  [ ] no
> Time-sensitive:   [ ] immediate    [ ] soon       [x] none

Dang, I got an Excedrin® headache!

No, it can't be fixed with NSAID, nor even morphine! It's simply and only able
to be fixed (not solved) by Excedrin® !

Now, after the big D, where's that "unsubscribe/ignore this thread" switch? :(
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@×××××××××××××××××××××.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting