Gentoo Archives: gentoo-user

From: Mark David Dumlao <madumlao@×××××.com>
To: gentoo-user@l.g.o
Subject: Re: Should /usr be merged with /? (Was: Re: [gentoo-user] Re: Anyone switched to eudev yet?)
Date: Fri, 28 Dec 2012 17:18:26
1 TLDR: FHS is unrealistic about its promises. if we move our binaries /
2 libraries to /usr and work it to make sure /usr is mounted, we will
3 better serve FHS goals and also happen to fix some systemic, but
4 silent bugs.
7 On Fri, Dec 28, 2012 at 12:20 PM, Michael Mol <mikemol@×××××.com> wrote:
8 > On Thu, Dec 27, 2012 at 5:37 PM, Mark David Dumlao <madumlao@×××××.com>
9 > wrote:
10 >> Second, going back to problem solving in general - just because you
11 >> can put down in words what you think the problem is, doesn't mean
12 >> you've mapped out an accurate or even consistent statement of the
13 >> problem. There really are cases where it's not enough to just give
14 >> general airy abstractions and rules of thumb to map out a problem,
15 >> where it isn't obvious that you're running into edge cases until you
16 >> really look at it deeply, and yes, the / and /usr split is one of
17 >> them.
18 >
19 > So let's look at it deeply, since nobody else will. I'm game. This is the
20 > most detailed technical discussion of the problem anyone's cared to
21 > actually have, as far as I've been able to observe.
23 For the purposes of clarity I'm going to compare two competing
24 standards, which I will be identifying as follows:
26 s1) "FHS", or "plain FHS", based on FHS2.3, as identified in
28 s2) "merged FHS", or "merged standard", based on FHS2.3 as above, but
29 with the caveat that all binaries and libraries are placed in /usr
30 instead of being split between /usr and /, as described by
33 It will be helpful to examine how each system reacts to strange cases
34 that challenge FHS.
36 I think some of the following considerations are helpful in
37 determining which one works better. Whichever one is emphasized
38 conspicuously depends on which systems you're interested in
39 maintaining, how many people are using them, your personal taste,
40 sense of justice, etc. Perhaps you could add some of your own.
41 g1) Primary FHS purpose: software/users can predict location of
42 installed files and directories
43 g2) make distro maintainers' job easier
44 g3) make sysads' job easier
45 g4) it does not directly conflict with general practice
47 It is my contention that in all goals, merged FHS is better than plain
48 FHS. Secondly, it is also my contention that plain FHS with a separate
49 /usr does not give enough information to reliably satisfy its own
50 primary goal (g1). Back to the cases below.
54 =========================================
55 === FUNDAMENTAL PROBLEM: / and /usr desync ===
56 =========================================
57 Thesis: FHS promise of /usr being sharable is not really deliverable
58 unless it contains the libraries in /.
60 >>>> And the "we have a standard" part is effectively not true anymore, on
61 >>>> the matter of the / and /usr split. That is - what the specification
62 >>>> says should happen is not happening, on a massive scale, because it
63 >>>> turns out that it's not that trivial to determine which binaries go in
64 >>>> / and which go in /usr.
65 >>>
66 >>> Give me an example, and I'll describe a reasonably detailed solution.
67 >>> It would be my pleasure.
68 >>
69 >> The most fundamental and relevant one for us Gentoo users is this:
70 >> - how can /usr be sharable among different hosts if it depends on
71 >> libraries in /?
72 >>
73 >> """
74 >>
75 >>
76 >> Purpose
77 >>
78 >> /usr is the second major section of the filesystem. /usr is shareable,
79 >> read-only data. That means that /usr should be shareable between
80 >> various FHS-compliant hosts and must not be written to. Any
81 >> information that is host-specific or varies with time is stored
82 >> elsewhere.
83 >> """
84 >>
85 >> Many distros place fundamental libraries that many programs in /usr
86 >> depend on in /lib. Especially bad for Gentoo - libraries in /lib may
87 >> be recompiled as same-version variants if you want to change the USE
88 >> flags, resulting in clients that don't synchronously recompile their
89 >> own libraries in /lib to both silently and noisily fail.
90 >>
91 >> In other words, many programs in /usr in practice are functionally
92 >> inseparable from the libraries in /, conflicting with the notion that
93 >> they were properly shared in the first place.
94 >
95 > There are certain implicit assumptions made in the spec that are important.
96 >
97 > First, it's assumed the binaries are compatible with all the hosts. It's
98 > assumed you're not sharing s360 binaries with x86 hosts, or sparc binaries
99 > with ppc hosts.
100 >
101 > From there, it's reasonable to assume that the authors of the spec assume
102 > the administrator to be smart enough to not do things like:
103 > * Mix compiler versions
104 > * Mix program compile options
105 > * Place a dependency on a binary that's going to be missing.
106 >
107 > The spec is very, very much a "do what you want within these guidelines;
108 > don't shoot yourself in the foot" thing, it's very much not a declarative
109 > bikeshedding of everything related to it.
111 Unfortunately, FHS actually does explicitly specify the meaning of shareable.
115 """"Shareable" files are those that can be stored on one host and used
116 on others. "Unshareable" files are those that are not shareable. For
117 example, the files in user home directories are shareable whereas
118 device lock files are not."""
120 Here I take the meaning "stored on one host and used on others", in
121 the case of an executable, to mean "stored on one host and
122 successfully executed on others".
124 I think it is plainly obvious that binaries are not usable on other
125 hosts unless they come with exact libraries they were compiled for. As
126 far as the user is concerned, the library and the binary are two parts
127 of the same thing, except that the library is stored in such a way
128 that memory / disk space is minimized because it can be loaded
129 multiple times.
131 Following the above reasoning, to properly satisfy plain FHS, basic
132 libraries such as readline, ncurses, gz, that various programs depend
133 on should at minimum have _copies_ in /usr. But even with that
134 satisfied, the client of the /usr share would still need to be
135 configured to use those copies for the programs in /usr, possibly
136 independently from its own programs in /.
138 You say that the spec assumes that the sysad / distro will not "place
139 a dependency on a binary that's going to be missing". Placing the
140 library in /usr as well is the only way guarantee that for arbitrary
141 /usr clients.
143 So compare the following shared NFS /usr setups:
145 plain FHS
146 - some libraries are in /lib, with copies in /usr.
147 - clients using the /usr share need to be configured to use the
148 libraries in /usr over the ones in / (specifically only for the
149 libraries in /usr)
150 - distro maintainer has to determine whether library belongs in / or
151 /usr. If it belongs in /, a copy will also be placed in /usr.
152 - chance that sysad can accidentally unsync a client's / with /usr,
153 resulting in arbitrary unpredictable failures (hard to tell which
154 programs will silently segfault!)
156 merged FHS
157 - only one copy of the libraries, all in /usr
158 - clients using the /usr share automatically use the libraries in /usr
159 without reconfiguration
160 - distro maintainer installs all libraries to one location only
161 - sysad cannot desync / and /usr.
163 > If I were doing a shared-/usr-over-NFS setup, here's how I'd do it:
164 >
165 > 1) Have two NFS shares for /usr, A and B.
166 > 2) Have all my clients configured to draw from A.
167 > 3) Perform my upgrade on B.
168 > 4) Prepare my upgraded client image matching the data on B. Upgraded client
169 > image would mount B for /usr.
170 > 5) Replace my clients targeting A with clients targeting B.
171 >
172 > And tick-tock between A and B. Or proceed A->B->C, removing old shares once
173 > they're no longer needed.
175 First, the reason you need this update procedure is that you're
176 compensating for the point of failure you're introducing with plain
177 FHS: manually having to sync / and /usr. Which is functionally just
178 one step of sharing /usr anyways. In other words, the fact of / and
179 /usr being separated is what's giving you all this extra work.
181 Second, since you're doing an image upgrade instead of a system
182 upgrade, you're also complicating your process by having to either
183 redo your host-specific files in /etc or being dependent on additional
184 programs for reconfiguring each client based on mac address. Further
185 opportunities for accidentally shooting yourself in the foot.
187 >
188 > And in any case, I wouldn't share /usr between images with different roles.
189 > That way lies madness.
191 It might be a weird setup, but it isn't any more invalid than wanting
192 special performance characteristics out of having /usr separated from
193 /.
195 Consider, for instance, that a classroom of thin clients all shares
196 the same architecture and compile farm, with some clients wanting to
197 boot to a "photoshop" class and other clients wanting to boot to a
198 "office productivity class". Students of each class would have
199 effectively different roles that might be set in the / image, but
200 there's no good reason for their software to have to be compiled
201 twice.
205 ==============================================
206 === Where do udev rules belong? And the direction of udev ===
207 ==============================================
208 Thesis: udev/systemd recommends the /usr merge because it fixes
209 subtle, silent bugs that nobody else wants to fix. We can have udev /
210 systemd try to work around these bugs, but right now merging /usr is
211 an available option.
213 >>>> * some teasers:
214 >>>> [1] udev rules themselves being a case in point. I mean, do the
215 >>>> requisite binaries belong in /?
216 >>>
217 >>> Udev is a dispatcher. Actually, in substance, it's a piece of the
218 >>> kernel that resides in userland; it exists because it was decided back
219 >>> around the time of devfs that what devfs was doing is something that
220 >>> ought to be outside the kernel. In reality, it's effectively been a
221 >>> userland kernel-support process its entire life.
222 >>>
223 >>> What should probably happen is that udev should be fixed to defer
224 >>> hotplug events until a rules file is able to sucessfully handle it.
225 >>
226 >> This is an interesting idea for a fix for the udev subproblem. However
227 >> again note that the / and /usr split/merge is a bigger thing than the
228 >> issues that just happen to manifest in udev. Perhaps right now they're
229 >> giving up on it because they'd rather not waste time fixing a sub
230 >> problem when fixing the / and /usr split fixes udev with less effort.
231 >
232 > When we went from x86 to x86_64, we didn't consider it the processor's fault
233 > for badly-written packages misbehaving. As we went from 8-bit codepages to
234 > UTF-8, we didn't consider it UTF-8's fault for badly-written packages
235 > misbehaving. As we go from IPv4 to IPv6, most don't consider it IPv6's fault
236 > when packages start misbehaving.
237 >
238 > We say the package needs to be fixed.
240 """...when fixing the / and /usr split fixes udev with less effort."""
242 >
243 > So what on earth makes this different?
245 I think an analogy with programming languages makes this clear. There
246 are a lot of people that use terrible PHP security practices, and I'm
247 probably one of them on bad days. That doesn't mean that PHP is buggy.
249 Yes, the delivered software, in the end, still needs to be fixed.
251 Now the PHP devs have the ability to "fix" those bugs by twiddling the
252 default settings / behavior of PHP. Those things "fix" the bug by
253 changing PHP or its settings. OR they can add language features that,
254 if used correctly, will fix the bugs. Ditto udev. The udev guys can
255 twiddle the default settings / behavior of it to "fix" the bugs in the
256 udev rules. Or they can add udev logic that makes the bugs go away.
258 Something _is_ special when you're talking about language features.
259 How much software out there is still using deprecated features in PHP
260 4.x / 3.x? Heck, deprecated features in HTML4?
262 It is a fact of life that "waiting for other people to implement a fix
263 on their broken software", in general, doesn't work. If there's a big
264 enough majority following some behavior, it's better to change the
265 default behavior to fix the bug.
267 Which the merged /usr already does. Yes, it'd be nice to have extra
268 udev logic for detecting that rules can't execute yet. It isn't
269 needed, though, if you merge / and /usr. So hats off to eudev for
270 perhaps their most obvious coding target.
272 > While I understand why *systemd*
273 > should care about badly-written packages with cross-mount-point
274 > dependencies, I don't understand why udev should, or why udev's direction so
275 > be so *completely* subsumed by systemd's direction.
277 Because it fixes the same bugs.
279 >>
280 >> On the other hand you can just edit the system so that the _default
281 >> setting_ makes it unnecessary to specify dependencies for like 99.xx%
282 >> of programs out there.
283 >
284 > And why is this a thing that needs to be done except on an as-needed basis?
288 >>> systemd does
289 >>> something interesting with its "After" clause; that makes perfect
290 >>> sense. And that's why I asked Canek why one couldn't write a systemd
291 >>> service file to treat /usr as a service
292 >>
293 >> Again, it's not enough to have just a passing understanding of the
294 >> problem and talk airy abstractions about solutions. You really have to
295 >> have an understanding of the problem space.
296 >>
297 >> systemd, for similar reasons to udev, is installed to /usr by default.
298 >
299 > Please, explain these reasons.
301 systemd (or some of the stuff it starts) depends on some stuff in
302 /usr. This makes their placement in /usr technically FHS bugs (for
303 systemd systems) that the upstreams won't fix.
304 - locales
305 - some udev rules
306 - udisks
307 - SMART
308 - PCI database
309 - USB database
311 (Actually some of those seem to kinda _really_ need to be in /
312 following plain FHS)
314 That it happens to work is just either systemd or the relevant
315 subsystems happening to fail silently and gracefully.
317 Placing systemd with all the other stuff fixes those silent failures
318 in the same way that placing udev with all the other stuff fixes bugs
319 in udev rules.
321 Reference: (Warning: Lennartspeak)
327 >
328 > Almost everything useful the kernel does depends on the presence of an init.
329 > Does that seriously challenge the assumption that init should be separate
330 > from the kernel?
332 Only if the kernel's design goal was to do those useful things itself
333 (it isn't). One of systemd's design goals is to minimise boilerplate
334 code in its unit files.
338 =============================================
339 === Hypothetical: does client software belong in / or /usr? ===
340 =============================================
341 Thesis: we cannot predict which programs are needed for mounting
342 arbitrary filesystems, even though mounting "other filesystems" is
343 required in the root filesystem in plain FHS.
345 >>>> [2] fuse-based filesystems allow an administrator the crazy
346 >>>> possibility of, for example, demanding that /home be an ssh
347 >>>> connection. Should the ssh client belong in /? ftp? substitute any
348 >>>> arbitrary client program.
349 >>>
350 >>> System dependent binaries and libraries aren't commonly placed in
351 >>> /home. Your better argument would have been fuse-mounted /
352 >>> which case it would be the administrator's responsibility to ensure
353 >>> said arbitrary client program is present in /bin, and its libraries in
354 >>> /lib.
355 >>
356 >> It's misleading to think that /home being in ssh is an issue, because
357 >> the point is that the purpose of the root filesystem is:
358 >
359 > Wha? *You're* the one who brought up /home in the first place.
360 >
361 >>
362 >> "To boot a system, enough must be present on the root partition to
363 >> mount other filesystems. This includes utilities, configuration, boot
364 >> loader information, and other essential start-up data."
365 >>
366 >> In other words, if /home is one of the "other filesystems" tools for
367 >> mounting it should, according to FHS, be in the root filesystem.
368 >
369 > Why would /home be "one of the 'other filesystems' tools for mounting"? You
370 > made a jump here I seriously don't follow. Are we talking credentials? That
371 > kind of thing that's usually kept under /etc.
372 >
374 You are misreading the spec. FHS says that the root filesystem needs
375 to be able to check and mount "other filesystems". This doesn't mean
376 "the /usr filesystem or any place where executables lie", but "all
377 other filesystems", including /home. If /home were mounted as a
378 separate filesystem, then the root filesystem needs to contain
379 whatever binaries are necessary to mount it, whatever filesystem type
380 it is.
382 It is following this logic that mount.fuse is expected to be found in /sbin.
384 It is _contrary_ to the spec that any files mount.fuse depends on is
385 found outside of the root filesystem.
387 As you've misunderstood the spec, I'll be dropping your comments on
388 this scenario being invalid.
390 > I'm frankly unfamiliar with emacs-daemon, but (from what you say), it's
391 > either broken, or was never, ever intended to be run as an init service.
392 >
394 I'll just drop this as it's not essential to the argument
398 >>
399 >> Now you say that it's alright, in this case, the sysad makes ssh
400 >> available at /bin. But this undermines the primary FHS rationale: for
401 >> binaries to be in _predictable_ locations for both the sysad AND the
402 >> software packager.
403 >
404 > The sysad controls the package manager. Regardless of the distribution,
405 > package placement should _always_ be done via the package manager. This is
406 > why every package management system that exists is accompanied with a tool
407 > for importing other package management systems' packages. Including binary
408 > and source tarballs.
409 >
411 Following this, for any distro to correctly FHS, there needs to be a
412 package manager switch to copy arbitrary packages (and dependent
413 libraries) from /usr to /. As of yet not implemented.
415 >> Compare: if all executables were in /usr FHS would have no problem
416 >> locating the client binaries, because they're all in the same place.
417 >
418 > And defeats existing functional systems without valid purpose.
420 What does it mean to "defeat existing functional systems"? Creating a
421 single merged FHS system on your network will make your other machines
422 explode?
424 There is an upgrade path from plain FHS to merged FHS (copy and
425 symlink). I'd know, because I've done it on my box.
428 So compare the following setups with strange and unpredictable new
429 FUSE filesystems:
431 plain FHS
432 - without knowledge of all fuse filesystems installed on the machine,
433 the sysad cannot determine if a binary should be in / or /usr
434 - binaries moved to / have the chance of confusing arbitrary sysads /
435 maintainance scripts that hardcode path /usr/bin
436 - package managers need a new unimplemented feature that tells
437 programs to be moved to /bin
438 - programs moved to / will also need _copies_ of requisite libraries
439 in /, independent of libraries in /usr
441 merged FHS
442 - only one copy of the programs and libraries, all in /usr
443 - sysads / maintainance scripts will always guess correctly by
444 guessing /usr/bin/foo.
445 - no need to change any package managers
449 =============================================
450 === Hypothetical: does server software belong in / or /usr? ===
451 =============================================
452 Thesis: Just as you cannot determine which client software is needed
453 for mounting arbitrary filesystems, neither can you determine which
454 server software will be the dependency of some filesystem.
456 >>>> [3] a fuse-based filesystem depends on a local network service being
457 >>>> started. For example, someone writes a crazy fuse mysql browser that
458 >>>> also is coincidentally mounted at boot time. Should the mysqld service
459 >>>> belong in /? ldap? substitute any arbitrary server program.
460 >>>
461 >>> And if an administrator decides to do this, it's his responsibility to
462 >>> make sure mysqld is located in /bin, its libraries are in /lib...and
463 >>> he's got to find some place other than /var for his database! By this
464 >>> point, you've gone so far into reducto ad absurdum I honestly can't
465 >>> imagine anyone apart from someone who has absolutely _no_ idea what
466 >>> they're doing landing themselves in that situation.
467 >>>
468 >>
469 >> More or less same as above. Since the admin is now manually moving
470 >> things around, the spec _fails to achieve its goal_.
471 >
472 > The presumption is that the admin would use the tools available to him to
473 > achieve his goals in a consistent fashion. We consider it an error for
474 > packages to be manually installed into /usr/ as opposed to /usr/local. Why
475 > would you think I wouldn't consider it an error for a package to be manually
476 > installed into /?
477 >
478 > *Anything* an admin does without the knowledge of his package manager is his
479 > own fault if it blows up in his face. This is why he is provided with a
480 > package manager in the first place.
481 >
483 As above, the necessary action is unimplemented in any package
484 managers I know of.
486 As you've blown yourself into a tangent avoiding the fundamental
487 question, this one goes more or less the same as the one with client
488 software. But I find that you're very myopic in what you are going to
489 allow in your Unix. Why, in a Plan9-like system it wouldn't be
490 uncommon for daemons to serve up a number of filesystems, and if
491 someone crazy enough makes a toy environment with FUSE it's not our
492 place to stop them from following the standard.
494 =====================================
495 === Hypothetical: Should sysad tools be in /? ===
496 =====================================
497 Thesis: Because the root user's home directory is intended for the
498 root user to place arbitrary emergency restore tools and information
499 in, it is possible for the root user to depend on tools that depend on
500 /usr being mounted. This creates an FHS violation for all those tools
501 that does not exist in merged FHS.
503 >>>> [4] /root (which is why it's separated from /home) contains docs and
504 >>>> custom utilities used by root user for recovery. Unfortunately,
505 >>>> there's a lot of perl scripts there specifically for doing filesystem
506 >>>> checks / reports. Should perl be in in /? substitute any scripting
507 >>>> language.
508 >>>
509 >>> You quoted FHS. I'll quote it back to you:
510 >>>
511 >>> "/usr, /opt, and /var are designed such that they may be located on
512 >>> other partitions or filesystems."
513 >>>
514 >>> /root is ridiculously out of the question. /home isn't defended by the
515 >>> spec, but it's commonly enough separated that it's difficult to
516 >>> imagine someone making that error twice.
517 >>
518 >> /root is the home directory of the root user. If it's not available
519 >> there, it defaults to the root directory (/). The point being the root
520 >> user has its own storage that defaults to the root directory,
521 >> independent of /home or whatnot.
522 >>
523 >>
524 >>
525 >> One good reason why it's separated from /home is because the root user
526 >> may download or store his own stuff there relevant to fixing that
527 >> machine. For example, post-install notes are often placed in /root,
528 >> which detail which packages were selected upon installation. Or while
529 >> performing lengthy system maintainance, the sysad may write down their
530 >> upgrade notes, or snip relevant tcpdumps, or what-have-you and store
531 >> them in the /root directory while the /home directory is unavailable.
532 >
533 > Of course. This is why /root as a separate filesystem is ridiculously out of
534 > the question.
536 I _didn't_ say /root was going to be a separate filesystem. I was
537 _only_ describing /root for the purposes of clarifying why I expect it
538 to have the following contents:
540 >> It is very conceivable for the /root directory to contain perl scripts
541 >> or whatever the sysad has downloaded in his adventures to grep through
542 >> his logs and find out what the heck is going on in his system and/or
543 >> repair it.
544 >>
545 >> Should perl be in / or /usr?
546 >
547 > Now that is a good question, if only because Perl traditionally _loathes_
548 > being in /bin, for its own philosophical reasons.
549 >
550 > Now, as a practical matter? WTF are the scripts written in Perl? Or in
551 > anything other than sh? If they're intended for emergency use, they've got
552 > some pretty fat dependencies, and should probably be launched from a full
553 > rescue environment instead.
555 The fact that Perl has fat dependencies doesn't mean that it
556 shouldn't, according to the spec, be placed in /. The fact is, in this
557 case plain FHS suggests that Perl be in /. It's that ambiguous about
558 it. This strongly conflicts with the history of Perl and general
559 practice, in a way that suggests that we'd rather break plain FHS than
560 follow the general practice (suggesting the plain FHS is in the
561 wrong).
563 > If you need a Cadillac for bootstrapping and emergency maintenance, sure.
564 > But this is one of those scenarios that experience and good sense teaches
565 > you to avoid.
567 Doesn't help the fact that it still violates plain FHS.
571 ====================================================
572 === Hypothetical: Can we tell portage to install to / instead of /usr? ===
573 ====================================================
574 >>> With a system such as portage, it should be entirely possible (with
575 >>> few code changes) to configure installation targets (/ vs /usr) on a
576 >>> per-package basis, and have that trickle down the dependency chain.
577 >>
578 >> Yes, that should be. In fact I think that's the cleanest way to push
579 >> through so far. Just add a USE=prefix-root to udev.
580 >
581 > Make it default. The change of default to /usr is what's ridiculously stupid
582 > about the current scenario.
583 >
584 > And, frankly, if this is as much an issue as it's described to be, then
585 > something beyond USE flags is necessary. A different per-package tunable is
586 > needed.
588 I don't think this is going to be trivial to implement, though. Any
589 package moved to / is going to also need copies of any library
590 dependencies to /. I only wonder how linking / revdep-rebuild is going
591 to work.
595 --
596 This email is: [ ] actionable [ ] fyi [x] social
597 Response needed: [ ] yes [x] up to you [ ] no
598 Time-sensitive: [ ] immediate [ ] soon [x] none


Subject Author
Re: Should /usr be merged with /? (Was: Re: [gentoo-user] Re: Anyone switched to eudev yet?) Bruce Hill <daddy@×××××××××××××××××××××.com>
Re: Should /usr be merged with /? (Was: Re: [gentoo-user] Re: Anyone switched to eudev yet?) Kevin Chadwick <ma1l1ists@××××××××.uk>