Re: [Bulk] Re: [gentoo-user] Re: Anyone switched to eudev yet?

[Michael Mol]

On Thu, Dec 27, 2012 at 5:37 PM, Mark David Dumlao <madumlao@×××××.com>
wrote:
> On Fri, Dec 28, 2012 at 4:59 AM, Michael Mol <mikemol@×××××.com> wrote:
>> On Thu, Dec 27, 2012 at 3:16 PM, Mark David Dumlao <madumlao@×××××.com>
wrote:
>>> On Fri, Dec 28, 2012 at 2:40 AM, Michael Mol <mikemol@×××××.com> wrote:
>>>>> or the fact that some udev programs tend to
>>>>> be located in /usr,
>>>>
>>>>
>>>> That's either a bug with those programs, or a need for architectural
>>>> improvements within udev. Both plausible answers.
>>>>
>>>
>>> The most obvious architectural improvement being: simply place udev
>>> where all its dependencies are and all those bugs turn to nothing.
>>> Which is what the udev guys did. And the part which seems to elude
>>> everyone is: it isn't even a limitation of the program. It can still
>>> be installed to /. Heck we could probably make a USE=root-prefix flag
>>> for udev that installs it to / instead of /usr.
>>
>> This came up today on Reddit. I think it's _highly_ relevant.
>>
>> http://www.runswift.ly/solving-bugs.html
>>
>> Moving into a full dependency on initr* for separate /usr is a 'fix',
>> not a solution.
>>
>
> This is where you stumble. It's not a fix. It's a WONTFIX. It's a
> "make a lot of noise so that something else gets fixed because this is
> outside of our domain and we're not going to be responsible for it as
> it wasnt our bug in the first place". And that something else happens
> to be the / and /usr split conflicting with the user programs.

I understand that. I made that point myself; that the Gentoo dev moved udev
into /usr to reduce the bug passing load on himself.

>
> If you give the squeaky wheel the grease - as in merge / and /usr -
> you apply the fix independently of udev, which was simply installed to
> the /usr prefix. THAT is a solution - one independent of udev and
> again, does not depend on initr*. You don't have to.

That's one solution, but the cure is worse than the disease.

>
>>>
>>>>
>>>>>
>>>>> or even just a solid detailed specification on the
>>>>> precise criteria for inclusion into /.
>>>>
>>>>
>>>> For anyone arguing that / and /usr should be separate, the answer to
this is
>>>> "that ought to be common sense."
>>>>
>>>> Since I'm not someone who knows all there is to know about the
software and
>>>> interactions thereof, the most I can say is:
>>>>
>>>> * / ought to contain all binaries, libraries and static data necessary
for
>>>> booting beyond the point where / is mounted, and any machine-specific
>>>> binaries, libraries and static data.
>>>> * /usr ought to contain all binaries, libraries and static data not
>>>> necessary for its own mount.
>>>>
>>>
>>> I'm sure you mean well, as did most of the architects of the past, but
>>> the reality is, this simplistic take on the problem misses out on some
>>> fundamental issues. Yes, you mention later that the spec just doesn't
>>> specify what happens in such and such case, and try to trivialize it
>>> into saying people think that specs should "be able to do their
>>> thinking for them". But unfortunately, specifying what happens is
>>> exactly what specs are for!
>>
>> Does the term "overspecification" mean anything to you? Specs cannot
>> and should not specify every possible conceivable related thing.
>
> Two things.
>
> First, I'm not saying that a spec should specify everything. I am
> saying, however, that there are specific cases that is within its
> domain to specify and that it should be specifying. And because those
> cases generate conflicts, the fact that they aren't is a bug.

The spec also doesn't say anything about /usr/lib vs /usr/lib32 vs
/usr/lib64, yet decisions about those can also cause conflicts. I suppose
you'd argue that that's also a bug.

I already gave you a pretty succinct definition of what I thought the
treatment of /usr should be. And you quoted FHS on the subject, which was
eerily similar to what I described. Now, further down, you actually raise
specific issues. Let's focus on those.

>
> Second, going back to problem solving in general - just because you
> can put down in words what you think the problem is, doesn't mean
> you've mapped out an accurate or even consistent statement of the
> problem. There really are cases where it's not enough to just give
> general airy abstractions and rules of thumb to map out a problem,
> where it isn't obvious that you're running into edge cases until you
> really look at it deeply, and yes, the / and /usr split is one of
> them.

So let's look at it deeply, since nobody else will. I'm game. This is the
most detailed technical discussion of the problem anyone's cared to
actually have, as far as I've been able to observe.

>
>>> And the "we have a standard" part is effectively not true anymore, on
>>> the matter of the / and /usr split. That is - what the specification
>>> says should happen is not happening, on a massive scale, because it
>>> turns out that it's not that trivial to determine which binaries go in
>>> / and which go in /usr.
>>
>> Give me an example, and I'll describe a reasonably detailed solution.
>> It would be my pleasure.
>
> The most fundamental and relevant one for us Gentoo users is this:
> - how can /usr be sharable among different hosts if it depends on
> libraries in /?
>
> """
> http://www.pathname.com/fhs/pub/fhs-2.3.html#THEUSRHIERARCHY
>
> Purpose
>
> /usr is the second major section of the filesystem. /usr is shareable,
> read-only data. That means that /usr should be shareable between
> various FHS-compliant hosts and must not be written to. Any
> information that is host-specific or varies with time is stored
> elsewhere.
> """
>
> Many distros place fundamental libraries that many programs in /usr
> depend on in /lib. Especially bad for Gentoo - libraries in /lib may
> be recompiled as same-version variants if you want to change the USE
> flags, resulting in clients that don't synchronously recompile their
> own libraries in /lib to both silently and noisily fail.
>
> In other words, many programs in /usr in practice are functionally
> inseparable from the libraries in /, conflicting with the notion that
> they were properly shared in the first place.

There are certain implicit assumptions made in the spec that are important.

First, it's assumed the binaries are compatible with all the hosts. It's
assumed you're not sharing s360 binaries with x86 hosts, or sparc binaries
with ppc hosts.

From there, it's reasonable to assume that the authors of the spec assume
the administrator to be smart enough to not do things like:
* Mix compiler versions
* Mix program compile options
* Place a dependency on a binary that's going to be missing.

The spec is very, very much a "do what you want within these guidelines;
don't shoot yourself in the foot" thing, it's very much not a declarative
bikeshedding of everything related to it.

>
> Compare this with the alternate spec that /usr contains both the
> programs and the libraries. In which case there is no possibility of
> the programs being out of sync with the libraries. From a maintainance
> perspective, this reduces the number of points of failure in upgrading
> the central NFS share.

If I were doing a shared-/usr-over-NFS setup, here's how I'd do it:

1) Have two NFS shares for /usr, A and B.
2) Have all my clients configured to draw from A.
3) Perform my upgrade on B.
4) Prepare my upgraded client image matching the data on B. Upgraded client
image would mount B for /usr.
5) Replace my clients targeting A with clients targeting B.

And tick-tock between A and B. Or proceed A->B->C, removing old shares once
they're no longer needed.

Either that, or do the whole thing flag-day style.

And in any case, I wouldn't share /usr between images with different roles.
That way lies madness.

>
> Please note, though, that the / vs /usr issue is a bigger thing after
> all than udev, and only happens to be a udev-centric discussion
> because it's one of the most obvious cases of the spec going awry.

It's a udev-centric discussion right now because the udev+systemd group
chose to take the position that the spec was at fault. I'm arguing with you
here to show why they're wrong.

>
>>> * some teasers:
>>> [1] udev rules themselves being a case in point. I mean, do the
>>> requisite binaries belong in /?
>>
>> Udev is a dispatcher. Actually, in substance, it's a piece of the
>> kernel that resides in userland; it exists because it was decided back
>> around the time of devfs that what devfs was doing is something that
>> ought to be outside the kernel. In reality, it's effectively been a
>> userland kernel-support process its entire life.
>>
>> What should probably happen is that udev should be fixed to defer
>> hotplug events until a rules file is able to sucessfully handle it.
>
> This is an interesting idea for a fix for the udev subproblem. However
> again note that the / and /usr split/merge is a bigger thing than the
> issues that just happen to manifest in udev. Perhaps right now they're
> giving up on it because they'd rather not waste time fixing a sub
> problem when fixing the / and /usr split fixes udev with less effort.

When we went from x86 to x86_64, we didn't consider it the processor's
fault for badly-written packages misbehaving. As we went from 8-bit
codepages to UTF-8, we didn't consider it UTF-8's fault for badly-written
packages misbehaving. As we go from IPv4 to IPv6, most don't consider it
IPv6's fault when packages start misbehaving.

We say the package needs to be fixed.

So what on earth makes this different? While I understand why *systemd*
should care about badly-written packages with cross-mount-point
dependencies, I don't understand why udev should, or why udev's direction
so be so *completely* subsumed by systemd's direction.

(For the record, I believe systemd likely cares about badly-written
packages because of its already extraordinarily sophisticated complex
behavior involving extremely high levels of concurrency that *will* make
debugging far more difficult. You'll need some amazing advancements in
concurrency-bug-detecting tools to cope with the kinds of debugging issues
systemd would make the *norm* for rolling distros like Gentoo, Arch or
Debian/unstable. Imagine trying to debug concurrency issues that crop up
because program A and program B have an unanticipated influence on each
other, and a race condition exists because they're launching in parallel.
The Linux scheduler and debugging APIs will need to add some means of
halting multiple processes at the same time, just to break into the mess!)

>
>> And rules files should perform sanity checking and feedback in the
>> form of "WTF? No, I can't do that yet. Wake me later."
>
> $ equery belongs udev/rules.d/|wc -l
> 36
>
> On just my system, 36 different packages and god knows how many
> different upstreams / maintainers write to udev/rules.d. And that's
> Gentoo. How about Fedora. How about Red Hat. How about Debian, Ubuntu
> and god knows how many packages and distros that nobody can guarantee
> is going to use entirely new syntax.

So? If a package is broken, *it should be fixed*. I keep pounding that
point over and over. Shoving crap into an initramfs, or shoving everything
into /usr, or merging / and /usr, doesn't _solve_ complexity problems, it
_hides_ it. That's the kind of problem I was trying to allude to with my
link to that "fix vs solutions" blog post.

>
> Face it - the udev guys are not going to be in control of whoever
> writes the rules.

I wouldn't expect them to. And I don't care; it's not *their*
responsibility. It's the responsibility of the package maintainer.

It seems like everybody is pointing at the udev folks as the people who
should implement a solution for other people's broken packages, and this
strikes me as bass-ackward.

> Yes, having wait / dependencies in udev might be a
> neat feature, but there's no guarantee they'll be implemented because
> they're not part of udev itself.

Perhaps they'll get implemented in eudev. If I had the time, I'd write up
the patches myself. The concepts aren't that hard.

>
> On the other hand you can just edit the system so that the _default
> setting_ makes it unnecessary to specify dependencies for like 99.xx%
> of programs out there.

And why is this a thing that needs to be done except on an as-needed basis?
The only answer I can come up with is that the systemd boot process is
going to be a royal pain to debug, given its high degree of concurrency.
Quite frankly, that's like issuing body armor to everyone in a factory
because someone got hit with a piece of broken glass from a beer bottle
crushed by a heavy piece of machinery; it's an overreaction that reduces
everyone's agility in response to something whose presence was already a
violation of policy and good sense.

>
>>  systemd does
>> something interesting with its "After" clause; that makes perfect
>> sense. And that's why I asked Canek why one couldn't write a systemd
>> service file to treat /usr as a service
>
> Again, it's not enough to have just a passing understanding of the
> problem and talk airy abstractions about solutions. You really have to
> have an understanding of the problem space.
>
> systemd, for similar reasons to udev, is installed to /usr by default.

Please, explain these reasons. I've described what I believe these reasons
to be, but the only reasons given to me so far are that "it makes a bunch
of other badly-written or badly-packages programs behave better." I hold
that that's a fundamental misplacement of responsibility.

> This means that systemd won't even be runnable if /usr isn't
> available, so it won't even be possible to treat /usr as a service. So
> what you're saying makes no sense as far as typical systemd
> installations are involved.
>
> Now, if you installed systemd to / instead of to /usr, then yes you
> could create a /usr service which everything else would be dependent
> on. But you'd conspicuously make almost everything useful it does
> dependent on /usr, challenging the assumption that it should be
> separate in the first place.

Almost everything useful the kernel does depends on the presence of an
init. Does that seriously challenge the assumption that init should be
separate from the kernel?

> You would, for example, force
> 90-something percent of your unit files to have an extra dependency
> line that smells like it came from boilerplate kitchen, one of the
> things systemd was supposed to be designed to minimise.

The problem with this argument is that it really does come down to a
tradeoff largely based in philosophy. It's a question of expressiveness vs
a combination of capability-of-completeness and flexibility.

And, frankly, if it matters, there are ways to improve the language. Every
dependency-aware package manager since the concept existed has had to
grapple with all the same problems. This is not a space where new work is
needed to come up with something useful, flexible and largely complete.
And, frankly, systemd is trying to reinvent it. It's the same as the cycle
of invention in everything in software; "A is ugly and complex. Let's write
B to be lean, mean and fast." "B isn't flexible enough to handle these use
cases we didn't anticipate, but need to support." "B is ugly and complex.
Let's write C to be lean, mean and fast."

(Heck, before you know it, init system configuration files will be written
in Common Lisp. :P )

>
>>> [2] fuse-based filesystems allow an administrator the crazy
>>> possibility of, for example, demanding that /home be an ssh
>>> connection. Should the ssh client belong in /? ftp? substitute any
>>> arbitrary client program.
>>
>> System dependent binaries and libraries aren't commonly placed in
>> /home. Your better argument would have been fuse-mounted /usr...in
>> which case it would be the administrator's responsibility to ensure
>> said arbitrary client program is present in /bin, and its libraries in
>> /lib.
>
> It's misleading to think that /home being in ssh is an issue, because
> the point is that the purpose of the root filesystem is:

Wha? *You're* the one who brought up /home in the first place.

>
> "To boot a system, enough must be present on the root partition to
> mount other filesystems. This includes utilities, configuration, boot
> loader information, and other essential start-up data."
>
> In other words, if /home is one of the "other filesystems" tools for
> mounting it should, according to FHS, be in the root filesystem.

Why would /home be "one of the 'other filesystems' tools for mounting"? You
made a jump here I seriously don't follow. Are we talking credentials? That
kind of thing that's usually kept under /etc.

> You're confusing that with the idea binaries are essential. And by the
> way, if you had emacs-daemon as an init service, it would depend on
> /home being mounted to be able to read the user's startup, or other
> similar things like that, so your system would not start up completely
> unless you had /home mounted.

I'm frankly unfamiliar with emacs-daemon, but (from what you say), it's
either broken, or was never, ever intended to be run as an init service.

>
> Now you say that it's alright, in this case, the sysad makes ssh
> available at /bin. But this undermines the primary FHS rationale: for
> binaries to be in _predictable_ locations for both the sysad AND the
> software packager.

The sysad controls the package manager. Regardless of the distribution,
package placement should _always_ be done via the package manager. This is
why every package management system that exists is accompanied with a tool
for importing other package management systems' packages. Including binary
and source tarballs.

>
> To implement this, then, as either the user / software packager /> distro
maintainer, I now have to do an intelligent check on /etc/fstab
> to determine that no filesystems are mounted on fuse.

This doesn't follow, since you haven't explained what you need /home for!
Credentials are typically kept under /etc.

> This demands, of
> course, a table of possible filesystem names and the corresponding
> client programs (currently not codified), plus that check to
> /etc/fstab. So the spec _fails to achieve its goal_ for systems where
> table of client-fuse mappings is not codified: practically all of
> them.

Irrelevant, because the scenario is invalid.

>
> Compare: if all executables were in /usr FHS would have no problem
> locating the client binaries, because they're all in the same place.

And defeats existing functional systems without valid purpose. (Where I
hold that presuming responsibility of packages' dependencies is somehow
udev/systemd's responsibility, rather than that of the package authors and
maintainers, is invalid.)

>
>>
>>> [3] a fuse-based filesystem depends on a local network service being
>>> started. For example, someone writes a crazy fuse mysql browser that
>>> also is coincidentally mounted at boot time. Should the mysqld service
>>> belong in /? ldap? substitute any arbitrary server program.
>>
>> I seriously cannot imagine anyone doing this except as a prank. The
>> only similar case I can think of would have the db server on a
>> separate machine.
>
> Well I did say he was crazy ;)
>
>>
>> And if an administrator decides to do this, it's his responsibility to
>> make sure mysqld is located in /bin, its libraries are in /lib...and
>> he's got to find some place other than /var for his database! By this
>> point, you've gone so far into reducto ad absurdum I honestly can't
>> imagine anyone apart from someone who has absolutely _no_ idea what
>> they're doing landing themselves in that situation.
>>
>
> More or less same as above. Since the admin is now manually moving
> things around, the spec _fails to achieve its goal_.

The presumption is that the admin would use the tools available to him to
achieve his goals in a consistent fashion. We consider it an error for
packages to be manually installed into /usr/ as opposed to /usr/local. Why
would you think I wouldn't consider it an error for a package to be
manually installed into /?

*Anything* an admin does without the knowledge of his package manager is
his own fault if it blows up in his face. This is why he is provided with a
package manager in the first place.

>
> Compare: if all executables were in /usr FHS would have no problem
> locating the server binaries, because they're all in the same place.

Hold off on the compares until we have something useful to compare to.

>
>>> [4] /root (which is why it's separated from /home) contains docs and
>>> custom utilities used by root user for recovery. Unfortunately,
>>> there's a lot of perl scripts there specifically for doing filesystem
>>> checks / reports. Should perl be in in /? substitute any scripting
>>> language.
>>
>> You quoted FHS. I'll quote it back to you:
>>
>> "/usr, /opt, and /var are designed such that they may be located on
>> other partitions or filesystems."
>>
>> /root is ridiculously out of the question. /home isn't defended by the
>> spec, but it's commonly enough separated that it's difficult to
>> imagine someone making that error twice.
>
> /root is the home directory of the root user. If it's not available
> there, it defaults to the root directory (/). The point being the root
> user has its own storage that defaults to the root directory,
> independent of /home or whatnot.
>
> http://www.pathname.com/fhs/pub/fhs-2.3.html#FTN.AEN1037
>
> One good reason why it's separated from /home is because the root user
> may download or store his own stuff there relevant to fixing that
> machine. For example, post-install notes are often placed in /root,
> which detail which packages were selected upon installation. Or while
> performing lengthy system maintainance, the sysad may write down their
> upgrade notes, or snip relevant tcpdumps, or what-have-you and store
> them in the /root directory while the /home directory is unavailable.

Of course. This is why /root as a separate filesystem is ridiculously out
of the question.

>
> It is very conceivable for the /root directory to contain perl scripts
> or whatever the sysad has downloaded in his adventures to grep through
> his logs and find out what the heck is going on in his system and/or
> repair it.
>
> Should perl be in / or /usr?

Now that is a good question, if only because Perl traditionally _loathes_
being in /bin, for its own philosophical reasons.

Now, as a practical matter? WTF are the scripts written in Perl? Or in
anything other than sh? If they're intended for emergency use, they've got
some pretty fat dependencies, and should probably be launched from a full
rescue environment instead. Or the log files should be copied to some place
with more featureful tools available.

>
> Again compare: if all executables were in /usr FHS would have no
> problem locating perl.

If you need a Cadillac for bootstrapping and emergency maintenance, sure.
But this is one of those scenarios that experience and good sense teaches
you to avoid. Once-in-a-blue-moon scripts are useful when they're
needed--until they blow up because they were written for a version of the
language that's incompatible with what's installed. That's why sh-fu is
such a vital skill for admins. sh is everywhere. Even where bash isn't.

So, I still disagree with your example scenario.

>
>> What's funny is the gratuitous misreading of the spec that exists, the
>> anticipation that it would explicitly state more than it does, the
>> weak technical arguments (so far) in favor of merging / and /usr, and
>> the dogmatic defense of the merge.
>
> You have to think the individual cases through ;)

I've responded.

>
>>
>> With a system such as portage, it should be entirely possible (with
>> few code changes) to configure installation targets (/ vs /usr) on a
>> per-package basis, and have that trickle down the dependency chain.
>
> Yes, that should be. In fact I think that's the cleanest way to push
> through so far. Just add a USE=prefix-root to udev.

Make it default. The change of default to /usr is what's ridiculously
stupid about the current scenario.

And, frankly, if this is as much an issue as it's described to be, then
something beyond USE flags is necessary. A different per-package tunable is
needed.

-- 
:wq