Re: [gentoo-dev] udev and /usr

[Robin H. Johnson]

On Thu, Sep 15, 2011 at 11:00:47PM +0200, Joost Roeleveld wrote:
> > See below on the existing udev retry queue that is hiding many of the
> > issues from you. This hidden issues are also negatively affecting boot
> > times (failures and retries take time).
> I don't actually mind too much about the boot time. If there are retries like
> this, I would expect this to be visible in the system logs.
udev does not log rule failures to the best of my knowledge.

> > The problem is that there is a bit of a catch-22 in running some udev
> > rules:
> > 0. You're going to have to declare interdependencies between ALL udev
> >    rules. This is because udev rules could be usable independently, or
> >    they could be interrelated (first rule sets some state variable or
> >    file, second one consumes it).
> Either udev does this already and the execution sequence is always the same.
> In which case my suggestion above would follow the same sequence as the queue
> would be on a First-in First-out basis.
> Or, if udev doesn't do this yet, udev will end up having the same problem.
It doesn't do it presently. The worst case I've seen is having an early
rule that succeeds, but gives different results w/ /var mounted vs. not
mounted. This rule didn't actual fail, so it does not get retried...

> > 1. While the binary invoked by a given rule might reside entirely on a
> >    mounting that is already available, how do you ensure that the other
> >    mountpoints required by said binary are ALSO available (the bluetooth
> >    and ALSA rules actually need /var, what if you have a bluetooth
> >    keyboard? [see footnote]).
> This is why I would suggest the "actiond" process to be started after
> localmount.
That's too late. What about all the udev rules required to get the
device nodes for localmount to succeed?

Anyway, take your proposed split actiond/udev solution to the upstream
udev list. I don't believe that we have the manpower to develop it here.
If we did develop it here, I don't believe it will gain enough traction
amongst other distros, and we'll be stuck supporting it.

I personally don't think your split solution covers the usage cases well
enough, but that's an actual decision best left to the upstream udev
developers. Please take the discussion there, and don't pursue it on
this list.

> > The upstream discussions I've been party to previously (both on lists
> > and in person), have been trying to avoid needing a full dependency
> > system in udev, because it's a huge degree of additional complexity.
> I don't see why it would not be possible to pause actioning of these scripts
> till the boot-process says all required mounts are available.
You still have to declare which scripts need to be paused, and/or which
rules inside the scripts need to be paused. Even worse are cases where
mounting some of localmount entries requires those scripts to have
completed.

> I see this as a "solution" for the situation where someone decides to use
> less-common hardware and force this solution onto everyone else.
I'm trying to get as little forced on us as possible. Gentoo is one of
the few distros at this point that doesn't already require initramfs.
While we're going to carry on supporting not requiring an initramfs as
long as possible (and documenting what is needed for that), we also
don't want this to become a stumbling block for users that just want
their system to work, and with how upstream udev and other projects are
going, there is a real possibility of breakage caused by their code, far
more than the potential of breakage because /usr failed to mount.

> If I would want to put my /usr filesystem on a bluetooth harddrive (for
> instance my mobile phone), then I would not expect to have this work without a
> lot of extra effort.
While that is in the realms of extreme, having /usr or /var on NFS isn't
extreme at all.

> > udev has a retry queue already, see udev-postmount:
> > ===
> > # Run the events that failed at first udev trigger
> > udevadm trigger --type=failed -v
> > ===
> This is a retry-queue. That's a good start already, but why not redo this
> queue and put ALL the scripts into that queue untill after localmount?
See above, about rules that are required for localmount to be able to
complete. The most prevalent ones would probably be devices by-uuid and
by-label. Those symlinks come from udev...

> > > With a smaller udev, the chances of it failing should also be less.
> > > (less
> > > code-lines to check) and as long as the /dev-entries are created, these
> > > can be used to manually mount the other partitions to get to the point
> > > where the system can be fixed to get it back to a workable state.
> >
> > The problem is NOT in the udev codebase. It's in udev rules. Even at the
> > rule level, it's mostly rules for packages other than udev itself.
>
> Yes, but as I already stated, the problem-rules do not exist on all systems.
> My systems for instance don't have any pointing to anything other then
> /etc/...
> These scripts also don't call anything that isn't mounted at the time.
Does your desktop use ALSA?
/lib/udev/rules.d/90-alsa-restore.rules 
invokes 
"/usr/sbin/alsactl restore ..."
Which in turn reads from /var/lib/alsa/asound.state.

We presently have the restore() function in /etc/init.d/alsasound that
repeats this, because that rule fails to work often during boot
(non-existence of the state file causes it to use built-in defaults
instead).

udev runs that rule as soon as the hardware turns up, which is often
before localmount.

> That system has been running without incident for several years. Why do I
> suddenly have to make that system more complex?
Just because there are no visible errors, doesn't mean that they don't
exist. This move to encourage initramfs is to ensure that there isn't
any major breakage incidents soon. What if udev upstream suddenely
starts hard requiring /usr to mounted, and not doing retries at all.
How many systems are going to break, and users going to complain about
needing to use livecds to recover?

>
> > > If, in the currently planned form, udev fails, it will be necessary to
> > > use a rescue-cd/usb to boot the system, try to fix it inside a chroot
> > > where it's not easy to check what is actually going wrong during the
> > > boot-process as the different tools can then not be run with the
> > > error-messages printed to the console.
> >
> > No, you're gotten the failure case wrong. Ok, so take the minimal
> > initramfs as I proposed on this list as the "working" case. Let's say
> > for some reason the initramfs doesn't load at all, so you have only /
> > mounted when you go into the rootfs init.
> >
> > If you had a setup that was complex enough to require udev to come up
> > for mounting /usr, you're going to end up at a real shell on your rootfs
> > by one of the following means:
> > - Pressing I for interactive boot, selecting shell (if you have not
> >   locked it down)
> > - Passing init=/bin/sh to your boot loader.
> >
> > The problem case that does NOT exist here is anything more complicated;
> > because if you have something like root-on-LVM, or encrypted root, you
> > already have an initramfs.
> >
> > If the initramfs itself does exist, but fails to mount anything, you
> > also get a rescue shell from the initramfs.
>
> >From my understanding, udev is needed to create the /dev-entries to be able to
> > mount /usr.
> If the changes proposed are actually done (moving everything out of / and into
> /usr) then udev won't be available to create the /dev-entries.
> A pre-populated set would work for most, but /dev/mapper used to require an
> initramfs as this device would have different numbers upon boot.
> If this is still the case, how would I be able to get LVM and MDADM to run to
> get to my partitions?
DEVTMPFS creates the first batch, and udev creates the rest.

The deciding case then becomes:
- Is the device for your /usr entry in fstab created by udev or
  something else?

MD: done by devtmpfs
LVM: done by udev+lvm
by-uuid/by-label: done by udev

by-uuid and by-label present a lot of annoyance to the minimal
initramfs. We have to ensure that we explicitly support them, which has
increased the complexity of the initramfs.

> I'm sorry, but I see bluetooth-keyboards still as a minority. If someone
> wants/has to use this, then an initramfs will be necessary.
...
> The vast majority doesn't use those.
Likewise, we're NOT going to force you to use an initramfs.
We're going to be providing it regardless. If the users choose not to
use the initramfs, they get to keep the broken pieces of their systems.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@g.o
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85