Re: [gentoo-dev] udev and /usr

[Joost Roeleveld]

On Thursday, September 15, 2011 10:18:27 PM Robin H. Johnson wrote:
> On Thu, Sep 15, 2011 at 11:00:47PM +0200, Joost Roeleveld wrote:
> > > See below on the existing udev retry queue that is hiding many of
> > > the
> > > issues from you. This hidden issues are also negatively affecting
> > > boot
> > > times (failures and retries take time).
> >
> > I don't actually mind too much about the boot time. If there are retries
> > like this, I would expect this to be visible in the system logs.
>
> udev does not log rule failures to the best of my knowledge.

In other words, it silently fails...
That is unfortunate.

> > Either udev does this already and the execution sequence is always the
> > same. In which case my suggestion above would follow the same sequence
> > as the queue would be on a First-in First-out basis.
> > Or, if udev doesn't do this yet, udev will end up having the same
> > problem.
> It doesn't do it presently. The worst case I've seen is having an early
> rule that succeeds, but gives different results w/ /var mounted vs. not
> mounted. This rule didn't actual fail, so it does not get retried...

And here is my main concern with this. The udev team don't list all the 
possible filesystems where things can go wrong. They only mention /usr.

> > > 1. While the binary invoked by a given rule might reside entirely on
> > > a
> > >
> > >    mounting that is already available, how do you ensure that the
> > >    other
> > >    mountpoints required by said binary are ALSO available (the
> > >    bluetooth and ALSA rules actually need /var, what if you have
> > >    a bluetooth keyboard? [see footnote]).
> >
> > This is why I would suggest the "actiond" process to be started after
> > localmount.
>
> That's too late. What about all the udev rules required to get the
> device nodes for localmount to succeed?

Shouldn't these already exist for currently working setups?

> Anyway, take your proposed split actiond/udev solution to the upstream
> udev list. I don't believe that we have the manpower to develop it here.
> If we did develop it here, I don't believe it will gain enough traction
> amongst other distros, and we'll be stuck supporting it.
>
> I personally don't think your split solution covers the usage cases well
> enough, but that's an actual decision best left to the upstream udev
> developers. Please take the discussion there, and don't pursue it on
> this list.

Ok.

> > > The upstream discussions I've been party to previously (both on
> > > lists
> > > and in person), have been trying to avoid needing a full dependency
> > > system in udev, because it's a huge degree of additional complexity.
> >
> > I don't see why it would not be possible to pause actioning of these
> > scripts till the boot-process says all required mounts are available.
>
> You still have to declare which scripts need to be paused, and/or which
> rules inside the scripts need to be paused. Even worse are cases where
> mounting some of localmount entries requires those scripts to have
> completed.

In other words, a dependency on the rules would be needed?

> > I see this as a "solution" for the situation where someone decides to
> > use
> > less-common hardware and force this solution onto everyone else.
>
> I'm trying to get as little forced on us as possible. Gentoo is one of
> the few distros at this point that doesn't already require initramfs.
> While we're going to carry on supporting not requiring an initramfs as
> long as possible (and documenting what is needed for that), we also
> don't want this to become a stumbling block for users that just want
> their system to work, and with how upstream udev and other projects are
> going, there is a real possibility of breakage caused by their code, far
> more than the potential of breakage because /usr failed to mount.

I agree with you on this one. That is also why I am trying to get a clear 
picture of all the possible alternatives.

> > If I would want to put my /usr filesystem on a bluetooth harddrive (for
> > instance my mobile phone), then I would not expect to have this work
> > without a lot of extra effort.
>
> While that is in the realms of extreme, having /usr or /var on NFS isn't
> extreme at all.

I agree, I just used this example to explain that it shouldn't be necessary to 
force an initramfs on all users just because there is a small group who wants 
to have an extreme setup.

> > > udev has a retry queue already, see udev-postmount:
> > > ===
> > > # Run the events that failed at first udev trigger
> > > udevadm trigger --type=failed -v
> > > ===
> >
> > This is a retry-queue. That's a good start already, but why not redo
> > this
> > queue and put ALL the scripts into that queue untill after localmount?
>
> See above, about rules that are required for localmount to be able to
> complete. The most prevalent ones would probably be devices by-uuid and
> by-label. Those symlinks come from udev...

These must also come from somewhere else as this also works in the initramfs 
stage. Which is, from what I gather, used to get to the stage where udev can 
run.

> > > > With a smaller udev, the chances of it failing should also be
> > > > less.
> > > > (less
> > > > code-lines to check) and as long as the /dev-entries are
> > > > created, these can be used to manually mount the other
> > > > partitions to get to the point where the system can be fixed to
> > > > get it back to a workable state.> >
> > > The problem is NOT in the udev codebase. It's in udev rules. Even at
> > > the rule level, it's mostly rules for packages other than udev
> > > itself.>
> > Yes, but as I already stated, the problem-rules do not exist on all
> > systems. My systems for instance don't have any pointing to anything
> > other then /etc/...
> > These scripts also don't call anything that isn't mounted at the time.
>
> Does your desktop use ALSA?
> /lib/udev/rules.d/90-alsa-restore.rules
> invokes
> "/usr/sbin/alsactl restore ..."
> Which in turn reads from /var/lib/alsa/asound.state.
>
> We presently have the restore() function in /etc/init.d/alsasound that
> repeats this, because that rule fails to work often during boot
> (non-existence of the state file causes it to use built-in defaults
> instead).
>
> udev runs that rule as soon as the hardware turns up, which is often
> before localmount.

I have doubts about having all these things handled by udev. As you said, 
there is an init-script that handles this. Is the ultimate goal to get rid of 
init-scripts and have everything done automagically?

> > That system has been running without incident for several years. Why do
> > I
> > suddenly have to make that system more complex?
>
> Just because there are no visible errors, doesn't mean that they don't
> exist. This move to encourage initramfs is to ensure that there isn't
> any major breakage incidents soon. What if udev upstream suddenely
> starts hard requiring /usr to mounted, and not doing retries at all.
> How many systems are going to break, and users going to complain about
> needing to use livecds to recover?

A lot. And those will be very vocal.
I have a few goals with this thread and one of them is to try to figure out 
how best to avoid users to get affected by this.
I am not a developer (I would like to try to do some programming), but I do 
have a lot of ideas. And due to the lack of information available to the 
users, I decided to check here.

I truly appreciate the time everyone is taking to try to answer the questions 
I have. I hope to be able to come to a clearer understanding of the whole 
thing.

> > If the changes proposed are actually done (moving everything out of /
> > and into /usr) then udev won't be available to create the /dev-entries.
> > A pre-populated set would work for most, but /dev/mapper used to require
> > an initramfs as this device would have different numbers upon boot. If
> > this is still the case, how would I be able to get LVM and MDADM to run
> > to get to my partitions?
>
> DEVTMPFS creates the first batch, and udev creates the rest.
>
> The deciding case then becomes:
> - Is the device for your /usr entry in fstab created by udev or
>   something else?
>
> MD: done by devtmpfs
> LVM: done by udev+lvm
> by-uuid/by-label: done by udev
>
> by-uuid and by-label present a lot of annoyance to the minimal
> initramfs. We have to ensure that we explicitly support them, which has
> increased the complexity of the initramfs.

My /usr is on LVM. That requires udev.

My understanding is:
- udev needs /usr to be mounted to work
- udev is needed to sort out LVM to get access to /usr

How does the initramfs handle this?
And why can't this be implemented in localmount?

--
Joost