[gentoo-amd64] Re: /run not mounting on 3.7.10 kernel

[Duncan]

Daiajo Tibdixious posted on Tue, 30 Apr 2013 18:30:36 +1000 as excerpted:

> During the startup on 3.7.10 /run fails to mount with this error:
> mount: wrong fs type, bad option, bad superblock on tmpfs, missing
> codepage or helper program, or other error
>
> Googling shows many people getting this error, and its something to do
> with openrc and moving from /var/run to /run.
>
> What I can't understand is why I can boot 3.7.9 without any problems,
> while 3.7.10 bombs. I have DEVTMPFS enabled in the kernel.
> Obviously there is something else wrong with my 3.7.10 kernel but I just
> can't figure what.
>
> Is /run supposed to be a physical directory? I thought it was supposed
> to be in ram. I got
> lrwxrwxrwx 1 root root 4 Dec  5 14:13 /var/run -> /run
> drwxr-xr-x 11 root root 360 Apr 30 18:17 /run
>
> Before the last upgrade of openrc (to 0.11.8) my 3.7.10 kernel was
> working fine.

Well, 3.7 is several months ago history here, as I'm running mainline-
linus-git and just rebooted to 3.9.0 yesterday, and I'm running live-git 
openrc-9999 as well, but 0.11.8 is the only release in-tree, so I guess 
it can't be too outdated, tho it does date from early December (07, Pearl 
Harbor day).  But...

Looking at kernel.org, first thing I note is that 3.7.10 is EOL for 3.7, 
so you should be thinking about updating anyway...  But a definitely-non-
kernel-coder look at its changelog...

There's one tmpfs commit listed for 3.7.10, and no block-layer or similar 
commits that look like they might trigger it, so a first-guess is that 
it's that tmpfs commit (the mentioned mpol option is numa related):

commit 95558dce307f5ac203cdd15192b8d9f028c0b6c4
Author: Greg Thelen <gthelen@××××××.com>
Date:   Fri Feb 22 16:36:01 2013 -0800

    tmpfs: fix use-after-free of mempolicy object
    
    commit 5f00110f7273f9ff04ac69a5f85bb535a4fd0987 upstream.
    
    The tmpfs remount logic preserves filesystem mempolicy if the mpol=M
    option is not specified in the remount request.  A new policy can be
    specified if mpol=M is given.
    
    Before this patch remounting an mpol bound tmpfs without specifying
    mpol= mount option in the remount request would set the filesystem's
    mempolicy object to a freed mempolicy object.
    
    [snip the reproducer and panic, you can look it up if curious]
    
    Non-debug kernels will not crash immediately because referencing the
    dangling mpol will not cause a fault.  Instead the filesystem will
    reference a freed mempolicy object, which will cause unpredictable
    behavior.
    
    The problem boils down to a dropped mpol reference below if
    shmem_parse_options() does not allocate a new mpol:
    
        config = *sbinfo
        shmem_parse_options(data, &config, true)
        mpol_put(sbinfo->mpol)
        sbinfo->mpol = config.mpol  /* BUG: saves unreferenced mpol */
    
    This patch avoids the crash by not releasing the mempolicy if
    shmem_parse_options() doesn't create a new mpol.
    
    How far back does this issue go? I see it in both 2.6.36 and 3.3.  I
    did not look back further.
    
    Signed-off-by: Greg Thelen <gthelen@××××××.com>
    Acked-by: Hugh Dickins <hughd@××××××.com>
    Signed-off-by: Andrew Morton <akpm@××××××××××××××××.org>
    Signed-off-by: Linus Torvalds <torvalds@××××××××××××××××.org>
    Signed-off-by: Greg Kroah-Hartman <gregkh@×××××××××××××××.org>

FWIW, that "commit upstream" appears as v3.9-rc1~99^2~8 according to git 
name-rev.  ~N means Nth generation ancestor ^P refers to parent P, so we 
99 commits previous to 3.9-rc1, 8 commits previous to that on the second 
parent side.  Or in plainer language, the first (mainline) tagged version 
it appeared in was 3.9-rc1, with the commit obviously appearing in-tree 
before that but after 3.8.0.

Which is to say it's about 8 weeks old in mainline, appearing in the 
pre-3.9-rc1 commit window.

So... given that I'm running a git kernel anyway, my reaction here would 
be to try reverting that patch and seeing if that fixed it.  If not, I'd 
given that we know 3.7.9 was fine and 3.7.10 wasn't, it's a 100% 
reproducer for you, and there's a very limited number of commits between 
the two, a git bisect would be child's play.  (Well, for a child knowing 
git anyway...)

Meanwhile, are you running a NUMA system?  My first amd64 system was a 
dual socket Opteron, so NUMA, tho my current system isn't.  Do you mount 
/run using custom mount-options or just take the default openrc options?  
Here, I'm using the following (from fstab):

run /run tmpfs size=2m,nodev,nosuid,noexec,noauto,nr_inodes=4k 0 0

Oh, do you build tmpfs as a module or build it into the kernel 
(monolithic)?  If it's a module, maybe the kernel either can't find the 
tmpfs module for some reason, or is confused somehow about what to load?  
If so, could you try building in tmpfs and see if that changes things?

As to your question about what /run is supposed to be... what did you 
/expect/ ls to show?  It's a directory in the filesystem, so ls has to 
show a directory, regardless of whether it's memory or local-disk or 
network or ... that backs the filesystem.

There does have to be a physical /run directory on / to serve as a 
mountpoint, so the directory must exist as a physical directory on / 
before /run is mounted.

After mount, since tmpfs is still a filesystem, only in memory, it'll 
still /appear/ as a directory to ls, even tho it's in memory.  To see 
what it actually is, you can use df /run, which tells you what filesystem 
it's on, followed by grep <filesystem> /proc/mounts, to give you what the 
kernel thinks about that filesystem and its mount options.  Here:

$ ls -dl /run
drwxrwxrwt 7 root root 360 Apr 29 16:05 /run/

$ df /run
Filesystem      Size  Used Avail Use% Mounted on
run             2.0M  724K  1.3M  36% /run

$ grep run /proc/mounts
run /run tmpfs rw,nosuid,nodev,noexec,relatime,size=2048k,nr_inodes=4096 
0 0

(FWIW, there's another unrelated run entry that shows in my grep as well, 
the bind-mount for my chrooted named server.  I didn't post that.)

As you can see ls shows /run as a normal dir... as it should since it's a 
filesystem, that it happens to be a filesystem in memory doesn't matter 
to ls.  But df shows it as its own filesystem, and a grep of /proc/mounts 
tells me what filesystem type (tmpfs, so it's in memory because that's 
where tmpfs creates its filesystem) as well as the options the kernel 
used to mount that filesystem.

Meanwhile, another way to tackle the problem, since you know the openrc 
version where it shows up as well, would be to bisect openrc instead of 
the kernel.  That'd tell you exactly what openrc commit was the problem.  
If necessary you could do both bisects, getting an even better picture of 
what was triggering it, from both the kernel and openrc sides.  Of course 
that'd be easier if you knew which version of openrc you were running 
previously, hopefully 0.11.7.x, as that'd give you the least commits to 
bisect down.


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman