Re: [gentoo-user] [SOLVED] Server fails to boot after update to 4.19.27-r1

[Dan Johansson]

On 12.03.19 12:16, J. Roeleveld wrote:
> On Monday, March 11, 2019 9:31:58 PM CET Dan Johansson wrote:
>> On 11.03.19 20:55, J. Roeleveld wrote:
>>> On March 10, 2019 1:24:14 PM UTC, Dan Johansson <Dan.Johansson@×××.nu> wrote:
>>>> After updating a server from kernel-4.14.83 to 4.19.27-r1 (same problem
>>>>
>>>> with 4.19.23) the server will not boot.
>>>>
>>>> Grub starts fine and I can select the new kernel.
>>>> The kernel starts booting and after mounting "/" and "/usr" (this is a
>>>> server with a separate /usr") the boot-process hangs.
>>>>
>>>> Here are the last few lines displayed before it hangs:
>>>>>> Initializing root device...
>>>>>> Detected root: /dev/md127
>>>>>> Mounting /dev/md127 as root...
>>>>>> Detected fstype: ext4
>>>>>> Using mount fstype: ext4
>>>>>> Using mount opts: -o ro
>>>>>>
>>>>        7.6104971 EXT4-fs (md127): mounted filesystem with ordered data
>>>>
>>>> mode.  Opts (null)
>>>>
>>>>        7.6572671 init (5708) used greatest stack depth: 13280 bytes left
>>>>>>
>>>>>> Mounting /dev/dm-O as /usr: mount -t ext4 -o noatime,user_xattr,ro
>>>>
>>>> /deu/dm-O /newroot/usr
>>>>
>>>>        7.6909561 EXT4-fs (dm-0): INFO: recouery required on readonly
>>>>
>>>> filesystem
>>>>
>>>>        7.6925551 EXT4-fs (dm-0): write access will be enabled diming
>>>>
>>>> recouery
>>>>
>>>>        7.9169781 EXT4-fs (dm-0): recovery complete
>>>>        7.9223701 EXT4-fs (dm-0): mounted filesystem with ordered data
>>>>
>>>> mode.  Opts: user_xattr
>>>>
>>>>       7.9233051 mount (5722) used greatest stack depth, 13000 bytes left
>>>>>>
>>>>>> /usr already mounted, skipping...
>>>>>> Booting (initramfs)
>>>>
>>>> sep-usr init: running user requested applet
>>>>
>>>>
>>>> As I said, the 4.14.83 kernel boots without problem with the same
>>>> configuration.
>>>>
>>>> Any suggestions?
>>>
>>> I updated my servers last weekend and all moved to 4.19.27, 2 use ZFS for
>>> the filesystem, several are VMs on top of Xen. None had any issues.
>>>
>>> The messages you show make me think they are from an initrd, not the
>>> actual kernel. I would investigate that first and make sure your initrd
>>> is actually updated as well. Did you copy the text? Or did you manage to
>>> grab the output somehow?
>>>
>>> Also, which init system and initrd are you using?
>>>
>>> --
>>> Joost
>>
>> The text was copied from a screenshot (IPMI-KVM).
>> I am using sys-apps/openrc with sys-fs/eudev and I use genkernel to
>> build the kernel and the initramfs.
>>
>> Yes, for me it also looks like it has to do with /ginit (busybox) or
>> /sbin/init (sys-apps/sysvinit) and not the kernel.
>>
>> I also have a bunch of other servers which all updated fine to 4.19.??
>>
>> I tried the suggestion from Hasan to run "make synconfig" but that did
>> not change any option in .config.
>>
>> I'll try to rebuild kernel/init/busybox/intel-microcode again next weekend.
>
> Are you booting with the updated initramfs? Or perhaps still with the initramfs belonging
> to an older kernel?
>
> Does the server respond to SSH after a while? It might simply be that the login-prompt is
> not showing on the correct console.

I was able to try a reboot today again, after rebuilding the kernel, 
busybox, sysvinit & intel-microcode but it still hanged on "sep-usr 
init: running user requested applet". When the server comes to this 
point all disk-activities ceases. Even waiting for about 20min did not 
change anything and the host did not even answer to ping.

So, just as a test, I removed "init=/ginit" from the kernel-boot-line 
and voila - the server booted again without problem :-)

So there seems to be some difference on how pre 4.19 kernels and post 
4.19 kernels handles separate /usr installs.

I am just glad it is solved.
Thanks for all suggestions.

Regards,
Dan





-- 
Dan Johansson
***************************************************
This message is printed on 100% recycled electrons!
***************************************************