Re: [gentoo-user] [SOLVED] Server fails to boot after update to 4.19.27-r1

[Dan Johansson]

On 2019-03-18 08:27, J. Roeleveld wrote:
> On March 17, 2019 5:04:13 PM UTC, Dan Johansson <dan@×××.nu> wrote:
>> On 12.03.19 12:16, J. Roeleveld wrote:
>>> On Monday, March 11, 2019 9:31:58 PM CET Dan Johansson wrote:
>>>> On 11.03.19 20:55, J. Roeleveld wrote:
>>>>> On March 10, 2019 1:24:14 PM UTC, Dan Johansson
>> <Dan.Johansson@×××.nu> wrote:
>>>>>> After updating a server from kernel-4.14.83 to 4.19.27-r1 (same
>> problem
>>>>>>
>>>>>> with 4.19.23) the server will not boot.
>>>>>>
>>>>>> Grub starts fine and I can select the new kernel.
>>>>>> The kernel starts booting and after mounting "/" and "/usr" (this
>> is a
>>>>>> server with a separate /usr") the boot-process hangs.
>>>>>>
>>>>>> Here are the last few lines displayed before it hangs:
>>>>>>>> Initializing root device...
>>>>>>>> Detected root: /dev/md127
>>>>>>>> Mounting /dev/md127 as root...
>>>>>>>> Detected fstype: ext4
>>>>>>>> Using mount fstype: ext4
>>>>>>>> Using mount opts: -o ro
>>>>>>>>
>>>>>>        7.6104971 EXT4-fs (md127): mounted filesystem with ordered
>> data
>>>>>>
>>>>>> mode.  Opts (null)
>>>>>>
>>>>>>        7.6572671 init (5708) used greatest stack depth: 13280
>> bytes left
>>>>>>>>
>>>>>>>> Mounting /dev/dm-O as /usr: mount -t ext4 -o
>> noatime,user_xattr,ro
>>>>>>
>>>>>> /deu/dm-O /newroot/usr
>>>>>>
>>>>>>        7.6909561 EXT4-fs (dm-0): INFO: recouery required on
>> readonly
>>>>>>
>>>>>> filesystem
>>>>>>
>>>>>>        7.6925551 EXT4-fs (dm-0): write access will be enabled
>> diming
>>>>>>
>>>>>> recouery
>>>>>>
>>>>>>        7.9169781 EXT4-fs (dm-0): recovery complete
>>>>>>        7.9223701 EXT4-fs (dm-0): mounted filesystem with ordered
>> data
>>>>>>
>>>>>> mode.  Opts: user_xattr
>>>>>>
>>>>>>       7.9233051 mount (5722) used greatest stack depth, 13000
>> bytes left
>>>>>>>>
>>>>>>>> /usr already mounted, skipping...
>>>>>>>> Booting (initramfs)
>>>>>>
>>>>>> sep-usr init: running user requested applet
>>>>>>
>>>>>>
>>>>>> As I said, the 4.14.83 kernel boots without problem with the same
>>>>>> configuration.
>>>>>>
>>>>>> Any suggestions?
>>>>>
>>>>> I updated my servers last weekend and all moved to 4.19.27, 2 use
>> ZFS for
>>>>> the filesystem, several are VMs on top of Xen. None had any issues.
>>>>>
>>>>> The messages you show make me think they are from an initrd, not
>> the
>>>>> actual kernel. I would investigate that first and make sure your
>> initrd
>>>>> is actually updated as well. Did you copy the text? Or did you
>> manage to
>>>>> grab the output somehow?
>>>>>
>>>>> Also, which init system and initrd are you using?
>>>>>
>>>>> --
>>>>> Joost
>>>>
>>>> The text was copied from a screenshot (IPMI-KVM).
>>>> I am using sys-apps/openrc with sys-fs/eudev and I use genkernel to
>>>> build the kernel and the initramfs.
>>>>
>>>> Yes, for me it also looks like it has to do with /ginit (busybox) or
>>>> /sbin/init (sys-apps/sysvinit) and not the kernel.
>>>>
>>>> I also have a bunch of other servers which all updated fine to
>> 4.19.??
>>>>
>>>> I tried the suggestion from Hasan to run "make synconfig" but that
>> did
>>>> not change any option in .config.
>>>>
>>>> I'll try to rebuild kernel/init/busybox/intel-microcode again next
>> weekend.
>>>
>>> Are you booting with the updated initramfs? Or perhaps still with the
>> initramfs belonging
>>> to an older kernel?
>>>
>>> Does the server respond to SSH after a while? It might simply be that
>> the login-prompt is
>>> not showing on the correct console.
>>
>> I was able to try a reboot today again, after rebuilding the kernel,
>> busybox, sysvinit & intel-microcode but it still hanged on "sep-usr
>> init: running user requested applet". When the server comes to this
>> point all disk-activities ceases. Even waiting for about 20min did not
>> change anything and the host did not even answer to ping.
>>
>> So, just as a test, I removed "init=/ginit" from the kernel-boot-line
>> and voila - the server booted again without problem :-)
>>
>> So there seems to be some difference on how pre 4.19 kernels and post
>> 4.19 kernels handles separate /usr installs.
>>
>> I am just glad it is solved.
>> Thanks for all suggestions.
>>
>> Regards,
>> Dan
>>
>>
>>
>>
>>
>> --
>> Dan Johansson
>> ***************************************************
>> This message is printed on 100% recycled electrons!
>> ***************************************************
>
> What is "ginit"?
>
> I use 2 types of initramfs.
> One I created myself which is really simple.
> The other is created using 'bliss-initramfs'.
>
> Neither of these require me to set a special init-boot option.
>
> I am guessing the boot fails when it tries to start 'ginit'.

/ginit is a init-wrapper gets installed from sys-apps/busybox if you 
have the use flag "sep-usr" enabled (this was setup years ago so I can 
not remember the URL to the Gentoo Wiki page describing the setup).
This was needed on this host before when I had no initramfs, after 
changing to an "initramfs-setup" this is no longer needed but it was 
still working until kernel > 4.19.

Regards
Dan