Re: [gentoo-user] strange behaviour in quite special case

[Andrew Savchenko]

Hi,

On Thu, 24 Aug 2017 18:27:22 -0300 Francisco Ares wrote:
> Hi, All.
>
> This is a rather special case, so I don't expect much, but who knows?
>
> I've built a Gentoo x86-64 system for an embedded application.
>
> Just after a lot of updates, which I am unable to track, it stopped working
> as usual.
>
> There is the development system, fully loaded of a lot of packages used for
> development, and the production system, that don't need all of those.
>
> There is a line in both systems in /etc/iniitab responsible for auto-login
> the production system user and the programs we need running (in its
> ".bash_profile" and ".xinitrc"):
>
> c6:2345:respawn:/sbin/agetty -a production-user 38400 tty6 linux
>
> The development system starts a WindowMaker session, and the production
> system starts a program that controls the rest of the hardware of this
> embedded system, with an X11 graphical interface.  That runs normally when
> simulated at the development system.
>
> The development system runs smoothly.  The production system, after
> removing the files from undesirable packages and creating a squashfs image
> of the ripped-off root partition behaves strangely at boot:
>
> It shows the initialization messages as expected, but when the auto-login
> and the controller program start should take place, it completely stalls up
> to I plug a USB keyboard and issue some times some of the key combinations
> to change to a text console and back to X11 (Ctrl-Alt-F1 and Ctrl-Alt-F6);
>  only then the things resume as expected.
>
> As you might suspect, there is no keyboard for the production system ;-) .
>
> As a matter of fact, I don't know where the stall take place, as when I try
> to switch to a text console to see the logs, it switches back to X11 and
> starts our program.  By the way, the logs just show that the events
> occurred at latter times than expected.
>
> Although the squashfs is read-only, some main directories are arranged in a
> way that, using tmpfs mounts and unionfs with the read-only directory to
> the read-write tmpfs directory to that main directory provide a way of
> creating temporary files that has been working for a few years now.
>
> For instance, in "/etc/fstab":
>
> tmpfs           /.etc.rw        tmpfs           defaults,mode=755
>                                         0 0
> union           /etc            unionfs
>         default_permissions,allow_other,use_ino,nonempty,suid,cow,dirs=/.
> etc.rw=rw:/.etc.ro=ro          0 0
>
> And there is a "/.etc.ro" with a copy of all files present in regular
> "/etc" , a "/.etc.rw" directory to be mounted tmpfs, and the original
> "/etc" directory, that needs to be there at boot, even before mounting all
> this.
>
> Does anyone have a clue?

Try to dissect your problem. Start with removing squashfs and all
tmpfs/unionfs manipulations. Create the same image, but on "normal"
writable file system and see how it goes. It may be fs-related bug,
may be you removed too many files and some "undesired" packages are
actually mandatory.

If you have some form on snapshots of your changes, you can try to
bisect them in a git bisect way.

Another approach is to run X server (or any other app suspected as
a troublemaker) under strace (or attach strace to a running process)
and see what is going on. You will have a lot of low level
information and extensive filtering will be required; strace is
capable of that, but you will need to dig into its documentation.

Best regards,
Andrew Savchenko