[gentoo-embedded] Mixed arch chroot to optimise building in qemu

[Ed W]

Hi all, traffic seems to have dropped off here significantly, but here goes

I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64). So to keep things simple I
have just been doing the whole thing using qemu up until now, by which I mean I have an arm stage 3
somewhere, I chroot into it and then using userspace qemu binaries I just run my whole script to
generate the target build from inside that chroot. This works but it's at least a 5x slowdown from
native

To optimise this I have tried

- turning on the various compiler options for python (claimed to give a 30% improvement) + LTO/PGO.
I don't notice any difference in the chroot - presume that the emulation overhead is dominant effect

- tried compiling qemu with -O3 and LTO (claimed to be supported since 6.0). Doesn't give any
noticeable different in performance of emerge

- Added a static compiled amd64 /bin/bash to the chroot - now this does give a noticeable boost to
compile and emerge speeds. (random benchmark went from 26s to 22s)


So motivated by the last item I want to try and see how many native exes I can push into the chroot
(since I'm running under usermode qemu! why not!). The obvious one is the compiler

Now, I have a cross compiler built, but a) that's not static, so I would need to find a way to get
native libc into the chroot, and b) I'm not clear how I would call it inside the chroot, could I
just move a symlink to the other compiler into the path? How does it find things like libgcc*.so etc?

Or perhaps this is easier than this? Can I just use some incantation in the same way that the
crosscompiler must be working to build myself a straight gcc inside the chroot which is native arch
and statically compiled? eg is it enough that assuming I can build gcc static, can I just do this
from outside the chroot and overwrite the native:

    ROOT=$PWD emerge -1v --nodeps gcc


It seems to me that this should work at least for the gcc binaries, etc. However, I'm completely
ignorant of whether I want things like the linker plugin in native arch or target arch? What about
the libgcc*.so files? (They don't actually exist in my cross compiler directories, but they are
linked in as dependencies in some binaries in target and exist in the native compiler dir)

Hacker news had someone do this recently and I believe meego used to do something similar, so really
just trying to work out the details for this on gentoo. Any thoughts?

Thanks

Ed W