Re: [gentoo-embedded] Mixed arch chroot to optimise building in qemu

[Michael Jones]

On Wed, Sep 22, 2021 at 1:20 PM Ed W <lists@××××××××××.com> wrote:

> Hi all, traffic seems to have dropped off here significantly, but here goes
>
> I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64).
> So to keep things simple I
> have just been doing the whole thing using qemu up until now, by which I
> mean I have an arm stage 3
> somewhere, I chroot into it and then using userspace qemu binaries I just
> run my whole script to
> generate the target build from inside that chroot. This works but it's at
> least a 5x slowdown from
> native
>
> To optimise this I have tried
>
> - turning on the various compiler options for python (claimed to give a
> 30% improvement) + LTO/PGO.
> I don't notice any difference in the chroot - presume that the emulation
> overhead is dominant effect
>
> - tried compiling qemu with -O3 and LTO (claimed to be supported since
> 6.0). Doesn't give any
> noticeable different in performance of emerge
>
> - Added a static compiled amd64 /bin/bash to the chroot - now this does
> give a noticeable boost to
> compile and emerge speeds. (random benchmark went from 26s to 22s)
>
>
> So motivated by the last item I want to try and see how many native exes I
> can push into the chroot
> (since I'm running under usermode qemu! why not!). The obvious one is the
> compiler
>
> Now, I have a cross compiler built, but a) that's not static, so I would
> need to find a way to get
> native libc into the chroot, and b) I'm not clear how I would call it
> inside the chroot, could I
> just move a symlink to the other compiler into the path? How does it find
> things like libgcc*.so etc?
>
> Or perhaps this is easier than this? Can I just use some incantation in
> the same way that the
> crosscompiler must be working to build myself a straight gcc inside the
> chroot which is native arch
> and statically compiled? eg is it enough that assuming I can build gcc
> static, can I just do this
> from outside the chroot and overwrite the native:
>
>     ROOT=$PWD emerge -1v --nodeps gcc
>
>
> It seems to me that this should work at least for the gcc binaries, etc.
> However, I'm completely
> ignorant of whether I want things like the linker plugin in native arch or
> target arch? What about
> the libgcc*.so files? (They don't actually exist in my cross compiler
> directories, but they are
> linked in as dependencies in some binaries in target and exist in the
> native compiler dir)
>
> Hacker news had someone do this recently and I believe meego used to do
> something similar, so really
> just trying to work out the details for this on gentoo. Any thoughts?
>
> Thanks
>
> Ed W
>


It's not clear to me if you're building gentoo images, or just building
some application.

If you're building gentoo images, you might consider this project
https://github.com/GenPi64 , we'd love to work with you on the mixed arch
situation, since we suffer the same problem.