Re: [gentoo-embedded] Mixed arch chroot to optimise building in qemu

[Michael Jones]

On Wed, Sep 22, 2021, 14:54 Ed W <lists@××××××××××.com> wrote:

> On 22/09/2021 20:26, Michael Jones wrote:
>
>
>
> On Wed, Sep 22, 2021 at 1:20 PM Ed W <lists@××××××××××.com> wrote:
>
>> Hi all, traffic seems to have dropped off here significantly, but here
>> goes
>>
>> I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64).
>> So to keep things simple I
>> have just been doing the whole thing using qemu up until now, by which I
>> mean I have an arm stage 3
>> somewhere, I chroot into it and then using userspace qemu binaries I just
>> run my whole script to
>> generate the target build from inside that chroot. This works but it's at
>> least a 5x slowdown from
>> native
>>
>> To optimise this I have tried
>>
>> - turning on the various compiler options for python (claimed to give a
>> 30% improvement) + LTO/PGO.
>> I don't notice any difference in the chroot - presume that the emulation
>> overhead is dominant effect
>>
>> - tried compiling qemu with -O3 and LTO (claimed to be supported since
>> 6.0). Doesn't give any
>> noticeable different in performance of emerge
>>
>> - Added a static compiled amd64 /bin/bash to the chroot - now this does
>> give a noticeable boost to
>> compile and emerge speeds. (random benchmark went from 26s to 22s)
>>
>>
>> So motivated by the last item I want to try and see how many native exes
>> I can push into the chroot
>> (since I'm running under usermode qemu! why not!). The obvious one is the
>> compiler
>>
>> Now, I have a cross compiler built, but a) that's not static, so I would
>> need to find a way to get
>> native libc into the chroot, and b) I'm not clear how I would call it
>> inside the chroot, could I
>> just move a symlink to the other compiler into the path? How does it find
>> things like libgcc*.so etc?
>>
>> Or perhaps this is easier than this? Can I just use some incantation in
>> the same way that the
>> crosscompiler must be working to build myself a straight gcc inside the
>> chroot which is native arch
>> and statically compiled? eg is it enough that assuming I can build gcc
>> static, can I just do this
>> from outside the chroot and overwrite the native:
>>
>>     ROOT=$PWD emerge -1v --nodeps gcc
>>
>>
>> It seems to me that this should work at least for the gcc binaries, etc.
>> However, I'm completely
>> ignorant of whether I want things like the linker plugin in native arch
>> or target arch? What about
>> the libgcc*.so files? (They don't actually exist in my cross compiler
>> directories, but they are
>> linked in as dependencies in some binaries in target and exist in the
>> native compiler dir)
>>
>> Hacker news had someone do this recently and I believe meego used to do
>> something similar, so really
>> just trying to work out the details for this on gentoo. Any thoughts?
>>
>> Thanks
>>
>> Ed W
>>
>
>
> It's not clear to me if you're building gentoo images, or just building
> some application.
>
> If you're building gentoo images, you might consider this project
> https://github.com/GenPi64 , we'd love to work with you on the mixed arch
> situation, since we suffer the same problem.
>
>
> These are whole gentoo images. :-)
>
> So it's nothing special, but something like I drop into the arm chroot,
> then there is a whole pile of something like:
>
>     ROOT=/mnt/new_image emerge $stuff
>
> And at the end of all of that you have a shiny image to boot from (on an
> imx based SOM as it happens).
>
> Nice thing about this approach is that I need to build the same system for
> i386, amd64 and 32bit arm, and basically it means only running the same
> build script in each individual chroot, so it's quite nice not needing to
> fixup stuff for each platform.
>
>
> There are arm64bit boxes you can rent from AWS and similar, but we see a
> few build oddities on this which still need fixing and at least as near as
> I can see they are still quite a bit slower than using an intel processor
> in native mode.
>
>
> I'm just about to (re) try using distcc, which basically achieves the
> required end goal, so that I can measure performance. So something like run
> up a side by side chroot using crossdev, then fire up distcc in there and
> talk to it from your arm chroot. This gives less speedup than you would
> like because it needs quite a lot of work on the arm qemu side and
> serialising stuff, etc. Also linking etc is still on the arm side.
>
> I think the replacing of the bash binary with a native static binary is
> giving a decent speedup. I'm about to try swapping in pypy to see how that
> behaves.
>
> However, there is no doubt that getting the native cross compiler into the
> chroot is the solution, but there are more than a few challenges here, such
> as how to get it statically compiled and how to insert some or all of it
> into the arm chroot.
>
> See here for inspiration and I guess also the meego stuff from history:
>
>     https://news.ycombinator.com/item?id=28376447
>
>
> Thanks for any tips!
>
> Ed W
>


The genpi64 project does use distcc for building images when configured.

Like I said, I think there'd be a big benefit to collaborating, but the
image builder is usable as is for your purpose, if I understand it
correctly. Its just missing the native binaries to speed things up.

>