[gentoo-dev] [EAPI 7] Cross-compile improvements (BDEPEND, BROOT, sysroot)

[James Le Cuirot]

Sorry for the length of this but you really need to consider the bigger
picture with this stuff and not all devs are familiar with it.

For EAPI 7, mgorny and I are seeking to address some cross-compiling
issues. This is a subject I've long had an interest in and before I
became a developer, I created a project called cross-boss that, to be
blunt, hacks its way through the shortcomings in Portage and the tree.
It does (or at least did when I last worked on it) achieve a higher
success rate than you would get from crossdev alone and allows you to
install a brand new standalone system from scratch as opposed to one
under /usr/${CHOST}.

I have heard some developers say that cross-compiling is simply not
worth the effort but I have succeeded in building some big ticket items
including MariaDB, GTK+, Qt4, and LibreOffice. This is one area where
Gentoo can really shine so I would like to exploit that potential.

It is unlikely that I'll be able to negate the need for cross-boss
entirely as some packages can be really stubborn but these changes
should at least allow some of the hacks to be removed.

The first change is a long-standing one and adds BDEPEND to the list of
dependency types. For those familiar with the experimental HDEPEND
implementation, this is the same thing under a different name that many
developers have said they prefer. It means "build depend" for packages
to be installed on / aka the CBUILD system.

You may be thinking that is what DEPEND is for but sometimes you need
to differentiate between build-time dependencies that are needed in /
for execution and those that are needed in ROOT for headers, static
libraries, and other things.

This is best illustrated with an example. pam currently DEPENDs on
flex as it links against libfl.a. By default, Portage will not install
DEPEND packages to ROOT so this fails when cross-compiling. flex could
be added to RDEPEND but that is suboptimal and may not be feasible in
other cases. We could run emerge with --root-deps but this tends to
pull in many more packages that play havoc when cross-compiling.

This feature can also be particularly useful for packages like autogen
and dbus-glib that execute themselves during the build. This doesn't
always work when cross-compiling but you can use BDEPEND in conjunction
with the targetroot USE flag (see man 5 ebuild) to install that package
to / first and then use that version to complete the cross build.

So when cross-compiling under EAPI 7, the traditional defaults change.
BDEPEND is installed to / as already stated and DEPEND is now installed
to ROOT. The --root-deps options have no effect. Won't this break
cross-compiling for future EAPI 7 ebuilds and eclasses that haven't
been given the BDEPEND treatment? Yes and no. crossdev's cross-emerge
wrapper has long included the --root-deps=rdeps option by default,
which simply throws DEPEND away and this is equally broken already.

So why did crossdev include --root-deps=rdeps in the first place if it
breaks things? The commit message doesn't really say and solar doesn't
seem to be around to answer that question. I'm guessing it was a
trade-off to save time. Portage currently installs all of DEPEND to /
by default even though most of those dependencies are solely needed
for headers and libraries. This means that when cross-compiling on a
minimal build host, you spend almost half the time building things that
you don't even need. This trade-off will no longer be necessary as
BDEPEND will allow us to make that saving while avoiding the breakages
at the same time.

We do not expect all developers to be mindful of BDEPEND going
forwards. mgorny said that this could be thought of in the same way
that we currently don't beat developers up over mishandling of EPREFIX.
This will mean that I or others may occasionally swoop in to apply
BDEPEND to your packages and I hope that this isn't seen as rude or
invasive. It should have no effect on the non-cross case and for what
it's worth, I think I'm quite careful. :)

Also on the table is a variable called BROOT, which has arisen from bug
#509568. When calling executables from the build host, you can usually
rely on the PATH but sometimes the binary you need isn't in the PATH.
This obviously calls for an absolute path but what if the build host is
prefixed? EPREFIX always refers to the target host so how can you find
the prefix of the build host? Since the build host is always anchored
to / and we always refer to it absolutely, a BPREFIX variable is of
little use so we have therefore chosen BROOT. It's basically EROOT for
the build host.

Up to this point, I have referred to just the cross and non-cross cases
for simplicity. It may really be this simple but we should explore the
possibilities to make sure because I think one particular use case is a
little less clear cut.

--------

1. Regular native
   CHOST = CBUILD, PORTAGE_CONFIGROOT = /, ROOT = /

The traditional setup we're all familiar with. Everything goes to / so
the distinction between BDEPEND and DEPEND is irrelevant here.

2. Rooted native, same config
   CHOST = CBUILD, PORTAGE_CONFIGROOT = /, ROOT != /

This is almost exactly like #1 but the packages are not installed to /
at the end. Headers and libraries are still sourced from / and as such,
BDEPEND still doesn't apply. Despite an identical Portage configuration,
unexpected problems can arise. I did an experiment while writing this
and confirmed that having older libraries installed in / can lead to
breakages.

For example, say you have ncurses:0/5 installed in / and then attempt
to install i7z in a brand new ROOT. It will install ncurses:0/6 to ROOT
but then link the new i7z against libncurses.so.5. Furthermore, Portage
will record it as having been built against 0/6.

I think the easiest way to avoid problems like this is to ensure your
build host is up to date before you start installing things to ROOT.
Portage doesn't check this for you. Maybe it should?

3. Cross
   CHOST != CBUILD, PORTAGE_CONFIGROOT != /, ROOT != /

For all its difficulties, the theory is quite simple. You build using
ROOT as much as possible except when you need to execute something.
This includes headers, which must never be sourced from / as this
quickly leads to breakages. Ensuring this happens is probably easier
than you think. More on that later.

4. Rooted native, different config
   CHOST = CBUILD, PORTAGE_CONFIGROOT != /, ROOT != /

This is the messy one and one I'd like some feedback on as to what
people expect from it and whether it should even be supported at all.
Out of the box, it suffers from the same problems as #2 but keeping
your build host up to date is no longer sufficient. The target's USE
flags could potentially be entirely different, meaning that even the
same version of the same library could differ significantly between /
and ROOT. Even the headers could differ.

As far as I can see, the only sane way to deal with this is to build
against headers and libraries in ROOT instead, much like you would when
cross-compiling in #3. However, this just leaves you with the same
problem in reverse. Since you wouldn't really be cross-compiling, the
build may attempt to execute something that was just built. This
execution will occur in the context of the build host, which may not
have the necessary libraries available.

Perhaps you could force it to treat the build like a cross-compile by
overriding various checks in eclasses and build systems. This does
sound painful though. Another way would be to install an alternative
toolchain like x86_64-cross-linux-gnu. I know of one user who
successfully used this approach with cross-boss. It's probably not
bullet proof though as not all build systems respect CHOST and CBUILD.
I'm leaning towards the idea that this scenario should simply be
unsupported. You might as well start with a stage tarball and chroot
like usual. Do you have any better ideas?

It may seem like I have drifted off the point but I mention this use
case because we need to decide exactly when BDEPEND should apply. If
headers and libraries are sourced from ROOT then it needs to apply,
regardless of whether it is treated as a cross-compile or not.

--------

Still with me? ;)

I raised one further point with mgorny that he feels could potentially
go into EAPI 7 but I think could remain an implementation detail. In
cases #3 and #4 (basically when ROOT != / and PORTAGE_CONFIGROOT != /),
the toolchain needs to know how to source headers and libraries from
ROOT instead of /.

gcc and friends support a --sysroot argument. It used to be the case
that this didn't work on a toolchain configured without a sysroot,
possibly creating issues for #4, but I've tested and it now works
regardless.

In cross-boss, this argument is injected through some wrappers present
in the PATH, in a similar manner to distcc wrappers. This is remarkably
effective but doesn't quite cover everything.

CMake supports a CMAKE_SYSROOT argument that you set in the toolchain
file. We currently don't set this but it could easily be added to
cmake-utils.eclass as well as (or instead of?) the similar variables
that we already do set.

The vast majority of configure scripts (using libtool and not ancient)
support a --with-sysroot argument that can easily be detected by
grepping for lt_sysroot. The behaviour of econf is largely defined by
PMS, hence why mgorny thinks this particular detail could go into
EAPI 7, but it only says which arguments must be passed and doesn't say
that the PM can't pass additional ones. I don't wish this force this
stuff upon Paludis and pkgcore unnecessarily.

There's also a further complication here that I forgot to mention to
mgorny. While calling configure with --with-sysroot certainly helps,
it still stumbles on a significant number of packages that do
relinking at the end of the build if elibtoolize hasn't been called.
elibtoolize has long patched ltmain.sh with ELT-patches/cross/link-ROOT
when cross-compiling and this still applies to the very latest libtool.
I filed several bugs about this before realising that fixing this
globally would be better. elibtoolize doesn't require anything to be
installed and the description does say "this function should always be
safe to run" but I suppose calling it unconditionally might screw up
patching in some isolated cases. What do you think?

Phew, I'm done. Please be gentle! :)

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer