proposal for cross-compie support in EAPI-5, was: Re: [gentoo-dev] Fwd: [gentoo-dev-announce] Call for items for September 13 council meeting

[Thomas Sachau]

Thomas Sachau schrieb:
> Tomáš Chvátal schrieb:
>> Start collecting ideas for EAPI5.
>
> 1) USE-flag based support to cross-compile packages (mostly implemented in multilib-portage)

let me extend this a bit, first the reasoning behind it:


For amd64 users, there is sometimes the issue, that they need 32bit libs for certain packages (e.g.
wine or many binary-only packages). Currently, they only get them prepackaged in binary form with
the emul-linux-x86-* packages. But those packages have a few issues (list does not have to be complete):

-they only contain a limited set of 32bit packages
-they are precompiled, so the user cannot define his own flags
-they have to be manually maintained and updated


So the idea was to add the ability to compile 32bit packages with support from the package manager,
so there is no need for additional packages to maintain. After this was originally implemented, it
was further extended to allow cross-compilation for other targets, not only limited to 32bit packages.


The basic way, how this should work:

First, there is the check, if the current setup is prepared for cross-compilation, for this, the
content of the MULTILIB_ABIS var is checked. If it has more than 1 (space seperated) value, those
values are taken and converted into USE flags in the format of multilib_abi_$ABI.
In the case of current amd64 multilib profile, which i will take as base for my examples, this means
2 additional USE flags: multilib_abi_amd64 and multilib_abi_x86.
Those USE flags are internally USE_EXPANDed, so the output after the normal USE flags is in the form
of this: MULTILIB_ABI="amd64 -x86"
This way, the user is able to define the target ABIs for each package independently from global
settings.
During dependency calculation, every dependency gets additional USE deps for every target ABI, in
this example: category/package[multilib_abi_amd64?,multilib_abi_x86?].
This ensures, that the required dependencies also have the needed 32bit libs/binaries for the
requested package.
Before the pkg_setup phase is executed, the environment is setup for the first target ABI (in this
example: amd64), to make it short, i will just copy the current code from multilib-portage for this:

>         # Set the CHOST native first so that we pick up the native
>         # toolchain and not a cross-compiler by accident #202811.
>         export CHOST=$(get_abi_var CHOST ${DEFAULT_ABI})
>         export AS="$(tc-getPROG AS as)"
>         export CC="$(tc-getPROG CC gcc)"
>         export CXX="$(tc-getPROG CXX g++)"
>         export FC="$(tc-getPROG FC gfortran)"
>         export CHOST=$(get_abi_var CHOST $1)
>         export CBUILD=$(get_abi_var CHOST $1)
>         export CDEFINE="$(get_abi_var CDEFINE $1)"
>         export CCASFLAGS="${CCASFLAGS:-${CFLAGS}} $(get_abi_var CFLAGS)"
>         export CFLAGS="${CFLAGS} $(get_abi_var CFLAGS)"
>         export CPPFLAGS="${CPPFLAGS} $(get_abi_var CPPFLAGS)"
>         export CXXFLAGS="${CXXFLAGS} $(get_abi_var CFLAGS)"
>         export FCFLAGS="${FCFLAGS} $(get_abi_var CFLAGS)"
>         export FFLAGS="${FFLAGS} $(get_abi_var CFLAGS)"
>         export ASFLAGS="${ASFLAGS} $(get_abi_var ASFLAGS)"
>         export LDFLAGS="${LDFLAGS} $(get_abi_var CFLAGS)"
>         local LIBDIR=$(get_abi_var LIBDIR $1)
>         export PKG_CONFIG_PATH="/usr/${LIBDIR}/pkgconfig"
>         if [[ "${ABI}" != "${DEFAULT_ABI}" ]]; then
>                 [[ -z ${CCACHE_DIR} ]] || export CCACHE_DIR=${CCACHE_DIR}/${ABI}
>         fi

After this, all phases until after src_install are executed as usual.

After src_install, there are some checks:
If there is another target ABI, the install dir is checked for possible abi-specific files (headers,
libs, binaries).
If both conditions are true, the current install dir is saved in a different location, everything
cleaned up and the package manager starts again at the preparation step for the target ABI before
pkg_setup. This is done until all target ABIs have been done.

Now comes the merging step: For each target ABI, the installed files get moved back to the original
install location with 2 exceptions:
-if the headers differ between the target ABIs, they get installed into abi-specific subdirectories
and the original header file becomes a wrapper file, which includes the abi-specific header file
depending on the current environenment.
-if there are binaries (and this step is not restricted), those get moved into their original
location, but with an abi-specific appendix, the original file is replaced by a symlink to an
abi-wrapper, which executes the abi-specific binary depending on the current environment.

After this is done, following phases are executed. the pkg_* phases after src_install are looped
over all currently enabled target ABIs. This ensures, that abi-specific commands are executed for
every target ABI.

Differences, options and other things with current multilib-portage implementation:

The binary wrapping in current multilib-portage is only happening, if the package is defined in the
MULTILIB_BINARIES var to avoid the modification of existing ebuilds. For my proposal, this wrapping
should happen by default, unless this is restricted in the ebuild, e.g. by RESTRICT="multilib-binaries".

Another usefull thing not currently enabled in multilib-portage: Instead of having the USE flag for
the current platform internally enabled (in this example, USE=amd64), the target ABI should be the
currently active internal USE flag (in this example, USE=x86 during compilation for x86). This
allows for abi-specific changes and would also allow to install abi-specific files for binary
packages (if there is a binary package for the target ABI and a loop over all enabled target ABIs in
pkg_fetch done).

Since none of this directly affects the ebuilds, the package manager can optionally enable this for
all packages (like multilib-portage does currently), the main point, why this needs to go into an
EAPI is the ability to require a package to provide files for a different target ABI (e.g. packages
for amd64 could directly depend on category/package[multilib_abi_x86], if they require the 32bit
libs of this package).


I hope, this can be at least the base (together with the existing implementation in
multilib-portage) for a spec to get cross-compile support into EAPI-5.

For reference:
-the code of current multilib-portage is in the portage repository on git.overlays.gentoo.org inside
the multilib branch
-an ebuild to install multilib-portage is in the multilib overlay in the portage-multilib branch
(current version in there is 2.2.0_alpha55-r1, i need to merge the most recent changes of portage,
so it might sometimes be a bit behind the latest portage version)
-a minimal stage4 with multilib-portage (also already some months old) can be found in a qemu image
on our distfiles mirrors inside experimental/amd64/qemu/, it is called
multilib-amd64-qemu-2010-12-08.img.lzma