Re: [gentoo-dev] Cross Compilation and Dynamic Slots

[Jason Stubbs]

On Thursday 12 August 2004 01:17, Ilya A. Volynets-Evenbakh wrote:
> Let me try to understand multilib on MIPS example (which is probably
> most complex ABI screw-up case).
> For starters, standrard mips toolchain, when compiled with multilibs
> enabled, is capable of producing binaries for any ABI. At the same time,
> this imposes specific file system layout
> (o32 libraries go into /lib /usr/lib /usr/lib/gcc-lib/lib,
> n32 - /lib32 /usr/lib32 /usr/lib/gcc-lib/lib32,
> n64 - /lib64 /usr/lib64 /usr/lib/gcc-lib/lib64).
>
> On one hand this is good - different ABIs can coexist, and even mix in
> some cases.

Sounds pretty much the same as AMD64 so far...

> On the other hand thus is bad - many programs do not expect to have
> libraried not in /lib /usr/lib places.

This issue affects any user of multilib and perhaps any user wishing to cross 
compile. This point hasn't really been addressed yet.

> As it is now, I have special patch, that makes glibc and gcc
> use /lib /usr/lib layout when using N32 and not using multilibs.

So N32 is the "default", then?

> So, for one thing, we should probably add another variable - ABI - since
> ARCH will be mips in all three cases.

ARCH comes from the expansion of the keyword, as per GLEP 22. For portage to 
be able to pass the ABI, under this proposal, keywords would need to be 
extended to include it - something like mips-n64-gnu-linux-glibc. 
Architectures not supporting more than one ABI could use be called "std". 

However, to get this included you will need to convince the general developer 
community that it is a Good Thing. It'd help if you can find another 
supported architecture that could make use of it.

> How do we handle the case, where we have two different apps, depending
> on same lib, and one is requested to be compiled as o32 and another as n32?

I covered this in the proposal but I'll cover it again below.

> And in general, how do we allow user to specify which ABI he wants to use
> for this specific compile (Yes - it is possible some will want to compile
> different apps with different ABIs - i.e. n32 is generally faster then n64
> because it has lmaller pointer size, but for that same reason its addressing
> is limited. Thus in order to utilize all 8G of ram in my origin, I will need
> certain apps to be n64).

I covered this too, but to sum it up "emerge mem-hungry-app&mips-n64"


I'll edit what I posted last time in order to clear up issues others have 
brought up (thanks solar & lv!)

On Wednesday 11 August 2004 23:21, Jason Stubbs wrote:
=== GENERAL ===

> EBUILDS
> pkg_select()
>    This function configures the system to use this package by default. If
>    it is successful or if there is no need to do so, it should return 0.
>    It should also export any information needed to revert the system back
>    to its original state. If it is not successful, it should do any
>    cleanup required immediately and return 1.
> pkg_revert()
>    This function reverts the system to back to the state it was in before
>    running pkg_select() based on information exported by pkg_select().

> PROFILE GLOBALS
keywords.desc
   Each "old" keyword gets two new fields. One field defines what the keyword
   is expanded to by default. The other field defines what class of keyword
   will match the "old" keyword. Using MIPS as an example, the three fields
   would look something like:
   mips mips-n32-gnu-linux-glibc mips-*-gnu-linux-glibc


=== DYNAMIC SLOTTING ===

> ATOMS
> cat/pkg{atom [atom]}
>    The braced atom specifies extra bounds for a matching cat/pkg's DSLOT.
   This is used to specify a particular combination of a package and one or
   more of the packages it is dynamically slotted.

> EBUILDS
> DSLOT="atom [atom ...]"
>    Portage scans installed and to be installed packages against the
>    specified atom(s) and builds the DSLOTted package once for each
>    individual package match, passing the matched package name to the
>    ebuild in the DSLOT variable.

> INSTALLED DB
> /var/db/pkg/.../pkg-ver[--cat--pkg--slot]
   The --cat--pkg--slot represents the package that pkg-ver was DLOTted
   against.


=== CROSS COMPILATION & MULTILIB ===

> PROFILES
make.defaults
   KEYWORDS will be replace ARCH to allow multiple keywords. When portage
   checks the masking status of a set of packages, it checks all packages
   against each keyword from first to last and stops when it finds a match.
   Any ACCEPT_KEYWORDS settings will also be reordered to match the ordering
   of KEYWORDS so that the profile defined precedence is adhered to.
> packages.cc
>    Defines a set of atoms that must be on the host system to be able to
>    compile for this profile.
> bashrc - available as of portage-2.0.51_pre17
   ARCH, ABI, USERLAND, LIBC & KERNEL are defined as those required for the
   package that is about to be compiled when sourced. These will be defined
   as per the keyword matched or the keyword specified by the requesting
   atom. Bashrc should define aliases, set up CFLAGS or anything required to
   compile for the passed (expanded) keyword using the packages defined in
   packages.cc.

> ATOMS
> cat/pkg&keyword
>    Defines that a package needs to be compiled as the specified keyword.
   A package will only match if it has keyword (or a derivative) listed in its
   KEYWORDS. This overrides the ordering set by the profile's KEYWORDS. If the
   specified keyword (or a derivative) does not exist in the profile's
   KEYWORDS, the atom will not match any packages.

> GCC
>
> To build a cross compiler, the atom would be "sys-devel/gcc&keyword". Gcc

This is wrong and is the result of lack of sleep. "sys-devel/gcc&keyword" 
would result in the package not being installed if 'keyword' did not exist in 
the profile's KEYWORDS. The only workaround I can think of for this is a 
per-profile package.keywords and have the profile specify all keywords for 
sys-devel/gcc. Gcc would then still be able to contain the following logic.

> would then receive the keyword in the form of ARCH, USERLAND, LIBC & KERNEL
> but would still be able to tell the host architecture by way of the keyword
> 'USE flag'. Portage would then slot the compiler into /var/db/pkg/keyword/.
> Gcc would also define pkg_select() and pkg_revert() to do the same as
> ocaml.

If anybody can think of a better way to do this, I'd love to hear it. As it 
stands, gcc is the only sore thumb to this whole proposal.

> INSTALLED DB
> /var/db/pkg/keyword/cat/...
   The keyword field is the KEYWORD that was matched when selecting the
   package to be installed. The keyword is always in its full form as per
   GLEP 22 (with any approved ABI extension).

DEPENDENCY RESOLUTION
   When portage resolves a package, all dependencies and sub dependencies are
   only matched against the keyword that the top package was matched against.
   For example, if the profile's KEYWORDS is "foo bar" and a user does emerge
   somepkg&bar (or the package does not have KEYWORDS="foo") somepkg and any
   of its dependencies will be installed as "bar".


=== EXAMPLE ===

I'll only cover cross compilation and multilib as I think the others speak for 
themselves. For the example, I'll theoretically go through the cross 
compilation of a package for both mips-n32 and mips-n64 into the one altroot.


-- Start --
* The user sets up ${alt} root with the mips profile to be compiled for.
  - The profile has KEYWORDS="mips-n32 mips-o32 mips-n64"
* The user executes "ROOT=${alt} emerge some/pkg some/pkg&mips-n64"
  - The some/pkg ebuild has KEYWORDS="mips"
* Portage checks ${alt} profile's package.cc and ensures that all packages 
   are installed.
  - packages.cc would contain sys-devel/gcc&mips and whatever else
  - If they weren't installed, they would be in the same fashion as the
    following.

-- some/pkg --
* Portage scans the profile's KEYWORDS and finds the first to be mips
* Portage scans the global keywords.desc to find mips expands to
  mips-n32-gnu-linux-glibc.
* Portage scans the ebuild's KEYWORDS and when it finds mips,
* Portage scans the global keywords.desc to find the mips keyword will
  match mips-*-gnu-linux-glibc.
* Portage chooses mips-n32-gnu-linux-glibc for this package.
* Portage runs pkg_select() for each of packages.cc's installed packages on
  the host to prepare for cross compilation.
* Portage expands the keyword into variables and passes them to ${alt}'s 
  bashrc.
* Bashrc sets up for n32 compilation.
* The ebuild does it's regular thing.
* Portage records the package to 
  ${alt}/var/db/pkg/mips-n32-gnu-linux-glibc/some/pkg
* Portage runs pkg_revert() on each of the earlier successful pkg_select()s

-- some/pkg&mips-n64 --
* Portage scans the global keywords.desc to find mips-n64 expands to
  mips-n64-gnu-linux-glibc.
* Portage scans and expands each of the profile's KEYWORDS to find that
  mips matches.
* Portage scans the ebuild's KEYWORDS and when it finds mips,
* Portage scans the global keywords.desc to find the mips keyword will
  match mips-*-gnu-linux-glibc.
* Portage chooses mips-n64-gnu-linux-glibc for this package.
* Portage runs pkg_select() for each of packages.cc's installed packages on
  the host to prepare for cross compilation.
* Portage expands the keyword into variables and passes them to ${alt}'s 
  bashrc.
* Bashrc sets up for n64 compilation.
* The ebuild does it's regular thing.
* Portage records the package to 
  ${alt}/var/db/pkg/mips-n64-gnu-linux-glibc/some/pkg
* Portage runs pkg_revert() on each of the earlier successful pkg_select()s


Yes, I realize that mips-n32 and mips-n64 were almost the same - that's the 
beauty of it! ;)

Regards,
Jason Stubbs

--
gentoo-dev@g.o mailing list