[gentoo-alt] AIX-specific early stage3 dependencies

[Michael Haubenwallner]

Hi Ruud,

(CCing gentoo-alt, as the reasoning below is worth to be archived in the public)

On 04/28/2014 04:41 AM, Ruud Koolen wrote:
> Hi haubi,
>
> As part of the new-bootstrap effort I'm trying to clean up old hacks (at least
> in stage3) and simplify platform-specific paths. I understand most of them
> but there are a few aspects of which I don't properly understand what they
> are for. Can you clear up the aix parts of that?
>
> Specifically, configure_toolchain() lists sys-apps/aix-miscutils,
> sys-apps/texinfo, and sys-apps/diffutils as extra early stage3 packages, and
> stage3 itself builds dev-libs/libiconv to "avoid hell with shared
> libiconv.a"; I want to know exactly what they're used for and at which point
> we need it.
>
> sys-apps/aix-miscutils contains ldd and aixdll. ldd sounds like something
> that's not necessary in the bootstrap, but aixdll sounds like something
> necessary for building shared libraries. Is it? If so, does aix need it to
> build ANY shared library, including such things as the early iconv and
> ncurses?

IIRC, aixdll is needed by portage itself for these tasks:
*) extract the binary dependencies for preserve-libs feature
*) merge library archives containing shared objects - even with "aix-soname".

> I don't have a good idea of what texinfo might possibly be used for, except
> for makeinfo for which there is now a more general workaround in place. Is it
> used for anything else?

IIRC nope, makeinfo might have been the only reason.

> sys-apps/diffutils is apparently needed to work around
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=14251 , but I don't see how --
> that bug needs gnu coreutils, not diffutils. Do you know what this is for?

AFAICT, the comparison stage in gcc uses cmp. IIRC I've seen failures when that is
not the GNU diffutils' cmp.

> Moreover, do you know whether this workaround is still necessary for the aix
> versions we support?

Comment it out or drop it, and the buildbots here whill tell.

> If I understand correctly, dev-libs/libiconv is a dependency for a lot of
> programs, including gcc, and as such needs to be installed before gcc. This
> is clear. Are there any other packages in the early bootstrap that depend on
> it? Could it be installed immediately before gcc? I see that it was recently
> moved from immediately before gcc, to one of the first things in stage3. Why
> is this? What hell are you avoiding this way that I should retain?

Okay, you've asked: The "hell with shared libiconv.a" is this one:

*) AIX provides:
    /usr/lib/libiconv.a[shr4.o]          32bit
    /usr/lib/libiconv.a[shr.o]           32bit
    /usr/lib/libiconv.a[shr4_64.o]       64bit

*) GNU libiconv eventually is installed in /opt/freeware, providing
    /opt/freeware/lib/libiconv.a[libiconv.so.2]   32bit
    /opt/freeware/lib/libiconv.a[libiconv.so.2]   64bit
    /opt/freeware/lib/libiconv.a[shr4.o]          32bit (mirroring /usr/lib/libiconv.a)
    /opt/freeware/lib/libiconv.a[shr.o]           32bit (mirroring /usr/lib/libiconv.a)
    /opt/freeware/lib/libiconv.a[shr4_64.o]       64bit (mirroring /usr/lib/libiconv.a)

*) gcc binary install (used for bootstrap) is installed in /opt/freeware:
    -L/opt/freeware/lib is added by gcc at linktime
    -L/usr/lib          is added by gcc at linktime

As you might notice, the FILENAME always is libiconv.a - only the shared member name varies.

For completion, a Shared Object can have an F_LOADONLY flag, meaning to be ignored by the
buildtime linker when it is an archive member, while the runtime linker still can use it.

The first useable (that is: either a Shared Object without the F_LOADONLY flag or a Static
Object) archive member, that provides any useful symbol, is used by the buildtime linker,
which records both the archive's file-name and the Shared Object's member-name into the
final binary for the runtime linker.

The runtime linker then walks through the runpath for any recorded archive file-name, to
load the recorded Shared Object's member-name from. Unfortunately, it does _not_ continue
searching for another archive file when loading from the first one found fails. This is
the reason why /opt/freeware/lib/libiconv.a mirrors /usr/lib/libiconv.a's Shared Objects.
This mirroring already is part of the GNU iconv's build system, so we don't have to care.

This actually is the "traditional" AIX way of shared library versioning.

Because of this crap, I've developed the new "aix-soname" way, using Import Files to provide
filename-based shared library versioning - where the archive file-name changes, while the
Shared Object's member-name is constant: shr.o for 32bit and shr_64.o for 64bit. As a result,

*) Prefix GNU libiconv with "aix-soname" enabled provides:
    /Gentoo/Prefix/usr/lib/libiconv.so.2[shr.imp]      "32bit" ("aix-soname" plaintext Import File)
    /Gentoo/Prefix/usr/lib/libiconv.so.2[shr.o]         32bit  (with F_LOADONLY flag set)
    /Gentoo/Prefix/usr/lib/libiconv.so -> libiconv.so.2        (used with -brtl or -G linker flag)
    /Gentoo/Prefix/usr/lib/libiconv.a[iconv.o]          32bit  (Static Object)
    /Gentoo/Prefix/usr/lib/libiconv.a[localcharset.o]   32bit  (Static Object)
    /Gentoo/Prefix/usr/lib/libiconv.a[relocatable.o]    32bit  (Static Object)
    /Gentoo/Prefix/usr/lib/libiconv.a[shr4.o]           32bit  (mirroring /usr/lib/libiconv.a)
    /Gentoo/Prefix/usr/lib/libiconv.a[shr.o]            32bit  (mirroring /usr/lib/libiconv.a)

Now for the bootstrap problem when there is GNU libiconv available in /opt/freeware:

Bootstrap binaries are linked against libiconv.a[libiconv.so.2] from /opt/freeware/lib.
Additionally, we ensure that /Gentoo/Prefix/usr/lib is part of the runpath. Usually, we make
use of /opt/freeware/bin/gcc via /usr/bin/gcc symlink, or via /opt/freeware/bin in PATH.
But our bootstrap does not know anything else about /opt/freeware at all.

Now, as soon as we have emerged Prefix GNU libiconv with "aix-soname" enabled, the runtime
linker fails to load [libiconv.so.2] from /Gentoo/Prefix/usr/lib/libiconv.a, because GNU iconv's
build system only knows to mirror Shared Objects found in /usr/lib/libiconv.a.

Thus, it is important to provide "aix-soname" enabled /Gentoo/Prefix/usr/lib/libiconv.so early
enough, to not link against /opt/freeware/lib/libiconv.a at all by accident.

An additional reason for filename-based shared library versioning is IBM-Java:
When executing any system binary from within some java application, LIBPATH is set to /usr/lib.
With traditional shared library versioning, the runtime loader fails to load [libiconv.so.2]
from /usr/lib/libiconv.a again.

For complete history see https://bugs.gentoo.org/show_bug.cgi?id=213277

/haubi/