Gentoo Archives: gentoo-alt

From: Fabian Groffen <grobian@g.o>
To: gentoo-alt@l.g.o
Subject: [gentoo-alt] [PREFIX] Compilers, linkers, 64-bits and multilib
Date: Fri, 04 Jan 2008 14:47:53
1 Per subject, let's open a can full of worms.
3 For the last few weeks I've been working on 64-bits non-Linux targets.
4 sparc64-solaris at first appeared to be a piece of cake,
5 powerpc64-apple-darwin8 and x86_64-apple-darwin9 tricky because of
6 python, openssl, and most annoying odcctools not being 64-bits
7 safe-code. With x86_64-pc-solaris2.10 I've almost reached the end (5
8 packages pending, mostly trivial) but on the way fixing bugs in
9 packages, I found some horrible scenario forced upon us. Let's start
10 off that x86_64-pc-solaris2.10 needed patches to both gcc and binutils
11 because both do not recognise it as a valid target. By relatively
12 simple patches they seem to work reasonably ok, though.
14 Now the real problem is in the library search paths. When compiling
15 wget on Solaris 10/x86_64 the linker started to complain that there were
16 undefined symbols: intl_gettext. Shame on me, as I ignored those on
17 sparc64-sun-solaris2.10 (you only get them with USE="nls").
18 The message is weird, and doesn't appear on i386-pc-solaris2.10.
19 Further investigation shows that the linker pulls in
20 /lib/64/ (or it's realpath
21 /lib/{amd64,sparcv9}/ instead of as we have in
22 prefix. Linking with -lgettextlib results in a warning that conflicts
23 may arise since two's are being included, as expected.
24 (gettextlib was probably because of libtool correctly linked against the
25 prefix
27 Further investigations (gcc .... -v -Wl,-t) revealed that gcc calls
28 collect2 nicely like ".../collect2 -L/usr/lib/amd64 -L/lib/amd64 ...",
29 which -- indeed -- pushes the host "64-bits libs" in front of any other
30 -L direction onto the linker. I considered that to be no good, and
31 quite explanatory for wget linking to /lib/amd64/ by
32 default. After some searching, the MULTILIB_OSDIRNAMES variable
33 appeared to be causing this. For the m32 targets of gcc on Solaris it's
34 set to ".", for the m64 targets it's set to either amd64 or sparcv9.
35 Since "gcc -m32 .... -v -Wl,-t" revealed no -L instructions for /usr/lib
36 and /lib being passed onto collect2, I locally patched my GCC to have
37 also in the 64-bits case MULTILIB_OSDIRNAMES to be ".". This is where
38 the fun starts.
40 Under Linux, where I've always happily assumed everything is ok, gcc
41 injects -L/usr/lib/../lib64 and -L/lib/../lib64. This fits nicely with
42 MULTILIB_OSDIRNAMES set to "../lib64". It is no good though, so my
43 patch rips them out for Linux as well. Now I started compilling on
44 Solaris, and at the final xgcc call to link together a new stage of the
45 compiler, the GNU linker neatly segfaults. Before it does, it complains
46 that i386 objects aren't compatible with x86_64 ones. Looking into what
47 gcc passes on to collect2, indeed no -L/usr/lib/amd64 is passed, so yay,
48 that worked, but those paths /need/ to be searched in the end, of
49 course. So, work to do for the binutils-wrapper script, that's what we
50 have it for in the end, right?
52 Well, not entirely it seems.
54 First, we never ever injected -L/usr/lib{,64} and -L/lib{,64} into the
55 search path. Binutils folks apparently stick to the standard there, so
56 the 64-bits libpaths are not in the default search path. Right they
57 are, because neither does the runtime linker/kernel do so. Great. Then
58 why did my amd64 linux box at work run so well? On Fedora multi-lib,
59 /lib is the 32-bits dir, unlike under Gentoo, where /lib is always
60 linked to the native bits libdir. To all joy, there is no runpath to
61 /lib64 in bin/bash, and yet ldd happily shows it links against
62 /lib64/ Conclusion, the runtime linker/kernel *does* use
63 lib64 on Fedora at least for a 64-bits object. Getting sweaty yet? I
64 do.
66 My GCC on the Fedora box nicely passes /usr/lib/../lib64 as first
67 argument to collect2. The ldwrapper adds -rpath=${EPREFIX}/usr/lib64.
68 The compiler checks for includes in ${EPREFIX}/usr/include first.
69 * compiler sees correct headers
70 * linker sees uncorrect libs when resolving symbols
71 * kernel sees correct libs, because of run_path directions in the object
72 It probably never failed for me so bad, since Fedora is quite up-to-date
73 with their stuff these days, so I never had clashes. This is scary of
74 course.
76 On Darwin, no -L paths are ever passed to collect2, but we would have
77 noticed it there much sooner. This is where the multilib story comes
78 in. The search order we basically need is: ${EPREFIX}-libs, /-libs. On
79 Darwin no lib64, lib/64, or lib/sparc64 exists. Indeed, they use "FAT"
80 objects, sort of archives where the kernel extracts the object for the
81 arch it needs. Neat or not, it means on Darwin there is only "lib" as
82 directory. We pass ${EPREFIX}-libs in the ldwrapper, and get /-libs
83 from the linker and runtime linker's defaults. So Darwin is correct in
84 this case, as any other 32-bits target. GCC doesn't pass -L/usr/lib as
85 it has been told that's the default anyway.
87 I implemented the 64-bits native targets for Solaris and Darwin as
88 entire prefixes just with 64-bits code and a 64-bits defaulting
89 compiler. I just didn't think of it actually, but I just use
90 ${EPREFIX}/{usr,}/lib for the libraries. E.g. no multilib profiles.
91 For inside the prefix this all works well, however with my patched gcc,
92 the 64-bits system libraries are no longer seen by the linker, as
93 neither gcc nor the ldwrapper give -L directions to them for platforms
94 where they are not /lib. Only since Darwin has FAT objects, it works
95 fine there. Not for Solaris and Linux. Well, Linux, depends on the
96 vendor. Would work for Gentoo, not for Fedora.
98 In prefix, I copied an amd64 profile, and it is a multilib-profile. I
99 don't know exactly why, but at the time I did it, I thought that was
100 good. From a desktop perspective multilib is good for Linux users, as
101 e.g. acroread doesn't come in a 64-bits version. However, should that
102 be our concern? Anyway, the ldwrapper adds:
103 ADDLIBDIR("/usr/lib64");
104 ADDLIBDIR("/usr/lib");
105 ADDLIBDIR("/lib64");
106 ADDLIBDIR("/lib");
107 For the prefix, and at runtime those libs are added if they exist on the
108 system the wrapper runs on. Now for most platforms a bunch of these are
109 not necessary. Darwin will never use the 64 variants, 32-bits installs
110 neither. But why do we use lib64 at all? If we have a compiler that
111 produces 64-bits code, shouldn't the prefix be fully 64-bits as well?
112 The ldwrapper has to be made smarter for cross-compiling anyway, but
113 shouldn't we just nuke lib64? ... or should we think about a "sane" way
114 to overlay a 32-bits prefix with a 64-bits one? Personally I'm against
115 that, better keep it clean, and add paths from both prefixes to your
116 path.
118 In any case, the ldwrapper will have to add /usr/lib64 and friends in
119 case the target system is one that uses those...
122 --
123 Fabian Groffen
124 Gentoo on a different level
125 --
126 gentoo-alt@g.o mailing list