Gentoo Archives: gentoo-alt

From: Ruud Koolen <redlizard@g.o>
To: gentoo-alt@l.g.o
Subject: [gentoo-alt] Restructuring the bootstrap (or: getting rid of gcc 4.2)
Date: Sun, 02 Feb 2014 02:31:14
Message-Id: 201402020331.51096.redlizard@gentoo.org
1 Hi all,
2
3 I have recently been doing some cleanup work to the bootstrap script, getting
4 rid of cruft here and there. My goals in this have been twofold: part of the
5 goal is simply to simplify the overly cluttered bootstrap process to improve
6 maintainability and robustness, and the other part is to ease the way for
7 merging in RAP support later. Work so far has consisted of straightforward
8 cleanup, but the remaining change I have planned is a lot more involved.
9
10 For bootstrapping RAP, I've been using a modified version of the bootstrap-
11 prefix.sh script whose overall working is quite different from the structure
12 used by mainline prefix. In working with it, I found that my experimental
13 version of the bootstrap process is considerably simpler and less error-prone
14 than the mainline process; most of the nasty stage3 hacks are gone, and there
15 is much less possibility to totally screw things up by manually emerging the
16 wrong package at the wrong time. More importantly, my bootstrap process
17 completely avoids using gcc 4.2 as a bootstrap compiler, which makes things a
18 lot simpler -- depending on such a thoroughly deprecated version just isn't
19 sustainable, and there have been problems with packages not compiling with it
20 or architectures just not supported by gcc 4.2. Obviously, these problems will
21 only get worse over time.
22
23 For those reasons and some others (summarized below), I feel my experimental
24 bootstrap proceduce is an improvement for mainline prefix too, and I propose
25 changing the bootstrap-prefix.sh script to use it. Of course, besides
26 improving the mainline prefix bootstrap, this will also make it possible to
27 merge in RAP support later.
28
29 The details of the new process are as follows:
30
31 In the mainline process, stage1 manually builds a handful or required tools
32 (such as make, patch, and python) in $EPREFIX/tmp, using the host toolchain.
33 stage2 then manually builds a portage and constructs a profile, both in
34 $EPREFIX. stage3 starts by using portage to build a toolchain in $EPREFIX
35 (along with minimal dependencies, such as bash and bison), built using the
36 host toolchain; afterwards, portage uses this toolchain to emerge a proper
37 portage (also including dependencies, like python), emerged on top of the
38 bootstrapped portage using USE=-collision-protect. At this point, portage is
39 basically operational, and what remains is emerge -e @system.
40
41 An important aspect of this system is that until the toolchain is built, no
42 libraries can be built using the host toolchain. The reason for this is that
43 host-built libraries may not be ABI-compatible with binaries built using the
44 gentoo toolchain; the host toolchain may build 32-bit libraries while prefix
45 builds 64-bit code, or maybe the most toolchain builds softfloat code as
46 opposed to hardfloat used by prefix. This is not a problem for binaries built
47 by the host toolchain (as long as the host-built binaries run in some way,
48 anything goes), but libraries built by the host and binaries built by prefix
49 don't match. This means that we cannot use toolchain components that
50 themselves require libraries, which means this first toolchain is restricted
51 to gcc 4.2 (later versions require the gmp, mpfr, mpc libraries).
52
53 In my experimental process, things work differently. The stage1 part is
54 unchanged, but stage2 bootstraps a portage *in $EPREFIX/tmp*, and portage's
55 cross-eprefix functionality is used to have this portage build packages in
56 both $EPREFIX/tmp and $EPREFIX. In stage3, portage builds a toolchain (and
57 dependencies) in $EPREFIX/tmp, but one that targets $EPREFIX; that is, it
58 builds a toolchain in $EPREFIX/tmp that builds executables that expect to run
59 in $EPREFIX, have $EPREFIX for rpath, and read headers from $EPREFIX. This
60 toolchain is then used to build a second toolchain in $EPREFIX (targeting
61 $EPREFIX as normal); this "real" toolchain is then used to build the final
62 portage, in $EPREFIX. At this point, the situation is similar to the end of
63 stage3 in the mainline process, with an operational and self-sufficient
64 toolchain and portage in $EPREFIX, and all that is left to be done is emerge -
65 e @system.
66
67 This process avoids the problem of host-built libraries, as it makes a sharp
68 distinction between host-built code and prefix-built code; all host-built code
69 goes in $EPREFIX/tmp, and all prefix-built code goes in $EPREFIX. This means
70 there is nothing stopping us from using libraries in the host-built code;
71 host-built libraries end up in $EPREFIX/tmp, and only host-built binaries (and
72 libraries) will link to it, so ABI incompatibilities aren't a problem. Thus,
73 the latest version of gcc and friends can be used, avoiding many problems.
74 Similarly, the $EPREFIX/tmp toolchain can build $EPREFIX libraries without
75 problems.
76
77 The system used in my experimental process has similarities to cross-
78 compiling: effectively, it regards the host and the prefix as two completely
79 incompatible systems, using the former to build the latter while keeping them
80 sharply separate, and making sure no host pollution ever ends up in the
81 prefix. (The RAP bootstrap uses real cross compilation to builds its stuff,
82 but that's besides the point of this discussion.)
83
84 As far as I can see, the advantages of my bootstrap method are as follows:
85 - It gets rid of all reliances on old package versions, and gcc in particular.
86 As explained above, gcc 4.2 has bugs that won't be fixed that are causing
87 actual practical problems (#262653, #490774), and has a dated set of supported
88 architectures (which was causing problems in supporting newer android
89 systems); and in general, old packages have a nasty tendency to bitrot, making
90 things unnecessarily flaky (bash-4.0 and 4.1 accidentally got removed from the
91 tree recently, breaking the bootstrap).
92 - It keeps the final prefix clean. The current process bootstraps a portage
93 into $EPREFIX, and overwrites it later with a portage-built portage; these two
94 portages need to be the exact same version, or old junk will end up in the
95 prefix that doesn't belong to any package, which can go wrong if using
96 LATEST_TREE_YES. In my process, nothing ever ends up in $EPREFIX that portage
97 didn't put there. Together with my earlier work to get rid of the stage2
98 profile hackery, this will keep $EPREFIX completely pollution-free.
99 - It gets rid of a large part of the stage3 hacks, most of which are due to
100 either mixing of $EPREFIX/tmp and $EPREFIX or gcc 4.2.
101 - No, it doesn't need to build more compilers than the current method. The
102 current bootstrap proces builds gcc4.2, upgrades to gcc 4.7 (or something),
103 and rebuilds it during emerge -e @system; my process builds a gcc in
104 $EPREFIX/tmp, uses it to build a gcc in $EPREFIX, and rebuilds it during
105 emerge -e @system.
106
107 I propose adopting this new bootstrap system for prefix, both in order to
108 improve the regular prefix bootstrap process, and to allow RAP work to
109 continue (which, by the way, is nearing the end of the road). I have an
110 almost-completely-working(1) implementation of it at the ready (2); when it's
111 properly ready in a few days, if you all agree, I want to emit a call-for-
112 testing to make sure it actually works as promised on all platforms (the only
113 one for which I have been able to do proper testing so far is linux), and get
114 it operational.
115
116 What say you, prefix people? Is this the proper way forward?
117
118 -- Ruud
119
120 1) The only open bug that I know of is that the early building of ncurses (for
121 bash) causes a dependency loop on aix and hpux. On those platforms, ncurses
122 depends on autotools as to use libtool when building ncurses, which of course
123 depends on half the world; giving ncurses the same bootstrap status as bash
124 ("this is a critical package that must not depend on autotools, use a self-
125 contained patch instead!") should fix this. Can someone with access to aix and
126 hpux have a look at this?
127 2) Preview version on http://files.redlizard.nl/prefix-experimental-
128 bootstrap.tar.gz . Note that the overlay is critical -- bootstrap using
129 PREFIX_OVERLAY=/path/to/prefix-experimental-bootstrap bootstrap-prefix.sh .

Replies