1 |
Hi all, |
2 |
|
3 |
I have recently been doing some cleanup work to the bootstrap script, getting |
4 |
rid of cruft here and there. My goals in this have been twofold: part of the |
5 |
goal is simply to simplify the overly cluttered bootstrap process to improve |
6 |
maintainability and robustness, and the other part is to ease the way for |
7 |
merging in RAP support later. Work so far has consisted of straightforward |
8 |
cleanup, but the remaining change I have planned is a lot more involved. |
9 |
|
10 |
For bootstrapping RAP, I've been using a modified version of the bootstrap- |
11 |
prefix.sh script whose overall working is quite different from the structure |
12 |
used by mainline prefix. In working with it, I found that my experimental |
13 |
version of the bootstrap process is considerably simpler and less error-prone |
14 |
than the mainline process; most of the nasty stage3 hacks are gone, and there |
15 |
is much less possibility to totally screw things up by manually emerging the |
16 |
wrong package at the wrong time. More importantly, my bootstrap process |
17 |
completely avoids using gcc 4.2 as a bootstrap compiler, which makes things a |
18 |
lot simpler -- depending on such a thoroughly deprecated version just isn't |
19 |
sustainable, and there have been problems with packages not compiling with it |
20 |
or architectures just not supported by gcc 4.2. Obviously, these problems will |
21 |
only get worse over time. |
22 |
|
23 |
For those reasons and some others (summarized below), I feel my experimental |
24 |
bootstrap proceduce is an improvement for mainline prefix too, and I propose |
25 |
changing the bootstrap-prefix.sh script to use it. Of course, besides |
26 |
improving the mainline prefix bootstrap, this will also make it possible to |
27 |
merge in RAP support later. |
28 |
|
29 |
The details of the new process are as follows: |
30 |
|
31 |
In the mainline process, stage1 manually builds a handful or required tools |
32 |
(such as make, patch, and python) in $EPREFIX/tmp, using the host toolchain. |
33 |
stage2 then manually builds a portage and constructs a profile, both in |
34 |
$EPREFIX. stage3 starts by using portage to build a toolchain in $EPREFIX |
35 |
(along with minimal dependencies, such as bash and bison), built using the |
36 |
host toolchain; afterwards, portage uses this toolchain to emerge a proper |
37 |
portage (also including dependencies, like python), emerged on top of the |
38 |
bootstrapped portage using USE=-collision-protect. At this point, portage is |
39 |
basically operational, and what remains is emerge -e @system. |
40 |
|
41 |
An important aspect of this system is that until the toolchain is built, no |
42 |
libraries can be built using the host toolchain. The reason for this is that |
43 |
host-built libraries may not be ABI-compatible with binaries built using the |
44 |
gentoo toolchain; the host toolchain may build 32-bit libraries while prefix |
45 |
builds 64-bit code, or maybe the most toolchain builds softfloat code as |
46 |
opposed to hardfloat used by prefix. This is not a problem for binaries built |
47 |
by the host toolchain (as long as the host-built binaries run in some way, |
48 |
anything goes), but libraries built by the host and binaries built by prefix |
49 |
don't match. This means that we cannot use toolchain components that |
50 |
themselves require libraries, which means this first toolchain is restricted |
51 |
to gcc 4.2 (later versions require the gmp, mpfr, mpc libraries). |
52 |
|
53 |
In my experimental process, things work differently. The stage1 part is |
54 |
unchanged, but stage2 bootstraps a portage *in $EPREFIX/tmp*, and portage's |
55 |
cross-eprefix functionality is used to have this portage build packages in |
56 |
both $EPREFIX/tmp and $EPREFIX. In stage3, portage builds a toolchain (and |
57 |
dependencies) in $EPREFIX/tmp, but one that targets $EPREFIX; that is, it |
58 |
builds a toolchain in $EPREFIX/tmp that builds executables that expect to run |
59 |
in $EPREFIX, have $EPREFIX for rpath, and read headers from $EPREFIX. This |
60 |
toolchain is then used to build a second toolchain in $EPREFIX (targeting |
61 |
$EPREFIX as normal); this "real" toolchain is then used to build the final |
62 |
portage, in $EPREFIX. At this point, the situation is similar to the end of |
63 |
stage3 in the mainline process, with an operational and self-sufficient |
64 |
toolchain and portage in $EPREFIX, and all that is left to be done is emerge - |
65 |
e @system. |
66 |
|
67 |
This process avoids the problem of host-built libraries, as it makes a sharp |
68 |
distinction between host-built code and prefix-built code; all host-built code |
69 |
goes in $EPREFIX/tmp, and all prefix-built code goes in $EPREFIX. This means |
70 |
there is nothing stopping us from using libraries in the host-built code; |
71 |
host-built libraries end up in $EPREFIX/tmp, and only host-built binaries (and |
72 |
libraries) will link to it, so ABI incompatibilities aren't a problem. Thus, |
73 |
the latest version of gcc and friends can be used, avoiding many problems. |
74 |
Similarly, the $EPREFIX/tmp toolchain can build $EPREFIX libraries without |
75 |
problems. |
76 |
|
77 |
The system used in my experimental process has similarities to cross- |
78 |
compiling: effectively, it regards the host and the prefix as two completely |
79 |
incompatible systems, using the former to build the latter while keeping them |
80 |
sharply separate, and making sure no host pollution ever ends up in the |
81 |
prefix. (The RAP bootstrap uses real cross compilation to builds its stuff, |
82 |
but that's besides the point of this discussion.) |
83 |
|
84 |
As far as I can see, the advantages of my bootstrap method are as follows: |
85 |
- It gets rid of all reliances on old package versions, and gcc in particular. |
86 |
As explained above, gcc 4.2 has bugs that won't be fixed that are causing |
87 |
actual practical problems (#262653, #490774), and has a dated set of supported |
88 |
architectures (which was causing problems in supporting newer android |
89 |
systems); and in general, old packages have a nasty tendency to bitrot, making |
90 |
things unnecessarily flaky (bash-4.0 and 4.1 accidentally got removed from the |
91 |
tree recently, breaking the bootstrap). |
92 |
- It keeps the final prefix clean. The current process bootstraps a portage |
93 |
into $EPREFIX, and overwrites it later with a portage-built portage; these two |
94 |
portages need to be the exact same version, or old junk will end up in the |
95 |
prefix that doesn't belong to any package, which can go wrong if using |
96 |
LATEST_TREE_YES. In my process, nothing ever ends up in $EPREFIX that portage |
97 |
didn't put there. Together with my earlier work to get rid of the stage2 |
98 |
profile hackery, this will keep $EPREFIX completely pollution-free. |
99 |
- It gets rid of a large part of the stage3 hacks, most of which are due to |
100 |
either mixing of $EPREFIX/tmp and $EPREFIX or gcc 4.2. |
101 |
- No, it doesn't need to build more compilers than the current method. The |
102 |
current bootstrap proces builds gcc4.2, upgrades to gcc 4.7 (or something), |
103 |
and rebuilds it during emerge -e @system; my process builds a gcc in |
104 |
$EPREFIX/tmp, uses it to build a gcc in $EPREFIX, and rebuilds it during |
105 |
emerge -e @system. |
106 |
|
107 |
I propose adopting this new bootstrap system for prefix, both in order to |
108 |
improve the regular prefix bootstrap process, and to allow RAP work to |
109 |
continue (which, by the way, is nearing the end of the road). I have an |
110 |
almost-completely-working(1) implementation of it at the ready (2); when it's |
111 |
properly ready in a few days, if you all agree, I want to emit a call-for- |
112 |
testing to make sure it actually works as promised on all platforms (the only |
113 |
one for which I have been able to do proper testing so far is linux), and get |
114 |
it operational. |
115 |
|
116 |
What say you, prefix people? Is this the proper way forward? |
117 |
|
118 |
-- Ruud |
119 |
|
120 |
1) The only open bug that I know of is that the early building of ncurses (for |
121 |
bash) causes a dependency loop on aix and hpux. On those platforms, ncurses |
122 |
depends on autotools as to use libtool when building ncurses, which of course |
123 |
depends on half the world; giving ncurses the same bootstrap status as bash |
124 |
("this is a critical package that must not depend on autotools, use a self- |
125 |
contained patch instead!") should fix this. Can someone with access to aix and |
126 |
hpux have a look at this? |
127 |
2) Preview version on http://files.redlizard.nl/prefix-experimental- |
128 |
bootstrap.tar.gz . Note that the overlay is critical -- bootstrap using |
129 |
PREFIX_OVERLAY=/path/to/prefix-experimental-bootstrap bootstrap-prefix.sh . |