[gentoo-alt] Bootstrap changes

[Ruud Koolen]

Hi all,

I just pushed the last parts of the bootstrap changes I announced almost a 
year ago [1]. Parts of these changes have been live for a few weeks. These 
changes have been pretty well tested and have solved many bootstrapping 
problems, and they make the bootstrap procedure more future proof. The main 
practical consequence of this is that the bootstrap procedure no longer 
depends on gcc 4.2 as a temporary component, which has resolved a lot of 
minor problems and nasty bootstrap hacks. This mail serves as a summary of 
what has changed, and why.

The old bootstrap procedure consists of four major steps:
- in stage1, tools that portage needs to do its job but which are not yet 
present are compiled; examples are make, wget, and patch. These tools are 
built by hand rather than by ebuild, and are installed in ${EPREFIX}/tmp. 
They are compiled using the host compiler.
- in stage2, a version of portage is manually installed in ${EPREFIX}. This 
version will later get forcibly overwritten by `emerge portage`, so to avoid 
orphaned files it is essential this is exactly the same version as the 
version installed later.
- in the beginning of stage3, a toolchain (gcc, binutils, bash, and relevant 
dependencies) are built by portage in ${EPREFIX}. All of this is built using 
the host compiler.
- in the remainder of the bootstrap, consisting of the second part of stage3 
and beyond, the native toolchain from the beginning of stage3 is used to 
build (and, as necessary, rebuild) a complete system. Somewhere along the 
way, the ${EPREFIX}/tmp tools are removed.

A key property of this scheme is that the early stage3 tools are all built 
using the host compiler. As long as all the components in early stage3 
consist of only executables without any libraries, this is fine. Once you 
want to use libraries in early stage3, however, things become problematic. 
For stuff compiled with the host compiler need not be ABI-compatible with 
stuff compiled with the prefix compiler; the host compiler might use a 
different bit size (-m64 versus -m32), softfloat instead of hardfloat, an 
entirely different non-gcc compiler that isn't binary compatible with gcc, 
whatever. A consequence of this incompatibility is that you can't upgrade 
those libraries or the executables that link against them. For instance, if 
early stage3 would contain a bash linked against readline, as soon as you 
would rebuild either of them later, bash would stop working because it can't 
link to readline.

As a consequence of the above, the old bootstrap procedure uses only early 
stage3 components that do not need any libraries. In particular, this means 
using gcc 4.2, for all later versions need the gmp and mpfr libraries. It 
also means that bash needs to be built without readline support, which causes 
nasty problems all around.

The new bootstrap procedure remedies this shortcoming by using the magic of 
cross-EPREFIX building to keep a clear separation between tools built using 
the host compiler and tools built using the prefix compiler. This separation 
is enforced by making sure that everything built by the host compiler only 
ever ends up in ${EPREFIX}/tmp, whereas everything built by the prefix 
compiler only ever ends up in ${EPREFIX}. Thus, libraries from the two 
ABI-incompatible worlds never end up fighting with each other.

With this split, portage can build a compiler that is built by the host 
compiler, and thus lives in ${EPREFIX}/tmp, but which creates binaries that 
live in ${EPREFIX}. To accomplish this, the bootstrap procedure needs an 
extra step between the installation of a temporary portage and stage3. I have 
called this step stage2, and moved the installation of a temporary portage 
into stage1. This results in the following procedure:
- in stage1, portage dependencies (make, wget, patch, etc) as well as portage 
itself are hand-bootstrapped into ${EPREFIX}/tmp. They are all built using 
the host compiler. In particular, portage is installed in ${EPREFIX}/tmp; 
this portage will be used to emerge packages both in ${EPREFIX}/tmp and 
${EPREFIX}, which it can do nowadays.
- in stage2, a toolchain is built that lives in ${EPREFIX}/tmp but creates 
binaries in ${EPREFIX}. This toolchain is built by portage using the host 
compiler.
- in early stage3, a native toolchain is built in ${EPREFIX}; that is, this is 
a toolchain that both lives in and targets ${EPREFIX}. It is built by 
portage, using the compiler from stage2.
- in the remainder of the bootstrap, the native compiler is used to build a 
portage in ${EPREFIX}, followed by the rest of the system. ${EPREFIX}/tmp is 
removed as soon as native portage is installed, at which point no further 
temporary tools are necessary.

As a side effect of the above, some minor other things have changed that 
warrant explanation:
- There is no longer a need for the manually bootstrapped portage to match the 
version in the bootstrap snapshot. This used to be necessary to avoid 
littering the final prefix with portage files that are no longer present in 
the portage-tree version; but because the temporary portage is installed in 
${EPREFIX}/tmp, it can litter as much as it likes.
- Most of the stage3 workarounds are no longer necessary, and for those that 
remain I removed the kludges that relied on temporarily patching the portage 
tree. What remains all works based on environment variables instead of 
polluting the profiles. *Please* keep it that way, as this makes the 
bootstrap process much more robust in case people need to sync the portage 
tree halfway through for whatever reason.
- Due the the split-off of ${EPREFIX}/tmp, the parts of ${EPREFIX} that are 
usually managed by portage are never hand-patched during the bootstrap 
script. There are a few things that *are* hand-hacked such as the 
${EPREFIX}/bin/sh symlink, but those are allowed exactly because they are not 
the purview of portage. The contents of ${EPREFIX}/tmp, on the other hand, 
can be hacked as much as necessary.

I hope this clears things up.
-- Ruud


References:
1. http://article.gmane.org/gmane.linux.gentoo.alt/6867