[gentoo-dev] An Introduction to Gentoo Prefix

[Fabian Groffen]

While some of you may have heard of it, we -- the Prefix team -- have
the impression that for most developers the Gentoo Prefix project is in
general an unknown, hidden and vague project, that primarily generates a
lot of commits.  Therefore, we have decided to try and explain what
Gentoo Prefix is, and why we Prefix devs are so passionately about it.

Let us start by pointing out that Gentoo Prefix is all but a "new"
project.  While the archeological signs of the project only go back to
2006[1], the project has existed before that in the minds of people no
longer with us any more.

The Gentoo Prefix project aims to bring the Gentoo Portage Tree to
non-Gentoo and/or non-Linux systems, through the following two
characteristics:
  1) a free to choose target directory offset (prefix), in which
     packages are installed, and
  2) the ability to install and operate without adminstrative privileges

To understand why these two characteristics were chosen, it is best to
get into the mindset of many Gentoo Prefix users (and devs).  There are
plenty of reasons, of which a few described in [2].  In short, the users
are looking for convenience on their platform of choice, or platform
they are doomed to use.

Some people just have to work with Windows.  Others just like Mac OS X
over any GNOME or KDE.  Some work with Solaris, others with AIX, or even
their Atari.  All of these systems have their merits and demerits.
(Gentoo) Linux users are used to a very rich and convenient userland
consisting of many GNU tools, such as sed, awk, grep, tar, etc., but
also development tools such as compilers, autotools, etc.  While these
tools are absent, simply outdated or not as flexible as on Gentoo, users
of said systems are in need of the flexible tool suite as delivered by
Gentoo.  An obvious solution is to get Portage running and to install
the tools on the system in use.  However, due to requirements,
privileges, or an insane love relation, the original system may not be
harmed by replacing or modifying existing software.  Yes, in some cases,
such modifications are even impossible by lack of privileges: no root
access!

In a really compact form, we can say that for Gentoo Prefix users:
  - an offset installation is necessary not to break the host OS software
  - sometimes root privileges are unavailable
  - unprivileged installations add an extra level of protection/safety
  - `rm -Rf <prefix-offset-location>` removes the entire Prefix and all
	traces of it

Gentoo Prefix delivers unprivileged offset installations, which we call
a "prefix".  The prefix here is the offset used for the entire
installation, such as /home/sally/gentoo.  A Gentoo Prefix system is
bootstrapped into such prefix, by following a number of instructions,
e.g.[3].

Using this prefix, the file system layout chosen by Prefix Portage is
exactly the same as for a normal Gentoo Linux system, but shifted into
the prefix.  This would for instance result in
/home/sally/gentoo/usr/bin for the previously mentioned example.  In
principle, it serves no use to have programs installed in bin, usr/bin,
sbin, usr/sbin, etc. under the prefix, and it would make more sense if
they would be installed all in one place.  However, following the Gentoo
Linux layout enables direct backwards compatability when the chosen
prefix is the empty string, and requires no extra patching or changes.

With this in mind, we would like to stress that it is not the desire of
the current Gentoo Prefix project to go beyond the ability to install in
an unprivileged install-time fixed offset.  We consider the current
approach as a milestone that delivers offset installations.  Pilots
within our own team have shown that on top of this milestone, further
steps can be made, such as variable offset locations and close
cooperation with e.g. a Gentoo Linux host system.  However, we do not
consider these works in progress as part of the *current* Gentoo Prefix
project.

Unfortunately, support for installing packages into a prefix, does not
come for free.  Most ebuilds need small changes, some ebuilds need
extensive patching to get programs to install and run correctly with
regard to the (offset) prefix.  To get this all running, we set up our
own tree, which contains all our "converted" ebuilds, extra profiles and
modified eclasses[4].  At some point our tree became pretty big[1], and
it turned out our enthusiasm (and that of our users) brought
overlays.g.o down to its knees.  At that point, our hobby got way out of
hand, and to resolve the issue we had to switch to rsync, resulting in
our own generated tree with metadata.  As side-effect this greatly sped
up the user experience[5], contributing to getting our hobby even more
out of hand.  We, as Prefix team, feel this puts our project in a bit
weird position, where we resemble closely to an in-house fork of the
Gentoo project itself.

It is our desire and intention to move away from this situation, which
not only consumes an awful lot of resources for maintaining the fork,
but also keeps a lot of work outside of the mainstream Gentoo
infrastructure.

As Prefix team, we feel that the current shape of the Gentoo Prefix
implementation is mature.  That is, it has been proven to be useful for
a number of users/scenarios, and it has been able to work for a
substantial number of different systems, without having to change the
core part.

Therefore, we plan to focus on merging back the many patches and
extensions from our Prefix overlay into the Gentoo mainline tree.  For
us, this roadmap looks as follows:
1) get the prefix USE-flag masked in profiles/base
   We use a special USE-flag 'prefix' to conditionalise actions in
   ebuilds and eclasses that really depend on being in an offset
   installation or not.  Obviously, this flag should be disabled in
   normal profiles.
2) move the prefix profiles to the mainline tree
   The Prefix project maintains a profiles/prefix directory with Gentoo
   Prefix specific profiles.  They inherit from base, and in the Linux
   case even from the default/linux profiles.
3) add prefix-only ebuilds to the mainline tree
   Gentoo Prefix has different systems, which sometimes need different
   packages.  Specialised ebuilds now live in the Prefix tree only.
4) file bugs for eclass changes to their maintainers
   Like ebuilds, eclasses need changes to work properly using an offset
   installation.  These changes are usually related to dealing with the
   offset prefix, but sometimes also add specialised support for the
   Prefix architectures.
5) bring back trivial patches
   Different systems bring different compile or installation problems.
   These patches can be trivial, and hence non-intrusive.  We like to
   move those patches to the Gentoo mainline tree.
6) solve complicated ebuild changes
   For a number of ebuilds, mostly for the toolchain, a set of extensive
   patches are applied in the Prefix tree.  While our changes are mostly
   designed to be backwards compatible, we prefer to get them reviewed
   by the respective maintainers and get them merged with their
   blessings.
7) merging prefix branch of Portage into trunk
   Prefix Portage currently lives in its own branch of the Portage
   sources.  With an empty offset string, Prefix Portage is equal in
   behaviour to trunk sources.  We would like to merge the Prefix branch
   with trunk and have a release, instead of keeping the branch and
   adding a Prefix Portage ebuild.
8) add prefix bits to ebuilds
   Prefix involves some variables which need to be used in ebuilds
   to make them offset-aware.  Often this is trivial, but requires
   understanding on what is exactly going on.  We plan to provide
   information about this.  At the same time we do this, we want to move
   over our Prefix arches keywords, such that we can start using the
   Gentoo mainline tree.

We hope this informative email sheds a light on the activities of the
Gentoo Prefix team, and the actions we aim to take in the near future.
We are open for any questions or comments.


[1] http://stats.prefix.freens.org/keywords-packages.png
[2] http://www.gentoo.org/proj/en/gentoo-alt/prefix/usecases.xml
[3] http://www.gentoo.org/proj/en/gentoo-alt/prefix/bootstrap-solaris.xml
[4] http://overlays.gentoo.org/proj/alt/browser/trunk/prefix-overlay
[5] http://stats.prefix.freens.org/rsync-usage.png


-- 
Fabian Groffen
Gentoo on a different level