Re: [gentoo-alt] prefix: Keeping old version of portage tree

[Michael Haubenwallner]

Hi Anders, sorry for being verbose, but:

On 07/15/2014 08:19 PM, Anders Bo Rasmussen wrote:
> Hi again.
>
> Today I did some more experiments with Gentoo Prefix and I must say
> that I think it looks very good. Basically I just compiled some
> different programs ending with gvim, testing that it ran on servers
> with different Linux distributions and as different users - and it all
> worked :)

Thanks! Actually, that's the goal of Gentoo Prefix. ;)

> What I want to use Gentoo Prefix for is to make a toolchain for
> compiling a software project (and at the same time also make a lot of
> programs available on different servers).

Seems you have very similar needs than ourself here, except that we do have
to provide more packages than available in the Gentoo Prefix portage tree,
on more platforms than RHEL and SLES. However, to provide a quite stable
development environment, we did have to fork the Gentoo Prefix tree, to be
able to maintain the stable keywords we have added, without the need to
follow each Gentoo package upgrade.

This fork was done in late 2009, and here we're currently in a process of
upgrading to the current Prefix tree - without breaking the existing one.

Additionally, we had to create my own profiles (versioned "2010.0"), to avoid
the need to set USE flags and the like on the target instances. For each
target platform there is a "stable", an "unstable" as well as a "buildbot"
profile with appropriate ACCEPT_KEYWORDS settings, plus the special keyword
called "~buildbot", accepted by the buildbot-profiles for each platform.

Each project-development machine now has both a "stable" and an "unstable"
instance of our 2010.0 Prefix distro with the appropriate profile selected.
Software-project developers now use the "stable" instance for testing their
releases as well as distributing to the customer's machine, while they use
the "unstable" instance for using newer features, testing bugfixes in the
Prefix distro and the like.

Also, for each target platform there's at least one machine having the
buildbot-slave doing fresh bootstraps once a week, as well as updating
that bootstrapped instances triggered by various commit hooks, especially
in our local overlay.

> My basic idea is to have a script that makes a new version of my Gentoo
> Prefix installation each time I run it, so it could be placed e.g. at
> /<somewhere_global_available>/gentoo-<date>. This script or small set
> of scripts should contain which packages to emerge, changes to
> make.conf, package.use and other changes that is needed to make an
> installation. And of course the scripts will be in a VCS. The software
> project will just point to the version it currently uses at
> /<somewhere_global_available>/gentoo-<date>. This way old versions of
> the software project can be compiled without problems, as all the
> tools will be the same version.

Here, we do have a cron job, once a week creating a snapshot of both the
merged tree (Prefix+overlay) and the distfiles from all ebuilds with at
least one stable keyword, as well as the setup script that knows how to
utilize these snapshot tarballs. Every once in a while (at least once a
year), we do set date-based VCS tags on the Prefix tree and the overlay,
matching the last snapshot content, and define this snapshot as release.

> What concerns me is that I think I should only update the Portage Tree
> when I need it. With the method described above I'll get a new version
> of the Portage Tree each time. That means that I'll not get the same
> installation if I run my script with 2 weeks in between, which I don't
> like, as I then can't fix small problems without changing everything.

This now released setup script (plus the two tarballs) then is used for
fresh distro installs on new target machines, where a subsequent emerge
sync+update is optional.

> Any ideas on how I somehow gets the same portage tree or another way
> to solve this problem?
> And I guess I should also store the source files somewhere and not get
> them downloaded, so I don't risk an old version is suddenly not
> available anymore - is there a way to do that?

Along with the forked Prefix tree we also do maintain a "master mirror",
providing LAN-access to the merged tree as well as the distfiles, while
the setup script mentioned above knows how to setup SYNC&GENTOO_MIRRORS.

But: As each of these additional scripts are bound to the existence of
that local overlay, they cannot be made public without modification.
While we would love to have public variants of them - allowing to manage
private distros with optional private overlays, we cannot promise anything
about splitting them into public scripts and private configs.

HTH,
/haubi/