Re: [gentoo-dev] avoiding urgent stabilizations

[Ed W]

Hi

> But, for me, even a trimmed-down Gentoo is still too large
> (has to contain the whole base packages, from portage to
> toolchain, includes, etc). I'd prefer having only the essential
> runtime stuff within the containers.

I'm just building some embedded devices on the side using gentoo and my 
minimal builds are only a few MB? Curious why you feel you need to move 
from Gentoo to get the size smaller?

Seems like your complaint is that you have gentoo installs which are 
full featured with a toolchain and portage, which you are comparing to 
an installation you built with a different tool that doesn't have a 
toolchain installed?  However, you can do the same using gentoo if you 
wish? (you just need a lightweight package installer to avoid installing 
portage)

I think your main options are:

1) Build your base images without a toolchain or portage and use a 
minimal package installer to install pre-built binary packages.  This 
seems fraught with issues long term though...

2) Build your base images without a toolchain, but with portage (and 
perhaps a very minimal python). This gives you full dependency tracking 
and obviously bind mount/nfs mount the actual portage tree to avoid 
space used there. This seems workable and minimal?

3) If we are talking virtual machines then who cares if your containers 
are individually quite large, if the files in them are duplicated across 
all containers?  Simply use an appropriate de-duplication strategy to 
coalesce the space and most of the disadvantages disappear?  eg 
linux-vserver you can simply hardlink all the common files across your 
installations and allow the COW patch to break hardlinks if anyone 
alters a file in a single instance. Or you could use aufs to mount a 
writeable layer over your common base VM instance?  Or you could use one 
of the filesystems which de-duplicates files in the background (some 
caveats apply here to avoid memory still being used multiple times in 
each VM).  Or under KVM there is the memory coalescing feature which 
merges similar code pages (forget it's name?)

Personally I think option 3) is quite interesting from the medium number 
of virtual machines, ie in the 10s to hundreds, ie simply don't worry 
about it, let the OS do the work.  In the hundreds to thousands plus 
level I guess you have unique challenges and I would be wrong to try and 
suggest a solution from the comfort of a laptop without having that 
responsibility, but I would have thought there was some advantage in a 
very rigidly deployed base OS generated and updated very precisely?


> For this we need a different approach (strictly separating build
> and production environments). Binary distros (eg. Debian) might
> be one option, but they're lacking the configurability and mostly
> are still too large. So I'm going a different route using my own
> buildsystem - called Briegel - which originally was designed for
> embedded/small-device targets.
>
> For now I didn't have the spare time to port all the packages
> required for complete server systems (most of it is making
> them all cleanly crosscompile'able, as this is a fundamental
> concept of Briegel). But maybe you'd like to join in and try it :)

Sounds like an interesting challenge, but I'm unconvinced you can't 
solve 90% of your problem within the constraints of Gentoo? This saves 
you a bunch of time that could be invested in the last 10% through more 
traditional means?


>> It does appear like managing large numbers of virtual machines is one
>> are that gentoo could score very well?  Interested to see any chatter on
>> how others solve this problem, or any general advocacy?  Probably we
>> should start a new thread though...
> I'm not sure if Gentoo really is the right distro for that purpose,
> as it's targeted to very different systems (i.g. Gentoo boxes are
> expected to be quite unique, beginning with different per-package
> useflags, even down to cflags, etc). But it might still be a good
> basis for building specific system images (let's call them stage5 ;-))

I won't disagree on your "where it's targeted", but just to re-iterate 
why I think Gentoo works well is that it does have a very workable 
binary package feature!

My way of working is to use (several) shared binary package repos and 
the guests largely pull from those shared package directories.  In fact 
what I do is have a minimal number of shared "/usr/portage/package" 
directories and I mount an appropriate one to the guest type at boot 
time.  At the moment my main two options are "32bit" and "64bit" for the 
package mounts, but I recently introduced a new machine type which is 
held back to perl 5.8 and that guest gets it's own package mount since 
it's obviously linking a lot of binaries differently

So, my process is to test an update on a small number of guests, either 
dedicated test guests or less important live guests.  If this looks good 
then I run the upgrade against all other Vms of the same type and they 
will update quickly from package binaries

Now, the icing is that this works extremely well even once you decide to 
lightly customise machine types.  So for example my binary packages are 
very high level (eg 32/64bit), my "profiles" would be fairly high level, 
eg I have www-apache and www-nginx base profiles.  However, a specific 
virtual machine running say nginx might itself need a specific PHP 
application installed, and that itself might need some dependencies, 
which in turn might require a specific set of customisation of use flags 
and versions.

Now, the neat thing is that the binary upgrade options are *either* to 
use *only* binary packages, OR to use binary packages *if* they were 
built with the correct USE flags. So for example I haven't bothered to 
split out my packages directory to be specific to the nginx/apache 
machines, however, this causes the PHP package to be regularly rebuilt 
depending on whether it was last used to upgrade an nginx or apache 
guest (different use flags needed for each guest).  I could fix this 
easily enough, but it's not a problem for me and it's automatically 
handled through the portage binary package updates

So the end result is that you can make efficient use of binary updates, 
but portage will still customise the odd package here or there where a 
local machine requires something which differs from the norm.  To my eye 
this keeps most of the benefits of an RPM/DEB style binary updater, with 
the flexibility of a per machine, customised USE flag gentoo installation?


> An setup for 100 equal webserver vm's could look like this:
>
> * run a normal Gentoo vm (tailored for the webserver appliance),
>    where do you do regular updates (emerge, revdep-rebuild, etc, etc)
> * from time to time take a snapshot, strip off the buildtime-only
>    stuff (hmm, could turn out to be a bit tricky ;-o)
> * this stripped snapshot now goes into testing vm's
> * when approved, the individual production vm's are switched over
>    to the new image (maybe using some mount magic, unionfs, etc)

This could work and perhaps for 100 identical Vms you have enough meat 
to work on something quite customised anyway?

Personally for 20-80 identical VMs running very limited variety of web 
software I would go for:
- Slightly cut down gentoo VM
- Hardlinked across all instances OR single installation which is read only
- Writeable data areas mounted to their own space (/var/www, /tmp, 
/home, etc)

By separating the data from the OS you have a lot of flexibility to 
upgrade the base webserver install and mount the data back on the new 
VM?  With linux-vservers or other container style, you will find that 
the OS shares code segments across all virtual machines (due to all 
files sharing the same inode) and the memory usage should be much lower 
and nearer to firing up an instance of the shared app and it then 
forking (ie data is duplicated, but the code segment is shared)


For 100+ Vms I guess I would be looking very strongly at a common 
read-only OS partition and container style virtualisation

For 20-80 near identical VMs, but running a wider variety of web 
software I would go for the hardlinked option with a straightforward 
"emerge" upgrade option across them.  Hardlinking keeps the memory usage 
sane where possible, without the pain of trying to keep the base install 
absolutely identical and read-only to make the common mount option work?


> At this point I've got a question for to the other folks here:
>
> emerge has an --root option which allows to (un)merge in a separate
> system image. So it should be possible to unmerge a lot of system
> packages which are just required for updating/building (even
> portage itself), but this still will be manual - what about
> dependency handling ?

This is correct.  In fact this is how you build a stage 1,2,3 etc and 
how catalyst works!

The information is a bit spread out over several out of date wiki 
articles, but perhaps start with:
     http://en.gentoo-wiki.com/wiki/Tiny_Gentoo

Roughly speaking you could "freshen" your current installation with 
(from memory):
     ROOT="/tmp/new_build" emerge -av world

This has minor gremlins when I test it, probably due to some symlinks 
being created differently if you follow the current catalyst build 
script through stage 1,2,3 etc, but roughly speaking it does the same 
thing only jumping straight to the end result and building a completely 
new identical install to your current OS...

Even more special is that you can set an alternative portage source, so 
if you want to build your new ROOT with alternative make.conf, 
/etc/portage/*, etc then just put your new files somewhere and set 
PORTAGE_CONFIGROOT to point to it.  Cross compiling is also done through 
an extension of this basic method

So, following your chain of thought - yes it's not too hard to quickly 
generate a customised base OS installation to use for your future VMs.  
Further, if you wish you can make those VMs have a reduced or missing 
toolchain etc.  In fact if you google a bit I think you will find some 
recipes for very minimal VMs using this method where the base VM is a 
very minimal install...

> Is there some way to drop at least parts of the standard system set,
> so eg. portage, python, gcc, etc, etc get unmerged by --depclean
> if nobody else (in world set) doesn't explicitly require them ?

You are almost thinking about it all wrong.  ("There is no spoon...")

This is gentoo, so at this more advanced level, stop thinking about 
"standard system set" and instead free your mind to start with 
"nothing".  Go read the old bootstrap from stage 1 instructions, plus 
the TinyGentoo pages and you can quickly see that Catalyst builds your 
working installation by starting from a working installation, creating 
an empty directory, adding some minimal packages to that directory and 
building up from there.

So absolutely nothing stops you from just starting with an empty 
directory and just emerging a few basic packages into it (couple MB) and 
then chrooting into it and having some fun...  There is *no* minimal 
package set, you can install whatever you want (as long as it boots). 
Largely the portage dependency tracker will help you pull in the minimal 
needed dependencies, but beware that system packages arent generally 
explicitly tracked so you may stumble across some deps when you are 
going really basic and omiting standard system packages (just use common 
sense: it should be fairly obvious if an application requires a compiler 
and you didn't install one then you have a conflict of interest...)


Have another look at gentoo!  I definitely believe that it's flexibility 
to build you highly customised packages, plus strong templating of those 
packages, plus decent ability to distribute binaries of the end result 
is a very strong combo!  Better binary support is really the only thing 
missing here, but it's pretty adequate as it stands!

Good luck

Ed W