Gentoo Archives: gentoo-dev

From: Ed W <lists@××××××××××.com>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] avoiding urgent stabilizations
Date: Sat, 26 Feb 2011 12:22:21
In Reply to: Re: [gentoo-dev] avoiding urgent stabilizations by Enrico Weigelt

> But, for me, even a trimmed-down Gentoo is still too large > (has to contain the whole base packages, from portage to > toolchain, includes, etc). I'd prefer having only the essential > runtime stuff within the containers.
I'm just building some embedded devices on the side using gentoo and my minimal builds are only a few MB? Curious why you feel you need to move from Gentoo to get the size smaller? Seems like your complaint is that you have gentoo installs which are full featured with a toolchain and portage, which you are comparing to an installation you built with a different tool that doesn't have a toolchain installed? However, you can do the same using gentoo if you wish? (you just need a lightweight package installer to avoid installing portage) I think your main options are: 1) Build your base images without a toolchain or portage and use a minimal package installer to install pre-built binary packages. This seems fraught with issues long term though... 2) Build your base images without a toolchain, but with portage (and perhaps a very minimal python). This gives you full dependency tracking and obviously bind mount/nfs mount the actual portage tree to avoid space used there. This seems workable and minimal? 3) If we are talking virtual machines then who cares if your containers are individually quite large, if the files in them are duplicated across all containers? Simply use an appropriate de-duplication strategy to coalesce the space and most of the disadvantages disappear? eg linux-vserver you can simply hardlink all the common files across your installations and allow the COW patch to break hardlinks if anyone alters a file in a single instance. Or you could use aufs to mount a writeable layer over your common base VM instance? Or you could use one of the filesystems which de-duplicates files in the background (some caveats apply here to avoid memory still being used multiple times in each VM). Or under KVM there is the memory coalescing feature which merges similar code pages (forget it's name?) Personally I think option 3) is quite interesting from the medium number of virtual machines, ie in the 10s to hundreds, ie simply don't worry about it, let the OS do the work. In the hundreds to thousands plus level I guess you have unique challenges and I would be wrong to try and suggest a solution from the comfort of a laptop without having that responsibility, but I would have thought there was some advantage in a very rigidly deployed base OS generated and updated very precisely?
> For this we need a different approach (strictly separating build > and production environments). Binary distros (eg. Debian) might > be one option, but they're lacking the configurability and mostly > are still too large. So I'm going a different route using my own > buildsystem - called Briegel - which originally was designed for > embedded/small-device targets. > > For now I didn't have the spare time to port all the packages > required for complete server systems (most of it is making > them all cleanly crosscompile'able, as this is a fundamental > concept of Briegel). But maybe you'd like to join in and try it :)
Sounds like an interesting challenge, but I'm unconvinced you can't solve 90% of your problem within the constraints of Gentoo? This saves you a bunch of time that could be invested in the last 10% through more traditional means?
>> It does appear like managing large numbers of virtual machines is one >> are that gentoo could score very well? Interested to see any chatter on >> how others solve this problem, or any general advocacy? Probably we >> should start a new thread though... > I'm not sure if Gentoo really is the right distro for that purpose, > as it's targeted to very different systems (i.g. Gentoo boxes are > expected to be quite unique, beginning with different per-package > useflags, even down to cflags, etc). But it might still be a good > basis for building specific system images (let's call them stage5 ;-))
I won't disagree on your "where it's targeted", but just to re-iterate why I think Gentoo works well is that it does have a very workable binary package feature! My way of working is to use (several) shared binary package repos and the guests largely pull from those shared package directories. In fact what I do is have a minimal number of shared "/usr/portage/package" directories and I mount an appropriate one to the guest type at boot time. At the moment my main two options are "32bit" and "64bit" for the package mounts, but I recently introduced a new machine type which is held back to perl 5.8 and that guest gets it's own package mount since it's obviously linking a lot of binaries differently So, my process is to test an update on a small number of guests, either dedicated test guests or less important live guests. If this looks good then I run the upgrade against all other Vms of the same type and they will update quickly from package binaries Now, the icing is that this works extremely well even once you decide to lightly customise machine types. So for example my binary packages are very high level (eg 32/64bit), my "profiles" would be fairly high level, eg I have www-apache and www-nginx base profiles. However, a specific virtual machine running say nginx might itself need a specific PHP application installed, and that itself might need some dependencies, which in turn might require a specific set of customisation of use flags and versions. Now, the neat thing is that the binary upgrade options are *either* to use *only* binary packages, OR to use binary packages *if* they were built with the correct USE flags. So for example I haven't bothered to split out my packages directory to be specific to the nginx/apache machines, however, this causes the PHP package to be regularly rebuilt depending on whether it was last used to upgrade an nginx or apache guest (different use flags needed for each guest). I could fix this easily enough, but it's not a problem for me and it's automatically handled through the portage binary package updates So the end result is that you can make efficient use of binary updates, but portage will still customise the odd package here or there where a local machine requires something which differs from the norm. To my eye this keeps most of the benefits of an RPM/DEB style binary updater, with the flexibility of a per machine, customised USE flag gentoo installation?
> An setup for 100 equal webserver vm's could look like this: > > * run a normal Gentoo vm (tailored for the webserver appliance), > where do you do regular updates (emerge, revdep-rebuild, etc, etc) > * from time to time take a snapshot, strip off the buildtime-only > stuff (hmm, could turn out to be a bit tricky ;-o) > * this stripped snapshot now goes into testing vm's > * when approved, the individual production vm's are switched over > to the new image (maybe using some mount magic, unionfs, etc)
This could work and perhaps for 100 identical Vms you have enough meat to work on something quite customised anyway? Personally for 20-80 identical VMs running very limited variety of web software I would go for: - Slightly cut down gentoo VM - Hardlinked across all instances OR single installation which is read only - Writeable data areas mounted to their own space (/var/www, /tmp, /home, etc) By separating the data from the OS you have a lot of flexibility to upgrade the base webserver install and mount the data back on the new VM? With linux-vservers or other container style, you will find that the OS shares code segments across all virtual machines (due to all files sharing the same inode) and the memory usage should be much lower and nearer to firing up an instance of the shared app and it then forking (ie data is duplicated, but the code segment is shared) For 100+ Vms I guess I would be looking very strongly at a common read-only OS partition and container style virtualisation For 20-80 near identical VMs, but running a wider variety of web software I would go for the hardlinked option with a straightforward "emerge" upgrade option across them. Hardlinking keeps the memory usage sane where possible, without the pain of trying to keep the base install absolutely identical and read-only to make the common mount option work?
> At this point I've got a question for to the other folks here: > > emerge has an --root option which allows to (un)merge in a separate > system image. So it should be possible to unmerge a lot of system > packages which are just required for updating/building (even > portage itself), but this still will be manual - what about > dependency handling ?
This is correct. In fact this is how you build a stage 1,2,3 etc and how catalyst works! The information is a bit spread out over several out of date wiki articles, but perhaps start with: Roughly speaking you could "freshen" your current installation with (from memory): ROOT="/tmp/new_build" emerge -av world This has minor gremlins when I test it, probably due to some symlinks being created differently if you follow the current catalyst build script through stage 1,2,3 etc, but roughly speaking it does the same thing only jumping straight to the end result and building a completely new identical install to your current OS... Even more special is that you can set an alternative portage source, so if you want to build your new ROOT with alternative make.conf, /etc/portage/*, etc then just put your new files somewhere and set PORTAGE_CONFIGROOT to point to it. Cross compiling is also done through an extension of this basic method So, following your chain of thought - yes it's not too hard to quickly generate a customised base OS installation to use for your future VMs. Further, if you wish you can make those VMs have a reduced or missing toolchain etc. In fact if you google a bit I think you will find some recipes for very minimal VMs using this method where the base VM is a very minimal install...
> Is there some way to drop at least parts of the standard system set, > so eg. portage, python, gcc, etc, etc get unmerged by --depclean > if nobody else (in world set) doesn't explicitly require them ?
You are almost thinking about it all wrong. ("There is no spoon...") This is gentoo, so at this more advanced level, stop thinking about "standard system set" and instead free your mind to start with "nothing". Go read the old bootstrap from stage 1 instructions, plus the TinyGentoo pages and you can quickly see that Catalyst builds your working installation by starting from a working installation, creating an empty directory, adding some minimal packages to that directory and building up from there. So absolutely nothing stops you from just starting with an empty directory and just emerging a few basic packages into it (couple MB) and then chrooting into it and having some fun... There is *no* minimal package set, you can install whatever you want (as long as it boots). Largely the portage dependency tracker will help you pull in the minimal needed dependencies, but beware that system packages arent generally explicitly tracked so you may stumble across some deps when you are going really basic and omiting standard system packages (just use common sense: it should be fairly obvious if an application requires a compiler and you didn't install one then you have a conflict of interest...) Have another look at gentoo! I definitely believe that it's flexibility to build you highly customised packages, plus strong templating of those packages, plus decent ability to distribute binaries of the end result is a very strong combo! Better binary support is really the only thing missing here, but it's pretty adequate as it stands! Good luck Ed W


Subject Author
Re: [gentoo-dev] avoiding urgent stabilizations Enrico Weigelt <weigelt@×××××.de>