Gentoo Archives: gentoo-dev

From: Ed W <lists@××××××××××.com>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] avoiding urgent stabilizations
Date: Sun, 27 Feb 2011 11:04:56
In Reply to: Re: [gentoo-dev] avoiding urgent stabilizations by Enrico Weigelt
On 26/02/2011 15:57, Enrico Weigelt wrote:
> * Ed W<lists@××××××××××.com> schrieb: > >> I'm just building some embedded devices on the side using >> gentoo and my minimal builds are only a few MB? > How to do you get out all the buildtime stuff (portage, toolchain, etc) ? > >> Seems like your complaint is that you have gentoo installs which are >> full featured with a toolchain and portage, which you are comparing to >> an installation you built with a different tool that doesn't have a >> toolchain installed? > Well, with Briegel, I'm strictly differenciating between the build > host and the target image. Everything's crosscompiled by design. > So I can explicitly define which packages need to be in the final > image (and only those are built). The build host can by any fairly > recent GNU/Linux system (happens to be Gentoo at my site)
I think you are missing the point that the gentoo build system is quite happily building you a destination build with as little as one package in it? Whether that destination boots is a function of careful package choice, but the point is you are definitely not constrained from building a highly minimal target if you wish? Just from the top of my head I should imagine this would likely boot: mkdir -p /newroot/{proc,dev,sys} cp -a /dev/{null,zero} /newroot/dev/ emerge —root /newroot glibc shadow baselayout openrc util-linux udev If you want to go more minimal then I have a uclibc chroot that I use as a build environment, the first line of my build script for my minimal embedded device is simply: emerge --root /embedded baselayout uclibc busybox I haven't checked, but give or take creating /proc, /sys and some minimal devices in /dev, the above should boot fine. If you were booting into a "container" based virtualisation then you don't need to manage /dev and mounting devices and so you can get even more minimal than the top example I posted (ie I don't think you need udev, etc) Obviously such minimal builds will have minimal package management, but this may not be a problem if you have a super fixed configuration? Simply construct your environment such that the data is separate from the OS and also make sure any OS customisation is scriptable and now you would have a way to occasionally build a new image which you rolled out to all virtual machines
>> 1) Build your base images without a toolchain or portage and use a >> minimal package installer to install pre-built binary packages. >> This seems fraught with issues long term though... > How can this be done w/ Gentoo ? > AFAIK it always has a system set containing things like portage, > toolchain, etc, etc.
"System" is just a statement of some defaults. At the level you are working at you can largely ignore it. Build what you like into the target build....
> Would be still too much for my goals: I don't wanna have anything in > the final system that's not necessary for runtime. Building and even > installation should be driven from outside (actually, the final system > could even have /bin, /usr, etc mounted ro and /var w/o noexec)
OK, so sounds like a simple build script like the one above would suit you? For more advanced use consider learning to use catalyst or Funtoos build system? These are full on scriptable ways to repeatably build non trivial target builds. (remember they can build as little as single package targets or more complicated builds)
> Note that Gentoo's binary packages (which are practically snapshots > of what will be copied in by merge stage) are alway bound to specific > domains, eg. the full configuration (make.conf, useflags, etc, etc), > and there may be a lot of interdependencies between installed packages > (and versions), so it takes some care to use it in stable ways. > > This might be very fine for you (eg. if you have dozens of really > equal systems), but you'll have to be aware of the limitations.
Well, your requirements suggest that you don't need a package manager and you will instead build a new read-only OS build every so often and deploy that. However, in the interests of exploring the idea further: - By definition binary packages only work where you have equal systems? This is true of gentoo AND rpm/deb. - If you need binary packages for non equal systems then you need some kind of namespacing method to separate the binaries. Gentoo has no automated way to do this, but it's reasonably workable to have binary package repos and simply tell your guest machine to use the appropriate binary repo - Gentoo's binaries are slightly cleverer than some notice in that they have some awareness of when they don't match the target and the operator has the option to either ignore that and continue the install, OR to ignore the binary package and generate it's own binaries using the local compiler, etc. So, I don't really see the problem, in fact Gentoo binaries can be seen as slightly more flexible than RPMs to some extent if you allow them to be "optional".
> As long as you have the same make.conf/useflag/etc settings for all > systems (within a guest type), this might work well.
But that's true of Redhat/Debian/Gentoo? RPMs don't allow you to have a single binary repo where some of the packages are linked against perl 5.8 and some against 5.12? I guess it would be possible to enhance the packaging format to cater for that, but largely the whole premise of binary packages is that you lock down your target options and build all the systems identically? There is no easy way around this? (In fact it's what you want)
> At this point, Gentoo's binary packages could become tricky. > The big questions eg. are: can it handle multiple binpkgs of the > same (source) package with differing useflags ? How are version- > specific dependencies handled, etc, etc.
As it stands, no it can't. However, neither can RPM/DEB... Remember the premise of binaries is that they are all the same... Pick a system configuration and build all your binaries with that configuration. If you need to support multiple configurations then setup multiple binary repositories
>> articles, but perhaps start with: >> >> >> Roughly speaking you could "freshen" your current installation with >> (from memory): >> ROOT="/tmp/new_build" emerge -av world > Is this just the installation root or also the sysroot for the > toolchain ?
I think you need to read through the tiny_gentoo type wiki pages. The general build process is: 1) Start with a working machine 2) Build a build environment 3) Jump into the build environment and repeat step 2) to build a target environment 4) Jump into the target environment Now in the absence of cross compiling you can often skip step 2) and build the target environment directly, but it's normally useful to have that build environment so that you can customise things a bit. Also this will help explain some of the catalyst recipes and why you see everything built twice, once to get a build environment, then again to build the target build
>> Further, if you wish you can make those VMs have a reduced or missing >> toolchain etc. > hmm, do I have to unmerge certain packages explicitly or can I > tell it to only install those packages required for runtime in > the first place ?
Well your really simplest "build" would be to take a stage3/4 and simply "unmerge" anything you don't need... bit crude though However, from the email above you can see that it's more desirable to instead "build up", starting with an empty build and from there add the packages you actually want.
>> You are almost thinking about it all wrong. ("There is no spoon...") >> >> This is gentoo, so at this more advanced level, stop thinking about >> "standard system set" and instead free your mind to start with >> "nothing". > With "standard system set" I mean those packages which are in @system > on a particular arch. In my vision, @system would be simply empty, > instead @world tells which packages you want and the dependencies > are pulled in automatically.
Seriously, "there is no spoon"... If you want to make your own build script then there isn't even an @system...
> No idea if that's valid anymore, but some (long?) time ago, there > was the rule that individual packages weren't allowed to explicitly > DEPEND= on system packages. Obviously this wouldn't work with an > empty system set (you'd have to find out the deps and add them > to world set manually).
I believe this is correct. However, there aren't many packages in system and so whilst it's kind of buyer beware once you start chopping stuff out of system, the point is only if you don't install gzip or gcc then you need to beware stuff which depends on either. I think in practice manually managing those dependencies is going to be very straightforward, ie you are more going to fix the applications to not require the dependencies. After all if you don't want gcc, then you really don't want gcc, simple really... eg, if some simple application uses make to update it's config files then you will likely look at some workaround rather than installing make...
> Well, that's where it gets complicated: if some packages depend on some > library libfoo in @system (which I would empty in this case), portage > lacks information to compute the full dependency tree, so I'd have to > find them out and add them to @world manually. Exactly what I do not want.
Oh come on. You are working at the level of building your OS image from scratch and you don't know if any of your applications require (at runtime) gcc, bison, m4, wget, man, os-headers, portage or ssh? It seems likely that you can quickly determine if you can live without most of the above (at runtime)...?!!
> OTOH, Briegel doenst have that issue, as all dependencies are stated > explicitly in each package (well, with one exception: libc).
So, you can probably already use that information to figure out your minimal set of system packages that you likely need (for runtime). Just checking that Briegel tracks both runtime and build time dependencies separately? eg you might need gzip to build something, but not to run it? So far I think you could easily convert your briegel recipes to a simple ebuild script and run them side by side? It's actually such an easy thing to do, to build a new stage2/3/4 that it's a shame folks don't do it more! Good luck Ed W