Re: [gentoo-dev] avoiding urgent stabilizations

[Ed W]

On 26/02/2011 15:57, Enrico Weigelt wrote:
> * Ed W<lists@××××××××××.com>  schrieb:
>
>> I'm just building some embedded devices on the side using
>> gentoo and my minimal builds are only a few MB?
> How to do you get out all the buildtime stuff (portage, toolchain, etc) ?
>
>> Seems like your complaint is that you have gentoo installs which are
>> full featured with a toolchain and portage, which you are comparing to
>> an installation you built with a different tool that doesn't have a
>> toolchain installed?
> Well, with Briegel, I'm strictly differenciating between the build
> host and the target image. Everything's crosscompiled by design.
> So I can explicitly define which packages need to be in the final
> image (and only those are built). The build host can by any fairly
> recent GNU/Linux system (happens to be Gentoo at my site)

I think you are missing the point that the gentoo build system is quite 
happily building you a destination build with as little as one package 
in it? Whether that destination boots is a function of careful package 
choice, but the point is you are definitely not constrained from 
building a highly minimal target if you wish?

Just from the top of my head I should imagine this would likely boot:

mkdir -p /newroot/{proc,dev,sys}
cp -a /dev/{null,zero} /newroot/dev/
emerge —root /newroot glibc shadow baselayout openrc util-linux udev


If you want to go more minimal then I have a uclibc chroot that I use as 
a build environment, the first line of my build script for my minimal 
embedded device is simply:

emerge --root /embedded baselayout uclibc busybox

I haven't checked, but give or take creating /proc, /sys and some 
minimal devices in /dev, the above should boot fine.



If you were booting into a "container" based virtualisation then you 
don't need to manage /dev and mounting devices and so you can get even 
more minimal than the top example I posted (ie I don't think you need 
udev, etc)

Obviously such minimal builds will have minimal package management, but 
this may not be a problem if you have a super fixed configuration? 
Simply construct your environment such that the data is separate from 
the OS and also make sure any OS customisation is scriptable and now you 
would have a way to occasionally build a new image which you rolled out 
to all virtual machines


>> 1) Build your base images without a toolchain or portage and use a
>> minimal package installer to install pre-built binary packages.
>> This seems fraught with issues long term though...
> How can this be done w/ Gentoo ?
> AFAIK it always has a system set containing things like portage,
> toolchain, etc, etc.

"System" is just a statement of some defaults. At the level you are 
working at you can largely ignore it.

Build what you like into the target build....


> Would be still too much for my goals: I don't wanna have anything in
> the final system that's not necessary for runtime. Building and even
> installation should be driven from outside (actually, the final system
> could even have /bin, /usr, etc mounted ro and /var w/o noexec)

OK, so sounds like a simple build script like the one above would suit you?

For more advanced use consider learning to use catalyst or Funtoos build 
system? These are full on scriptable ways to repeatably build non 
trivial target builds. (remember they can build as little as single 
package targets or more complicated builds)


> Note that Gentoo's binary packages (which are practically snapshots
> of what will be copied in by merge stage) are alway bound to specific
> domains, eg. the full configuration (make.conf, useflags, etc, etc),
> and there may be a lot of interdependencies between installed packages
> (and versions), so it takes some care to use it in stable ways.
>
> This might be very fine for you (eg. if you have dozens of really
> equal systems), but you'll have to be aware of the limitations.

Well, your requirements suggest that you don't need a package manager 
and you will instead build a new read-only OS build every so often and 
deploy that. However, in the interests of exploring the idea further:

- By definition binary packages only work where you have equal systems? 
This is true of gentoo AND rpm/deb.
- If you need binary packages for non equal systems then you need some 
kind of namespacing method to separate the binaries. Gentoo has no 
automated way to do this, but it's reasonably workable to have binary 
package repos and simply tell your guest machine to use the appropriate 
binary repo
- Gentoo's binaries are slightly cleverer than some notice in that they 
have some awareness of when they don't match the target and the operator 
has the option to either ignore that and continue the install, OR to 
ignore the binary package and generate it's own binaries using the local 
compiler, etc.

So, I don't really see the problem, in fact Gentoo binaries can be seen 
as slightly more flexible than RPMs to some extent if you allow them to 
be "optional".


> As long as you have the same make.conf/useflag/etc settings for all
> systems (within a guest type), this might work well.

But that's true of Redhat/Debian/Gentoo?

RPMs don't allow you to have a single binary repo where some of the 
packages are linked against perl 5.8 and some against 5.12? I guess it 
would be possible to enhance the packaging format to cater for that, but 
largely the whole premise of binary packages is that you lock down your 
target options and build all the systems identically? There is no easy 
way around this? (In fact it's what you want)


> At this point, Gentoo's binary packages could become tricky.
> The big questions eg. are: can it handle multiple binpkgs of the
> same (source) package with differing useflags ? How are version-
> specific dependencies handled, etc, etc.

As it stands, no it can't. However, neither can RPM/DEB...

Remember the premise of binaries is that they are all the same... Pick a 
system configuration and build all your binaries with that 
configuration. If you need to support multiple configurations then setup 
multiple binary repositories

>> articles, but perhaps start with:
>>      http://en.gentoo-wiki.com/wiki/Tiny_Gentoo
>>
>> Roughly speaking you could "freshen" your current installation with
>> (from memory):
>>      ROOT="/tmp/new_build" emerge -av world
> Is this just the installation root or also the sysroot for the
> toolchain ?

I think you need to read through the tiny_gentoo type wiki pages. The 
general build process is:

1) Start with a working machine
2) Build a build environment
3) Jump into the build environment and repeat step 2) to build a target 
environment
4) Jump into the target environment

Now in the absence of cross compiling you can often skip step 2) and 
build the target environment directly, but it's normally useful to have 
that build environment so that you can customise things a bit.

Also this will help explain some of the catalyst recipes and why you see 
everything built twice, once to get a build environment, then again to 
build the target build


>> Further, if you wish you can make those VMs have a reduced or missing
>> toolchain etc.
> hmm, do I have to unmerge certain packages explicitly or can I
> tell it to only install those packages required for runtime in
> the first place ?

Well your really simplest "build" would be to take a stage3/4 and simply 
"unmerge" anything you don't need... bit crude though

However, from the email above you can see that it's more desirable to 
instead "build up", starting with an empty build and from there add the 
packages you actually want.


>> You are almost thinking about it all wrong.  ("There is no spoon...")
>>
>> This is gentoo, so at this more advanced level, stop thinking about
>> "standard system set" and instead free your mind to start with
>> "nothing".
> With "standard system set" I mean those packages which are in @system
> on a particular arch. In my vision, @system would be simply empty,
> instead @world tells which packages you want and the dependencies
> are pulled in automatically.

Seriously, "there is no spoon"...

If you want to make your own build script then there isn't even an 
@system...

> No idea if that's valid anymore, but some (long?) time ago, there
> was the rule that individual packages weren't allowed to explicitly
> DEPEND= on system packages. Obviously this wouldn't work with an
> empty system set (you'd have to find out the deps and add them
> to world set manually).

I believe this is correct. However, there aren't many packages in system 
and so whilst it's kind of buyer beware once you start chopping stuff 
out of system, the point is only if you don't install gzip or gcc then 
you need to beware stuff which depends on either.

I think in practice manually managing those dependencies is going to be 
very straightforward, ie you are more going to fix the applications to 
not require the dependencies. After all if you don't want gcc, then you 
really don't want gcc, simple really... eg, if some simple application 
uses make to update it's config files then you will likely look at some 
workaround rather than installing make...


> Well, that's where it gets complicated: if some packages depend on some
> library libfoo in @system (which I would empty in this case), portage
> lacks information to compute the full dependency tree, so I'd have to
> find them out and add them to @world manually. Exactly what I do not want.

Oh come on. You are working at the level of building your OS image from 
scratch and you don't know if any of your applications require (at 
runtime) gcc, bison, m4, wget, man, os-headers, portage or ssh?

It seems likely that you can quickly determine if you can live without 
most of the above (at runtime)...?!!

> OTOH, Briegel doenst have that issue, as all dependencies are stated
> explicitly in each package (well, with one exception: libc).

So, you can probably already use that information to figure out your 
minimal set of system packages that you likely need (for runtime).

Just checking that Briegel tracks both runtime and build time 
dependencies separately? eg you might need gzip to build something, but 
not to run it?


So far I think you could easily convert your briegel recipes to a simple 
ebuild script and run them side by side? It's actually such an easy 
thing to do, to build a new stage2/3/4 that it's a shame folks don't do 
it more!

Good luck

Ed W