Re: [gentoo-dev] Portage FEATURE suggestion - limited-visibility builds

[Michael Mol]

On Thu, Jul 26, 2012 at 2:26 PM, Rich Freeman <rich0@g.o> wrote:
> I've been messing around with namespaces and some of what systemd has
> been doing with them, and I have an idea for a portage feature.
>
> But before doing a brain dump of ideas, how useful would it be to have
> a FEATURE for portage to do a limited-visibility build?  That is, the
> build would be run in an environment where the root filesystem appears
> to contain everything in a DEPEND (including @system currently) and
> nothing else?  It might be useful both in development/testing, and
> also in production use (not sure how performance would work in the
> real world - I was able in a script to get it to build an enviornment
> in a few seconds for a few packages).

I very much like this, as it'd greatly simplify identifying any
unintended or unrecognized dependencies in my code. Furthermore, if
the mechanism for identifying and declaring specified-required content
can be generalized, this would make distributed builds potentially
more efficient by allowing the dispatching host to distribute the
necessary header and library files to the machines doing the building.
(Really, this observation is more about simply making the information
available; distcc could consume that information if someone chose to
do the work to add that functionality.)

> I really crazy idea would be to try to run packages in a similar
> environment, but I think that needs better kernel/etc level support
> since the performance hit would be much more noticeable, except for
> things like daemons that only start once.
>
> Implementing it wouldn't necessarily be hard - just create a tmpfs
> under /var/tmp/portage, unshare off a new mount namespace, and
> read-only bind-mount everything needed from the root filesystem
> (including /var/tmp/portage/...), and chroot into it.  When the build
> is done the process governing it terminates and the kernel wipes out
> all the mounts and then portage unmounts the tmpfs.  You wouldn't need
> to use a tmpfs for the build - it would actually be zero-size as
> reported by df since it just contains a bazillion bind mounts, though
> all those mounts would consume slab memory.
>
> Thoughts?

You've done 90% of the conceptual work needed for an idea I had; I've
wanted to do something similar at the 'make' level, to make
identifying and fixing broken parallel build systems easier. If I
could limit a make instance to only be able to see consequences of
jobs it declared as dependencies, that'd go a long way. I was going to
go by way of FUSE for this, though...

-- 
:wq