Re: [gentoo-alt] 'Continuous Integration' for Gentoo Prefix?

[Fabian Groffen]

Hi Sam,

On 26-11-2018 15:02:26 +1100, Sam Pfeiffer wrote:
> Hello,
>
> First, let me apologise if I have any wrong preexisting ideas/guesses about
> Gentoo Prefix and how it is developed. Secondly, sorry if this email is a bit
> too long.
>
> I'll present myself, explain my use-case and my troubles and tricks, and then
> ask for feedback.
>
> Hello, I'm Sammy Pfeiffer. I am a PhD student at University of Technology
> Sydney. I'm a (software) robotics engineer pursuing my PhD in a lab with a bunch
> of different robots and machine learning setups. I'm using Gentoo Prefix for
> deploying a big bunch of software into robots where the OS is old and frozen
> (and also to deploy in computation farms experiments... once again with no
> option to touch the OS). (The bunch of software is ROS, the 'Robotics Operating
> System' which has a ton of dependencies, and there is an existing overlay which
> I'm helping on maintaining and improving
> ([1]https://github.com/ros/ros-overlay)).
>
> I'd like to share the trick I found to overcome a few annoying bits of my
> platforms:
>
> * I have no root access in the machines I need to deploy.
>
> * The machines don't have Docker installed (too old kernel anyways).
>
> * I have 32 bit and 64 bit machines (the OS running in them is).
>
> * I have a different username (and home folder) on each machine.
>
> * Each machine has a different disk(s)/partition(s) structure (main issue here
> is that I may need to use a different disk to store my data, cause of size
> constrains).
>
> The trick to be able to use Gentoo Prefix with all these constraints
> bootstrapping it only once is to set the EPREFIX to /tmp/gentoo.

That will work best, I presume.

> Then you can deploy the full bootstrapped system in any folder/disk and just do
> a softlink to /tmp/gentoo. This works nicely (I was scared of the softlink
> breaking stuff somewhere, but it was alright).
>
> The other trick is to bootstrap a 32bit Gentoo Prefix, which can be run in any
> 32b or 64b box (it's not ideal but simplifies my deployment currently).
>
> Also, having all these machines use the same Gentoo Prefix in the same place,
> with the same 32bit compilation, I can setup a binary package server, which all
> the deployments can point to and just get all the packages skipping the long
> compilation times (specially in very old and low powered machines).

Prefix branch knows a trick which can "relocate" packages.  The
constrains is that it can only "shorten" a prefix.  Thus, if you'd build
for /a/very/very/very/long/prefix you could install those binary
packages to /a/shorter/prefix.  I think this will also addresses your
userid problems (if any).

> The final trick I'm experimenting with is to use a set of Docker images (and
> soon in a continuous integration environment) to bootstrap all the system. With
> that I can save snapshot of successfully built systems & packages to serve as an
> easy deployment (and easy installation of extra packages thru the binary package
> server).

Neat!

> Given my particular annoyance of the 32bit system (and noting that my hosts are
> 64bits, as is the standard nowadays) I found that I can use either:
>
> * [2]https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
>
> * [3]https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit Gentoo
> Docker image
>
> And then execute the build step of Docker with:
>
> setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .
>
> Which will trick any program trying to do uname -m to assume 32bit machine.
> (Previously I used the variable CHOST=i686-pc-linux-gnu for bootstrapping, but I
> found some problems, which didn't appear with this method).

Hmmm.

> Once I have all this setup working nicely, I'd like to trigger rebuilds every X
> time, and on changes on main players (or all dependencies actually) of my setup,
> like the bootstrapping of Gentoo Prefix (Ideally, on every change of a part of
> the system just trigger a rebuild from that point on, Docker layers make this
> possible).

This may be a bit too much, but you could start from every day
(LATEST_TREE_YES), or we can see to building a tar more frequently from
rsync0.

> With Azure announcing unlimited minutes on CI/CD for open source projects: 
>
> [4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
> long to compile, is possible.

I'm still missing the infra to administer this, but I have a wrapper
script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
an unattended bootstrap.  I want this to run automatically on some boxes
I have, but some of them will take several days to complete.
Another thing is for a bootstrapped prefix to periodically emerge --sync
and emerge -Dua world.  Add on top of that detecting which packages have
keywords for said prefix and installing them, and we should be able to
be notice breakage rather sooner than later.

> The point is: I have been trying to build Gentoo Prefix during the last days
> after a few months of break since the last time I touched the system. And it's
> failing. I haven't managed yet to bootstrap it completely. I feel there is no
> CI/CD setup to catch these issues and be able to offer a working version of
> Gentoo Prefix at any time.

Sorry about that.  I need to make another pass over it.

> I was going to build it for myself (cause I need it), but instead, I'd like to
> offer my help to build it for the community. At least offer as an option
> ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a Docker image,
> just copy the full folder structure, do the softlink, and you are ready to play
> with Gentoo. If you mess up anything, just re-deploy.
>
> To do this I'd need a bit of help as I'm quite new to Gentoo and I tend to get
> blocked on little issues that take a while to google or debug (specially with
> the long compilation times of the bootstrap and some big packages).
>
> I've posted a short issue in the Docker repo of Gentoo images about this
> ([5]https://github.com/gentoo/gentoo-docker-images/issues/62) but I think the
> maintainers probably don't usually work with Gentoo Prefix.

I'm not very familiar myself with Docker (mostly hiding under a rock
here), but I suspect the gentoo-docker-images project responds better to
email to their list (as in Contributing from their README).  I may be
wrong though.

> Thank you very much for your time, and for your open source efforts.

Thanks for your efforts sofar.  I hope we can pull something together.
At least in the coming period I hope to be able to fix the bootstrapping
issues for Solaris/OpenIndiana/Darwin.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level