[gentoo-alt] Re: 'Continuous Integration' for Gentoo Prefix?

[Sam Pfeiffer]

Hello,

I just wanted to say I'm very happy to see this project advancing and that
you found my work useful :) I've been offline for quite a while and it has
been great to find all this activity!

On Mon, Mar 18, 2019 at 11:53 PM Michael Haubenwallner <haubi@g.o>
wrote:

> Hi,
>
> based on Sammy's work I've set up another Azure build matrix behind a nice
> url:
> https://dev.azure.com/gentoo-prefix/ci-builds/_build
>
> This does bootstrap Gentoo Prefix (using self hosted Windows Server
> build agents for the long running Cygwin builds) in all these setups:
>
> ---------------------+---------------+---------------+
>                      |    Prefix     |      RAP      |
>  64bit OS            | 64bit | 32bit | 64bit | 32bit |
> ---------------------+-------+-------+-------+-------+
> Fedora current       |  yes  |  yes  |  yes  |  yes  |
> Fedora 28            |  yes  |  yes  |  yes  |  yes  |
> OpenSUSE current     |  yes  |  yes  |  yes  |  yes  |
> Ubuntu current       |   -   |   -   |  yes  |  yes  |
> Ubuntu 16.04         |   -   |   -   |  yes  |  yes  |
> MacOS current        |  yes  |   -   |   -   |   -   |
> Cygwin Server 2012R2 |  yes  |   -   |   -   |   -   |
> Cygwin Server 2016   |  yes  |   -   |   -   |   -   |
> Cygwin Server 2019   |  yes  |   -   |   -   |   -   |
> ---------------------+-------+-------+-------+-------+
>
> One Cygwin agent on Windows Server 2019 also does submit results to
> http://bootstrap.prefix.bitzolder.nl/results/
>
> Uploading docker images with bootstrapped /tmp/gentoo Prefix I'm happy
> to leave up to Sammy's jobs still.
>
> Thanks!
> /haubi/
>
> On 11/26/18 5:02 AM, Sam Pfeiffer wrote:
> > Hello,
> >
> > First, let me apologise if I have any wrong preexisting ideas/guesses
> about Gentoo Prefix and how it is developed. Secondly, sorry if this email
> is a bit too long.
> >
> > I'll present myself, explain my use-case and my troubles and tricks, and
> then ask for feedback.
> >
> > Hello, I'm Sammy Pfeiffer. I am a PhD student at University of
> Technology Sydney. I'm a (software) robotics engineer pursuing my PhD in a
> lab with a bunch of different robots and machine learning setups. I'm using
> Gentoo Prefix for deploying a big bunch of software into robots where the
> OS is old and frozen (and also to deploy in computation farms
> experiments... once again with no option to touch the OS). (The bunch of
> software is ROS, the 'Robotics Operating System' which has a ton of
> dependencies, and there is an existing overlay which I'm helping on
> maintaining and improving (https://github.com/ros/ros-overlay)).
> >
> > I'd like to share the trick I found to overcome a few annoying bits of
> my platforms:
> > * I have no root access in the machines I need to deploy.
> > * The machines don't have Docker installed (too old kernel anyways).
> > * I have 32 bit and 64 bit machines (the OS running in them is).
> > * I have a different username (and home folder) on each machine.
> > * Each machine has a different disk(s)/partition(s) structure (main
> issue here is that I may need to use a different disk to store my data,
> cause of size constrains).
> >
> > The trick to be able to use Gentoo Prefix with all these constraints
> bootstrapping it only once is to set the EPREFIX to /tmp/gentoo.
> > Then you can deploy the full bootstrapped system in any folder/disk and
> just do a softlink to /tmp/gentoo. This works nicely (I was scared of the
> softlink breaking stuff somewhere, but it was alright).
> > The other trick is to bootstrap a 32bit Gentoo Prefix, which can be run
> in any 32b or 64b box (it's not ideal but simplifies my deployment
> currently).
> >
> > Also, having all these machines use the same Gentoo Prefix in the same
> place, with the same 32bit compilation, I can setup a binary package
> server, which all the deployments can point to and just get all the
> packages skipping the long compilation times (specially in very old and low
> powered machines).
> >
> > The final trick I'm experimenting with is to use a set of Docker images
> (and soon in a continuous integration environment) to bootstrap all the
> system. With that I can save snapshot of successfully built systems &
> packages to serve as an easy deployment (and easy installation of extra
> packages thru the binary package server).
> >
> > Given my particular annoyance of the 32bit system (and noting that my
> hosts are 64bits, as is the standard nowadays) I found that I can use
> either:
> > * https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
> > * https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit
> Gentoo Docker image
> >
> > And then execute the build step of Docker with:
> >
> > setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .
> >
> > Which will trick any program trying to do uname -m to assume 32bit
> machine. (Previously I used the variable CHOST=i686-pc-linux-gnu for
> bootstrapping, but I found some problems, which didn't appear with this
> method).
> >
> > Once I have all this setup working nicely, I'd like to trigger rebuilds
> every X time, and on changes on main players (or all dependencies actually)
> of my setup, like the bootstrapping of Gentoo Prefix (Ideally, on every
> change of a part of the system just trigger a rebuild from that point on,
> Docker layers make this possible).
> >
> > With Azure announcing unlimited minutes on CI/CD for open source
> projects:
> >
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
> >
> > Even bootstrapping Gentoo prefix, with pieces of software like gcc
> taking very long to compile, is possible.
> >
> > The point is: I have been trying to build Gentoo Prefix during the last
> days after a few months of break since the last time I touched the system.
> And it's failing. I haven't managed yet to bootstrap it completely. I feel
> there is no CI/CD setup to catch these issues and be able to offer a
> working version of Gentoo Prefix at any time.
> >
> > I was going to build it for myself (cause I need it), but instead, I'd
> like to offer my help to build it for the community. At least offer as an
> option ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a
> Docker image, just copy the full folder structure, do the softlink, and you
> are ready to play with Gentoo. If you mess up anything, just re-deploy.
> >
> > To do this I'd need a bit of help as I'm quite new to Gentoo and I tend
> to get blocked on little issues that take a while to google or debug
> (specially with the long compilation times of the bootstrap and some big
> packages).
> >
> > I've posted a short issue in the Docker repo of Gentoo images about this
> (https://github.com/gentoo/gentoo-docker-images/issues/62) but I think
> the maintainers probably don't usually work with Gentoo Prefix.
> >
> > Thank you very much for your time, and for your open source efforts.
> >
> > --
> > *
> > *
> > *Sammy Pfeiffer*
> > PhD Candidate at The Magic Lab within UTS.
>
>

-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.