Re: [gentoo-alt] Re: 'Continuous Integration' for Gentoo Prefix?

[Sam Pfeiffer]

Hello,

Just wanted to update you up to where I got.

Now I have two working repositories:
* https://github.com/awesomebytes/gentoo_prefix_ci
* https://github.com/awesomebytes/gentoo_prefix_ci_32b

They have continuous integration setup with Azure pipelines where every
night they will try to bootstrap Gentoo Prefix on amd64 and x86 using
Docker images.

As the READMEs state, this is currently done in 3 steps. This is done for
two reasons. First, the 6h limit of one job running. Secondly, to be able
to have intermediate state Docker images to maybe try to fix the current
issues bootstrapping Gentoo Prefix in a more elegant way.

Using as an example the amd64 build:

On the releases section:
https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases one can find
.tar.gz files with the latest (currently done by hand, I'll automate that
soon on successful builds) Gentoo Prefix that bootstrapped. It's
bootstrapped in /tmp/gentoo and explained how to use it as I explained in
this email thread before.

On the builds page:
https://dev.azure.com/12719821/12719821/_build?definitionId=2 one can find
the full logs of every step.

This fulfils my immediate needs, now I'll need to spend some time doing
something similar to emerge all the stuff I need for ros_overlay
<https://github.com/ros/ros-overlay> and offer a binary repo. But I'm open
to talk about what I did, improve it, maybe move it somewhere else... You
let me know!

P.S.: Most of the work, if not all, is documented in bug #668940
<https://bugs.gentoo.org/668940> and more detailed and in order in this
notepad originally from Olivier Huber
<https://pad.crans.org/p/gentoo-prefix>.
P.S.2: The help I got from the people in the IRC at #gentoo-prefix was
great.


On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner <haubi@g.o>
wrote:

> On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> > On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <grobian@g.o
> <mailto:grobian@g.o>> wrote:
> >
> > > I don't want to depress this entire discussion, but it would be really
> > > nice if we could somehow interact with special machines people have at
> > > their company or at home.  Prefix needs testing on many different
> > > machines (non-Linux) which usually don't exist in docker images.
>
> I second this - and let me add a further aspect here:
> What I know from buildbot setup is that the master does provide (mostly
> shell)
> commands to be executed on the slave. This is fine as long as there is
> limited
> visibility for the master. But when a public buildbot master is being
> hijacked,
> it feels too easy to execute malicious commands even on the slave machines.
>
> So over a buildbot like setup, I would prefer a Jenkins like setup, where
> the
> master does provide only trigger information to slaves. And even more
> appealing
> would be a standalone slave setup, where the master does just receive the
> build
> logs for the public, without access to slave machines at all.
>
> > That's alright, we can use QEMU for some more esoteric hardware
> platforms,> if it's an OS that runs on a normal amd64/x86 computer a Docker
> image can be
> > built (I'm not an expert but there are images to learn how to do it).>
> Or in the worst case we can create an old-school VM for those weird OS
> > and automate the interaction with it (I did it for a robot by dumping all
> > disk once and creating a VM from it, it worked ok).
>
> Well... there's a bunch of OSs I fail to imagine the use of cloud driven
> hardware for them, like hppa-hpux or ia64-hpux for past ones, and ppc-aix,
> ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.
>
> > > That said, focussing on the (usually fast) boxes like this to catch
> > > dependency problems and more is useful.  In the case below it looks
> like
> > > the ld-wrapper is having issues.  Would it be possible to enter the
> > > environment for that failed run?
> >
> > Glad you see the use of it :) Yeah as I mentioned in the previous mail,
> > having docker installed in your machine, to access that exact environment
> > after the failed bootstrap just do:
> >
> > # This will download the image to your machine (it may take a bit of
> time if its the first time you use docker its around 1GB of data I think)
> > docker pull awesomebytes/gentoo_prefix_latest_image
> > # This will drop you into a shell in that environment, ready to play!
> > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
> >
> > When you are done you can just type exit.
>
> Nevertheless, having the breaking environment as a docker image where
> possible (true for the major OSs we support) really is awesome!
>
> /haubi/
>
>

-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.