Re: [gentoo-alt] Re: 'Continuous Integration' for Gentoo Prefix?

[Fabian Groffen]

Cool!  Thanks a lot!

Fabian

On 03-12-2018 21:48:15 +1100, Sam Pfeiffer wrote:
> Hello,
>
> Just wanted to update you up to where I got.
>
> Now I have two working repositories:
>
> * [1]https://github.com/awesomebytes/gentoo_prefix_ci
>
> * [2]https://github.com/awesomebytes/gentoo_prefix_ci_32b
>
> They have continuous integration setup with Azure pipelines where every night
> they will try to bootstrap Gentoo Prefix on amd64 and x86 using Docker images.
>
> As the READMEs state, this is currently done in 3 steps. This is done for two
> reasons. First, the 6h limit of one job running. Secondly, to be able to have
> intermediate state Docker images to maybe try to fix the current issues
> bootstrapping Gentoo Prefix in a more elegant way.
>
> Using as an example the amd64 build:
>
> On the releases
> section: [3]https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases one
> can find .tar.gz files with the latest (currently done by hand, I'll automate
> that soon on successful builds) Gentoo Prefix that bootstrapped. It's
> bootstrapped in /tmp/gentoo and explained how to use it as I explained in this
> email thread before.
>
> On the builds
> page: [4]https://dev.azure.com/12719821/12719821/_build?definitionId=2 one can
> find the full logs of every step.
>
> This fulfils my immediate needs, now I'll need to spend some time doing
> something similar to emerge all the stuff I need for [5]ros_overlay and offer a
> binary repo. But I'm open to talk about what I did, improve it, maybe move it
> somewhere else... You let me know!
>
> P.S.: Most of the work, if not all, is documented in bug [6]#668940 and more
> detailed and in order in [7]this notepad originally from Olivier Huber.
>
> P.S.2: The help I got from the people in the IRC at #gentoo-prefix was great.
>
> On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner <[8]haubi@g.o>
> wrote:
>
> > On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> > > On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <[9]grobian@g.o
> > <mailto:[10]grobian@g.o>> wrote:
> > >
> > > > I don't want to depress this entire discussion, but it would be really
> > > > nice if we could somehow interact with special machines people have at
> > > > their company or at home.  Prefix needs testing on many different
> > > > machines (non-Linux) which usually don't exist in docker images.
>
> > I second this - and let me add a further aspect here:
> > What I know from buildbot setup is that the master does provide (mostly shell)
> > commands to be executed on the slave. This is fine as long as there is limited
> > visibility for the master. But when a public buildbot master is being
> > hijacked,
> > it feels too easy to execute malicious commands even on the slave machines.
>
> > So over a buildbot like setup, I would prefer a Jenkins like setup, where the
> > master does provide only trigger information to slaves. And even more
> > appealing
> > would be a standalone slave setup, where the master does just receive the
> > build
> > logs for the public, without access to slave machines at all.
>
> > > That's alright, we can use QEMU for some more esoteric hardware platforms,>
> > if it's an OS that runs on a normal amd64/x86 computer a Docker image can be
> > > built (I'm not an expert but there are images to learn how to do it).> Or in
> > the worst case we can create an old-school VM for those weird OS
> > > and automate the interaction with it (I did it for a robot by dumping all
> > > disk once and creating a VM from it, it worked ok).
>
> > Well... there's a bunch of OSs I fail to imagine the use of cloud driven
> > hardware for them, like hppa-hpux or ia64-hpux for past ones, and ppc-aix,
> > ppc-macos, sparc-solaris, arm-linux or m68k-mint for current ones.
>
> > > > That said, focussing on the (usually fast) boxes like this to catch
> > > > dependency problems and more is useful.  In the case below it looks like
> > > > the ld-wrapper is having issues.  Would it be possible to enter the
> > > > environment for that failed run?
> > >
> > > Glad you see the use of it :) Yeah as I mentioned in the previous mail,
> > > having docker installed in your machine, to access that exact environment
> > > after the failed bootstrap just do:
> > >
> > > # This will download the image to your machine (it may take a bit of time if
> > its the first time you use docker its around 1GB of data I think)
> > > docker pull awesomebytes/gentoo_prefix_latest_image
> > > # This will drop you into a shell in that environment, ready to play!
> > > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
> > >
> > > When you are done you can just type exit.
>
> > Nevertheless, having the breaking environment as a docker image where
> > possible (true for the major OSs we support) really is awesome!
>
> > /haubi/
>
> --
>
> Sammy Pfeiffer
> PhD Candidate at The Magic Lab within UTS.
>
>
>
>  References:
>    1. https://github.com/awesomebytes/gentoo_prefix_ci
>    2. https://github.com/awesomebytes/gentoo_prefix_ci_32b
>    3. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
>    4. https://dev.azure.com/12719821/12719821/_build?definitionId=2
>    5. https://github.com/ros/ros-overlay
>    6. https://bugs.gentoo.org/668940
>    7. https://pad.crans.org/p/gentoo-prefix
>    8. mailto:haubi@g.o
>    9. mailto:grobian@g.o
>   10. mailto:grobian@g.o
>
> read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
-- 
Fabian Groffen
Gentoo on a different level