Re: [gentoo-alt] Re: 'Continuous Integration' for Gentoo Prefix?

[Sam Pfeiffer]

That's great news tho!

I've been building upon my amd64 and x86 bootstrapped images (which keep
building nightly) and I've got a system that emerges 400 packages correctly
over this (well, I needed to add a few hacks on the way, which I'm
reporting them all and proposing patches when I know how).
Given there are 19000~ packages available, it does maybe not mean that
much, but I emerge everything up to showing 3D visualization qt windows
which has quite a few dependencies.



On Sat, Dec 15, 2018 at 5:37 AM Fabian Groffen <grobian@g.o> wrote:

> x86-solaris and x64-solaris bootstraps work, I expected the perl problem
> to show up for those.  The pipeline still seems working on this.
>
> On 11-12-2018 15:09:46 +0100, Fabian Groffen wrote:
> > Hi Sam,
> >
> > At this point I managed to get bootstrap to succeed on ppc-macos
> > (minority arch), I will continue with x64-macos and x64-solaris shortly.
> >
> > I thought I fixed the Perl problem, aparently not.  I'll release a new
> > snapshot today/tomorrow, perhaps this brings some of the fixes forward.
> >
> > Thanks,
> > Fabian
> >
> > On 12-12-2018 00:57:35 +1100, Sam Pfeiffer wrote:
> > > Hello Fabian,
> > >
> > > I was unaware I could do that. Looks cleaner and more convenient!
> Thanks!
> > >
> > > You could say it is 'by design'. I have no idea how to fix the perl
> stuff,
> > > neither the other problems (9 workarounds needed in total to be able to
> > > bootstrap Gentoo Prefix, documented here
> > > [1]https://pad.crans.org/p/gentoo-prefix and in my Dockerfiles).
> Given I need
> > > Gentoo Prefix bootstrappable to keep building stuff upon it... That's
> the best I
> > > could do.
> > >
> > > I'd love to remove the hacks and be able to just run the
> bootstrap-prefix.sh
> > > script and call it a day though.
> > >
> > > On Wed, Dec 12, 2018 at 12:40 AM Fabian Groffen <[2]grobian@g.o>
> wrote:
> > >
> > > > Hey Sam,
> > >
> > > > Examining some of your build logs, I wonder if you are aware you can
> do
> > > > this:
> > >
> > > >   ./bootstrap-prefix.sh ${EPREFIX} noninteractive
> > >
> > > > instead of forcing input to the script.
> > >
> > > > Perl currently fails it seems in the logs, though the bootstrap is
> > > > reported to be successful.  Is this by design?
> > >
> > > > Thanks,
> > > > Fabian
> > >
> > > > On 03-12-2018 11:53:02 +0100, Fabian Groffen wrote:
> > > > > Cool!  Thanks a lot!
> > > > >
> > > > > Fabian
> > > > >
> > > > > On 03-12-2018 21:48:15 +1100, Sam Pfeiffer wrote:
> > > > > > Hello,
> > > > > >
> > > > > > Just wanted to update you up to where I got.
> > > > > >
> > > > > > Now I have two working repositories:
> > > > > >
> > > > > > * [1][3]https://github.com/awesomebytes/gentoo_prefix_ci
> > > > > >
> > > > > > * [2][4]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > > > > >
> > > > > > They have continuous integration setup with Azure pipelines
> where every
> > > > night
> > > > > > they will try to bootstrap Gentoo Prefix on amd64 and x86 using
> Docker
> > > > images.
> > > > > >
> > > > > > As the READMEs state, this is currently done in 3 steps. This is
> done for
> > > > two
> > > > > > reasons. First, the 6h limit of one job running. Secondly, to be
> able to
> > > > have
> > > > > > intermediate state Docker images to maybe try to fix the current
> issues
> > > > > > bootstrapping Gentoo Prefix in a more elegant way.
> > > > > >
> > > > > > Using as an example the amd64 build:
> > > > > >
> > > > > > On the releases
> > > > > >
> > > > section: [3][5]
> https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > > > one
> > > > > > can find .tar.gz files with the latest (currently done by hand,
> I'll
> > > > automate
> > > > > > that soon on successful builds) Gentoo Prefix that bootstrapped.
> It's
> > > > > > bootstrapped in /tmp/gentoo and explained how to use it as I
> explained in
> > > > this
> > > > > > email thread before.
> > > > > >
> > > > > > On the builds
> > > > > > page: [4][6]
> https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > > > one can
> > > > > > find the full logs of every step.
> > > > > >
> > > > > > This fulfils my immediate needs, now I'll need to spend some
> time doing
> > > > > > something similar to emerge all the stuff I need for
> [5]ros_overlay and
> > > > offer a
> > > > > > binary repo. But I'm open to talk about what I did, improve it,
> maybe move
> > > > it
> > > > > > somewhere else... You let me know!
> > > > > >
> > > > > > P.S.: Most of the work, if not all, is documented in bug
> [6]#668940 and
> > > > more
> > > > > > detailed and in order in [7]this notepad originally from Olivier
> Huber.
> > > > > >
> > > > > > P.S.2: The help I got from the people in the IRC at
> #gentoo-prefix was
> > > > great.
> > > > > >
> > > > > > On Tue, Nov 27, 2018 at 8:21 PM Michael Haubenwallner
> > > > <[8][7]haubi@g.o>
> > > > > > wrote:
> > > > > >
> > > > > > > On 11/27/2018 09:37 AM, Sam Pfeiffer wrote:
> > > > > > > > On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen
> > > > <[9][8]grobian@g.o
> > > > > > > <mailto:[10][9]grobian@g.o>> wrote:
> > > > > > > >
> > > > > > > > > I don't want to depress this entire discussion, but it
> would be
> > > > really
> > > > > > > > > nice if we could somehow interact with special machines
> people have
> > > > at
> > > > > > > > > their company or at home.  Prefix needs testing on many
> different
> > > > > > > > > machines (non-Linux) which usually don't exist in docker
> images.
> > > > > >
> > > > > > > I second this - and let me add a further aspect here:
> > > > > > > What I know from buildbot setup is that the master does
> provide (mostly
> > > > shell)
> > > > > > > commands to be executed on the slave. This is fine as long as
> there is
> > > > limited
> > > > > > > visibility for the master. But when a public buildbot master
> is being
> > > > > > > hijacked,
> > > > > > > it feels too easy to execute malicious commands even on the
> slave
> > > > machines.
> > > > > >
> > > > > > > So over a buildbot like setup, I would prefer a Jenkins like
> setup,
> > > > where the
> > > > > > > master does provide only trigger information to slaves. And
> even more
> > > > > > > appealing
> > > > > > > would be a standalone slave setup, where the master does just
> receive
> > > > the
> > > > > > > build
> > > > > > > logs for the public, without access to slave machines at all.
> > > > > >
> > > > > > > > That's alright, we can use QEMU for some more esoteric
> hardware
> > > > platforms,>
> > > > > > > if it's an OS that runs on a normal amd64/x86 computer a
> Docker image
> > > > can be
> > > > > > > > built (I'm not an expert but there are images to learn how
> to do it).>
> > > > Or in
> > > > > > > the worst case we can create an old-school VM for those weird
> OS
> > > > > > > > and automate the interaction with it (I did it for a robot
> by dumping
> > > > all
> > > > > > > > disk once and creating a VM from it, it worked ok).
> > > > > >
> > > > > > > Well... there's a bunch of OSs I fail to imagine the use of
> cloud driven
> > > > > > > hardware for them, like hppa-hpux or ia64-hpux for past ones,
> and
> > > > ppc-aix,
> > > > > > > ppc-macos, sparc-solaris, arm-linux or m68k-mint for current
> ones.
> > > > > >
> > > > > > > > > That said, focussing on the (usually fast) boxes like this
> to catch
> > > > > > > > > dependency problems and more is useful.  In the case below
> it looks
> > > > like
> > > > > > > > > the ld-wrapper is having issues.  Would it be possible to
> enter the
> > > > > > > > > environment for that failed run?
> > > > > > > >
> > > > > > > > Glad you see the use of it :) Yeah as I mentioned in the
> previous
> > > > mail,
> > > > > > > > having docker installed in your machine, to access that exact
> > > > environment
> > > > > > > > after the failed bootstrap just do:
> > > > > > > >
> > > > > > > > # This will download the image to your machine (it may take
> a bit of
> > > > time if
> > > > > > > its the first time you use docker its around 1GB of data I
> think)
> > > > > > > > docker pull awesomebytes/gentoo_prefix_latest_image
> > > > > > > > # This will drop you into a shell in that environment, ready
> to play!
> > > > > > > > docker run -it awesomebytes/gentoo_prefix_latest_image
> /bin/bash
> > > > > > > >
> > > > > > > > When you are done you can just type exit.
> > > > > >
> > > > > > > Nevertheless, having the breaking environment as a docker
> image where
> > > > > > > possible (true for the major OSs we support) really is awesome!
> > > > > >
> > > > > > > /haubi/
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Sammy Pfeiffer
> > > > > > PhD Candidate at The Magic Lab within UTS.
> > > > > >
> > > > > >
> > > > > >
> > > > > >  References:
> > > > > >    1. [10]https://github.com/awesomebytes/gentoo_prefix_ci
> > > > > >    2. [11]https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > > > > >    3. [12]
> https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > > > > >    4. [13]
> https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > > > > >    5. [14]https://github.com/ros/ros-overlay
> > > > > >    6. [15]https://bugs.gentoo.org/668940
> > > > > >    7. [16]https://pad.crans.org/p/gentoo-prefix
> > > > > >    8. mailto:[17]haubi@g.o
> > > > > >    9. mailto:[18]grobian@g.o
> > > > > >   10. mailto:[19]grobian@g.o
> > > > > >
> > > > > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > > > > --
> > > > > Fabian Groffen
> > > > > Gentoo on a different level
> > >
> > > > --
> > > > Fabian Groffen
> > > > Gentoo on a different level
> > >
> > > --
> > >
> > > Sammy Pfeiffer
> > > PhD Candidate at The Magic Lab within UTS.
> > >
> > >
> > >
> > >  References:
> > >    1. https://pad.crans.org/p/gentoo-prefix
> > >    2. mailto:grobian@g.o
> > >    3. https://github.com/awesomebytes/gentoo_prefix_ci
> > >    4. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > >    5. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > >    6. https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > >    7. mailto:haubi@g.o
> > >    8. mailto:grobian@g.o
> > >    9. mailto:grobian@g.o
> > >   10. https://github.com/awesomebytes/gentoo_prefix_ci
> > >   11. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> > >   12. https://github.com/awesomebytes/gentoo_prefix_ci_32b/releases
> > >   13. https://dev.azure.com/12719821/12719821/_build?definitionId=2
> > >   14. https://github.com/ros/ros-overlay
> > >   15. https://bugs.gentoo.org/668940
> > >   16. https://pad.crans.org/p/gentoo-prefix
> > >   17. mailto:haubi@g.o
> > >   18. mailto:grobian@g.o
> > >   19. mailto:grobian@g.o
> > >
> > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > --
> > Fabian Groffen
> > Gentoo on a different level
>
>
>
> --
> Fabian Groffen
> Gentoo on a different level
>


-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.