Re: [gentoo-alt] 'Continuous Integration' for Gentoo Prefix?

[Sam Pfeiffer]

On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <grobian@g.o> wrote:

> I don't want to depress this entire discussion, but it would be really
> nice if we could somehow interact with special machines people have at
> their company or at home.  Prefix needs testing on many different
> machines (non-Linux) which usually don't exist in docker images.
>

That's alright, we can use QEMU for some more esoteric hardware platforms,
if it's an OS that runs on a normal amd64/x86 computer a Docker image can
be built (I'm not an expert but there are images to learn how to do it). Or
in the worst case we can create an old-school VM for those weird OS and
automate the interaction with it (I did it for a robot by dumping all disk
once and creating a VM from it, it worked ok).


>
> That said, focussing on the (usually fast) boxes like this to catch
> dependency problems and more is useful.  In the case below it looks like
> the ld-wrapper is having issues.  Would it be possible to enter the
> environment for that failed run?
>

Glad you see the use of it :) Yeah as I mentioned in the previous mail,
having docker installed in your machine, to access that exact environment
after the failed bootstrap just do:

# This will download the image to your machine (it may take a bit of time
if its the first time you use docker its around 1GB of data I think)
docker pull awesomebytes/gentoo_prefix_latest_image
# This will drop you into a shell in that environment, ready to play!
docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash

When you are done you can just type exit.


>
> Thanks,
> Fabian
>
> On 27-11-2018 17:14:19 +1100, Sam Pfeiffer wrote:
> > Hello all,
> >
> > Another little update on my test, it took me a while to find the actual
> way to
> > increase the timeout to the maximum (or in other words, get out of the
> default
> > of 60minutes), but I finally found how.
> >
> > Now I have a job that just tries to bootstrap Gentoo Prefix, the last
> run I made
> > can be found
> > here: [1]
> https://dev.azure.com/12719821/12719821/_build/results?buildId=21&view=logs
> >
> > You can see all the log. In this case it failed after 1h35min. It failed
> > compiling GCC 8.2.0-r4.
> >
> > The error was:
> >
> > 2018-11-27T03:20:31.7540250Z
> >
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/xg++
> >
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/
> > -B/tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ -nostdinc++
> >
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
> >
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs
> > -isystem
> >
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu
> > -isystem
> >
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include
> > -isystem
> >
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0/libstdc++-v3/libsupc++
> >
> -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
> >
> -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs
> > -no-pie -fno-PIE    -m64 -O2 -pipe -O2 -pipe -gtoggle -DIN_GCC
> >  -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall
> -Wno-narrowing
> > -Wwrite-strings -Wcast-qual -Wmissing-format-attribute
> -Woverloaded-virtual
> > -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings
> >  -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc  gcov.o \
> >
> > 2018-11-27T03:20:31.7545947Z hash-table.o ggc-none.o libcommon.a
> > ../libcpp/libcpp.a   ../libbacktrace/.libs/libbacktrace.a
> > ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a  -o gcov
> >
> > 2018-11-27T03:20:31.7670930Z
> >
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/collect-ld:
> > 106: exec: /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld: Invalid argument
> >
> > 2018-11-27T03:20:31.8025313Z collect2: error: ld returned 2 exit status
> >
> > 2018-11-27T03:20:31.8026192Z Makefile:2935: recipe for target 'gcov'
> failed
> >
> > In the end of the log we get the usual:
> >
> > 2018-11-27T03:23:00.8458538Z  * ERROR: sys-devel/gcc-8.2.0-r4::gentoo
> failed
> > (compile phase):
> >
> > 2018-11-27T03:23:00.8480853Z  *   emake failed
> >
> > 2018-11-27T03:23:00.8501574Z  *
> >
> > 2018-11-27T03:23:00.8524914Z  * If you need support, post the output of
> `emerge
> > --info '=sys-devel/gcc-8.2.0-r4::gentoo'`,
> >
> > 2018-11-27T03:23:00.8550010Z  * the complete build log and the output of
> `emerge
> > -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'`.
> >
> > 2018-11-27T03:23:00.8572142Z  * The complete build log is located at
> > '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/build.log'.
> >
> > 2018-11-27T03:23:00.8593188Z  * The ebuild environment file is located at
> > '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/environment'.
> >
> > 2018-11-27T03:23:00.8622509Z  * Working directory:
> > '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build'
> >
> > 2018-11-27T03:23:00.8642956Z  * S:
> > '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0'
> >
> > 2018-11-27T03:23:02.9054285Z  *
> >
> > 2018-11-27T03:23:02.9078678Z  * Please include
> >
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-build-logs.tar.bz2
> > in your bug report.
> >
> > The cool thing comes now, how do you actually execute those commands to
> fill up
> > a bug report?
> >
> > Easy, in the job I upload a Docker image with the system exactly as it
> was after
> > the boostrap-prefix.sh command.
> >
> > So, if anyone wants to debug what is going on, they just need to do
> (this works
> > right now):
> >
> > docker pull awesomebytes/gentoo_prefix_latest_image
> >
> > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
> >
> > And you are inside of the box:
> >
> > > user@b70d59ff9703:~$ ls
> >
> > > bootstrap-prefix.sh  start_bootstrap_date.txt
> >
> > > user@b70d59ff9703:~$ cat start_bootstrap_date.txt
> >
> > > Tue Nov 27 01:52:20 UTC 2018
> >
> > > user@b70d59ff9703:~$ ls /tmp/gentoo
> >
> > > bin  etc  lib  lib64  run  sbin  stage1.log  stage2.log  stage3.log
> tmp  usr
> > > var
> >
> > So you can do those commands suggested by emerge:
> >
> > /tmp/gentoo/tmp/usr/bin/emerge --info '=sys-devel/gcc-8.2.0-r4::gentoo'
> -->
> > [2]https://pastebin.com/LWS3Bb7S
> >
> > /tmp/gentoo/tmp/usr/bin/emerge -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'
> > --> [3]https://pastebin.com/gipspmnD
> >
> > etc.
> >
> > Or... we could even generate a new bug report automatically from this
> will all
> > the necessary information, including the instructions on how to get into
> this
> > box with exactly that state for anyone to help on fixing it.
> >
> > I'll try to dig further in with things that I find useful (and I hope
> you also
> > find useful).
> >
> > Have a nice day!
> >
> > P.S.: With this I created a bug report easily: [4]
> https://bugs.gentoo.org/672042
> >
> > On Tue, Nov 27, 2018 at 2:07 AM Sam Pfeiffer <[5]sammypfeiffer@×××××.com>
> wrote:
> >
> > > Little update: The full build log is viewable to anyone with the link,
> so here
> > > you can see the progress of the current job:
> >
> > > [6]
> https://dev.azure.com/12719821/12719821/_build/results?buildId=17&view=logs
> >
> > > (Or I should say, the log of it, for whenever you open the link).
> >
> > > On Tue, Nov 27, 2018 at 2:02 AM Sam Pfeiffer <[7]
> sammypfeiffer@×××××.com>
> > > wrote:
> >
> > >> Hello everyone,
> >
> > >> I'm very excited to see you are interested in adding continuous
> integration!
> >
> > >> I don't know that much about continuous integration, I've only used
> it (with
> > >> systems already setup for me) with in-house Jenkins servers and with
> the ROS
> > >> buildfarm, based on Travis CI on Github. Also a little bit of Gitlab
> CI in my
> > >> lab.
> >
> > >> I did a bit of research/testing.
> >
> > >> Given it's quite a hassle to maintain custom machines, I'd try to use
> some of
> > >> the free for opensource CI services. I've checked the conditions of a
> few to
> > >> see which could fit better:
> >
> > >> * Gitlab CI: 2000 minutes / month
> >
> > >> * Travis CI: Unlimited minutes / month. But only 50 minutes long per
> step
> > >> (like per script executed).
> >
> > >> * Azure pipelines: Unlimited minutes / month. 360 minutes long per
> step (like
> > >> per script executed).
> >
> > >> There are probably many more, but those are the ones I knew about.
> >
> > >> Given that I wanted to give a try to Azure pipelines. And I did!
> >
> > >> I created this repo: [8]
> https://github.com/awesomebytes/gentoo_prefix_ci_test
> >
> > >> Where I activated Azure pipelines on it. After around 15min of
> reading the
> > >> docs and playing around with the web-gui I got my first pipeline
> running.
> >
> > >> As an initial setup I thought I would create a Docker image where I
> bootstrap
> > >> Gentoo Prefix from a Ubuntu 16.04 (as I'm familiar with it with my
> projects).
> >
> > >> The repo contains two important things:
> >
> > >> 1) The Dockerfile where I mainly trigger the
> > >> bootstrap: [9]
> https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
> >
> > >> 2) The configuration file for Azure pipelines on what to
> > >> do: [10]
> https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml
> >
> > >> I've implemented here that it tries to build Gentoo Prefix, and
> whatever the
> > >> result, it uploads a Docker image to my DockerHub account with the
> results.
> > >> This implies that:
> >
> > >> If the bootstrap is successful, one can just [docker pull] and
> [docker run]
> > >> the image to play with Gentoo Prefix.
> >
> > >> If the bootstrap is unsuccessful, one can just [docker pull] and
> [docker run]
> > >> to find oneself in the exact state of the system after the bootstrap
> command.
> > >> And one can recover the full console log from the Azure pipelines web
> > >> interface (even tho it would be nice to find out how to post it
> publicly
> > >> straight away).
> >
> > >> If all goes well in a few hours anyone will be able to find in my
> DockerHub
> > >> account said image (most probably the failed one), just doing:
> >
> > >> docker pull awesomebytes/gentoo_prefix_latest_image:latest
> >
> > >> docker run -it gentoo_prefix_latest_image /bin/bash
> >
> > >> You'll be inside of a Ubuntu 16.04 box with a user called 'user' and
> with all
> > >> the bootstrapped stuff in /tmp/gentoo.
> >
> > >> As curiosity, I checked the machines I got served as 'agents' to run
> my jobs,
> > >> and they were of the kind:
> >
> > >> CPUs: 2x Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
> >
> > >> RAM: 7GB
> >
> > >> Disk: 94GB free disk space
> >
> > >> More than enough to bootstrap Gentoo Prefix!
> >
> > >> I don't know if this is the way to go. But at least is interesting to
> have it
> > >> in mind.
> >
> > >> On Tue, Nov 27, 2018 at 12:01 AM Benda Xu <[11]heroxbd@g.o>
> wrote:
> >
> > >>> Hi Sam,
> >
> > >>> Sam Pfeiffer <[12]sammypfeiffer@×××××.com> writes:
> >
> > >>> > With Azure announcing unlimited minutes on CI/CD for open source
> > >>> > projects:
> > >>> >
> > >>> [13]
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
> > >>> >
> > >>> > Even bootstrapping Gentoo prefix, with pieces of software like gcc
> > >>> > taking very long to compile, is possible.
> > >>> >
> > >>> > The point is: I have been trying to build Gentoo Prefix during the
> > >>> > last days after a few months of break since the last time I touched
> > >>> > the system. And it's failing. I haven't managed yet to bootstrap it
> > >>> > completely. I feel there is no CI/CD setup to catch these issues
> and
> > >>> > be able to offer a working version of Gentoo Prefix at any time.
> >
> > >>> I completely agree with you.  I hope you can carry on this project to
> > >>> setup proper CI for Gentoo Prefix.  I am all in for help,
> portage/ebuild
> > >>> mentoring and coorperation.
> >
> > >>> A CI for Gentoo Prefix has been on my list for ages.  Thank you for
> > >>> triggering this.
> >
> > >>> Yours,
> > >>> Benda
> >
> > >> --
> >
> > >> Sammy Pfeiffer
> > >> PhD Candidate at The Magic Lab within UTS.
> >
> > > --
> >
> > > Sammy Pfeiffer
> > > PhD Candidate at The Magic Lab within UTS.
> >
> > --
> >
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
> >
> >
> >
> >  References:
> >    1.
> https://dev.azure.com/12719821/12719821/_build/results?buildId=21&amp;view=logs
> >    2. https://pastebin.com/LWS3Bb7S
> >    3. https://pastebin.com/gipspmnD
> >    4. https://bugs.gentoo.org/672042
> >    5. mailto:sammypfeiffer@×××××.com
> >    6.
> https://dev.azure.com/12719821/12719821/_build/results?buildId=17&amp;view=logs
> >    7. mailto:sammypfeiffer@×××××.com
> >    8. https://github.com/awesomebytes/gentoo_prefix_ci_test
> >    9.
> https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
> >   10.
> https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml
> >   11. mailto:heroxbd@g.o
> >   12. mailto:sammypfeiffer@×××××.com
> >   13.
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
> >
> > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> --
> Fabian Groffen
> Gentoo on a different level
>


-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.