Re: [gentoo-alt] 'Continuous Integration' for Gentoo Prefix?

[Sam Pfeiffer]

Hello all,

Another little update on my test, it took me a while to find the actual way
to increase the timeout to the maximum (or in other words, get out of the
default of 60minutes), but I finally found how.

Now I have a job that just tries to bootstrap Gentoo Prefix, the last run I
made can be found here:
https://dev.azure.com/12719821/12719821/_build/results?buildId=21&view=logs
You can see all the log. In this case it failed after 1h35min. It failed
compiling GCC 8.2.0-r4.

The error was:

2018-11-27T03:20:31.7540250Z
/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/xg++
-B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/
-B/tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ -nostdinc++
-B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
-B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs
-isystem
/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu
-isystem
/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include
-isystem
/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0/libstdc++-v3/libsupc++
-L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
-L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs
-no-pie -fno-PIE    -m64 -O2 -pipe -O2 -pipe -gtoggle -DIN_GCC
 -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall
-Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute
-Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros
-Wno-overlength-strings   -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc
gcov.o \
2018-11-27T03:20:31.7545947Z hash-table.o ggc-none.o libcommon.a
../libcpp/libcpp.a   ../libbacktrace/.libs/libbacktrace.a
../libiberty/libiberty.a ../libdecnumber/libdecnumber.a  -o gcov
2018-11-27T03:20:31.7670930Z
/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/collect-ld:
106: exec: /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld: Invalid argument
2018-11-27T03:20:31.8025313Z collect2: error: ld returned 2 exit status
2018-11-27T03:20:31.8026192Z Makefile:2935: recipe for target 'gcov' failed

In the end of the log we get the usual:

2018-11-27T03:23:00.8458538Z  * ERROR: sys-devel/gcc-8.2.0-r4::gentoo
failed (compile phase):
2018-11-27T03:23:00.8480853Z  *   emake failed
2018-11-27T03:23:00.8501574Z  *
2018-11-27T03:23:00.8524914Z  * If you need support, post the output of
`emerge --info '=sys-devel/gcc-8.2.0-r4::gentoo'`,
2018-11-27T03:23:00.8550010Z  * the complete build log and the output of
`emerge -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'`.
2018-11-27T03:23:00.8572142Z  * The complete build log is located at
'/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/build.log'.
2018-11-27T03:23:00.8593188Z  * The ebuild environment file is located at
'/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/environment'.
2018-11-27T03:23:00.8622509Z  * Working directory:
'/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build'
2018-11-27T03:23:00.8642956Z  * S:
'/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0'
2018-11-27T03:23:02.9054285Z  *
2018-11-27T03:23:02.9078678Z  * Please include
/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-build-logs.tar.bz2
in your bug report.

The cool thing comes now, how do you actually execute those commands to
fill up a bug report?

Easy, in the job I upload a Docker image with the system exactly as it was
after the boostrap-prefix.sh command.

So, if anyone wants to debug what is going on, they just need to do (this
works right now):

docker pull awesomebytes/gentoo_prefix_latest_image
docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash

And you are inside of the box:

user@b70d59ff9703:~$ ls
bootstrap-prefix.sh  start_bootstrap_date.txt
user@b70d59ff9703:~$ cat start_bootstrap_date.txt
Tue Nov 27 01:52:20 UTC 2018
user@b70d59ff9703:~$ ls /tmp/gentoo
bin  etc  lib  lib64  run  sbin  stage1.log  stage2.log  stage3.log  tmp
usr  var


So you can do those commands suggested by emerge:
/tmp/gentoo/tmp/usr/bin/emerge --info '=sys-devel/gcc-8.2.0-r4::gentoo' -->
https://pastebin.com/LWS3Bb7S
/tmp/gentoo/tmp/usr/bin/emerge -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'  -->
https://pastebin.com/gipspmnD
etc.

Or... we could even generate a new bug report automatically from this will
all the necessary information, including the instructions on how to get
into this box with exactly that state for anyone to help on fixing it.

I'll try to dig further in with things that I find useful (and I hope you
also find useful).

Have a nice day!

P.S.: With this I created a bug report easily:
https://bugs.gentoo.org/672042


On Tue, Nov 27, 2018 at 2:07 AM Sam Pfeiffer <sammypfeiffer@×××××.com>
wrote:

> Little update: The full build log is viewable to anyone with the link, so
> here you can see the progress of the current job:
>
> https://dev.azure.com/12719821/12719821/_build/results?buildId=17&view=logs
>
> (Or I should say, the log of it, for whenever you open the link).
>
>
> On Tue, Nov 27, 2018 at 2:02 AM Sam Pfeiffer <sammypfeiffer@×××××.com>
> wrote:
>
>> Hello everyone,
>>
>> I'm very excited to see you are interested in adding continuous
>> integration!
>>
>> I don't know that much about continuous integration, I've only used it
>> (with systems already setup for me) with in-house Jenkins servers and with
>> the ROS buildfarm, based on Travis CI on Github. Also a little bit of
>> Gitlab CI in my lab.
>>
>> I did a bit of research/testing.
>>
>> Given it's quite a hassle to maintain custom machines, I'd try to use
>> some of the free for opensource CI services. I've checked the conditions of
>> a few to see which could fit better:
>>
>> * Gitlab CI: 2000 minutes / month
>> * Travis CI: Unlimited minutes / month. But only 50 minutes long per step
>> (like per script executed).
>> * Azure pipelines: Unlimited minutes / month. 360 minutes long per step
>> (like per script executed).
>>
>> There are probably many more, but those are the ones I knew about.
>>
>> Given that I wanted to give a try to Azure pipelines. And I did!
>>
>> I created this repo:
>> https://github.com/awesomebytes/gentoo_prefix_ci_test
>>
>> Where I activated Azure pipelines on it. After around 15min of reading
>> the docs and playing around with the web-gui I got my first pipeline
>> running.
>>
>> As an initial setup I thought I would create a Docker image where I
>> bootstrap Gentoo Prefix from a Ubuntu 16.04 (as I'm familiar with it with
>> my projects).
>>
>> The repo contains two important things:
>> 1) The Dockerfile where I mainly trigger the bootstrap:
>> https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
>> 2) The configuration file for Azure pipelines on what to do:
>> https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml
>>
>> I've implemented here that it tries to build Gentoo Prefix, and whatever
>> the result, it uploads a Docker image to my DockerHub account with the
>> results. This implies that:
>> If the bootstrap is successful, one can just [docker pull] and [docker
>> run] the image to play with Gentoo Prefix.
>> If the bootstrap is unsuccessful, one can just [docker pull] and [docker
>> run] to find oneself in the exact state of the system after the bootstrap
>> command. And one can recover the full console log from the Azure pipelines
>> web interface (even tho it would be nice to find out how to post it
>> publicly straight away).
>>
>> If all goes well in a few hours anyone will be able to find in my
>> DockerHub account said image (most probably the failed one), just doing:
>> docker pull awesomebytes/gentoo_prefix_latest_image:latest
>> docker run -it gentoo_prefix_latest_image /bin/bash
>> You'll be inside of a Ubuntu 16.04 box with a user called 'user' and with
>> all the bootstrapped stuff in /tmp/gentoo.
>>
>> As curiosity, I checked the machines I got served as 'agents' to run my
>> jobs, and they were of the kind:
>>
>> CPUs: 2x Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
>> RAM: 7GB
>> Disk: 94GB free disk space
>>
>> More than enough to bootstrap Gentoo Prefix!
>>
>> I don't know if this is the way to go. But at least is interesting to
>> have it in mind.
>>
>>
>> On Tue, Nov 27, 2018 at 12:01 AM Benda Xu <heroxbd@g.o> wrote:
>>
>>> Hi Sam,
>>>
>>> Sam Pfeiffer <sammypfeiffer@×××××.com> writes:
>>>
>>> > With Azure announcing unlimited minutes on CI/CD for open source
>>> > projects:
>>> >
>>> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>>> >
>>> > Even bootstrapping Gentoo prefix, with pieces of software like gcc
>>> > taking very long to compile, is possible.
>>> >
>>> > The point is: I have been trying to build Gentoo Prefix during the
>>> > last days after a few months of break since the last time I touched
>>> > the system. And it's failing. I haven't managed yet to bootstrap it
>>> > completely. I feel there is no CI/CD setup to catch these issues and
>>> > be able to offer a working version of Gentoo Prefix at any time.
>>>
>>> I completely agree with you.  I hope you can carry on this project to
>>> setup proper CI for Gentoo Prefix.  I am all in for help, portage/ebuild
>>> mentoring and coorperation.
>>>
>>> A CI for Gentoo Prefix has been on my list for ages.  Thank you for
>>> triggering this.
>>>
>>> Yours,
>>> Benda
>>>
>>>
>>
>> --
>>
>> *Sammy Pfeiffer*
>> PhD Candidate at The Magic Lab within UTS.
>>
>
>
> --
>
> *Sammy Pfeiffer*
> PhD Candidate at The Magic Lab within UTS.
>


-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.