Re: [gentoo-alt] 'Continuous Integration' for Gentoo Prefix?

[Fabian Groffen]

I don't want to depress this entire discussion, but it would be really
nice if we could somehow interact with special machines people have at
their company or at home.  Prefix needs testing on many different
machines (non-Linux) which usually don't exist in docker images.

That said, focussing on the (usually fast) boxes like this to catch
dependency problems and more is useful.  In the case below it looks like
the ld-wrapper is having issues.  Would it be possible to enter the
environment for that failed run?

Thanks,
Fabian

On 27-11-2018 17:14:19 +1100, Sam Pfeiffer wrote:
> Hello all,
>
> Another little update on my test, it took me a while to find the actual way to
> increase the timeout to the maximum (or in other words, get out of the default
> of 60minutes), but I finally found how.
>
> Now I have a job that just tries to bootstrap Gentoo Prefix, the last run I made
> can be found
> here: [1]https://dev.azure.com/12719821/12719821/_build/results?buildId=21&view=logs
>
> You can see all the log. In this case it failed after 1h35min. It failed
> compiling GCC 8.2.0-r4.
>
> The error was:
>
> 2018-11-27T03:20:31.7540250Z
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/xg++
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/
> -B/tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ -nostdinc++
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
> -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs 
> -isystem
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu 
> -isystem
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include 
> -isystem
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0/libstdc++-v3/libsupc++
> -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
> -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs
> -no-pie -fno-PIE    -m64 -O2 -pipe -O2 -pipe -gtoggle -DIN_GCC   
>  -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing
> -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual
> -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings 
>  -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc  gcov.o \
>
> 2018-11-27T03:20:31.7545947Z hash-table.o ggc-none.o libcommon.a
> ../libcpp/libcpp.a   ../libbacktrace/.libs/libbacktrace.a
> ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a  -o gcov
>
> 2018-11-27T03:20:31.7670930Z
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/collect-ld:
> 106: exec: /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld: Invalid argument
>
> 2018-11-27T03:20:31.8025313Z collect2: error: ld returned 2 exit status
>
> 2018-11-27T03:20:31.8026192Z Makefile:2935: recipe for target 'gcov' failed
>
> In the end of the log we get the usual:
>
> 2018-11-27T03:23:00.8458538Z  * ERROR: sys-devel/gcc-8.2.0-r4::gentoo failed
> (compile phase):
>
> 2018-11-27T03:23:00.8480853Z  *   emake failed
>
> 2018-11-27T03:23:00.8501574Z  * 
>
> 2018-11-27T03:23:00.8524914Z  * If you need support, post the output of `emerge
> --info '=sys-devel/gcc-8.2.0-r4::gentoo'`,
>
> 2018-11-27T03:23:00.8550010Z  * the complete build log and the output of `emerge
> -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'`.
>
> 2018-11-27T03:23:00.8572142Z  * The complete build log is located at
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/build.log'.
>
> 2018-11-27T03:23:00.8593188Z  * The ebuild environment file is located at
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/environment'.
>
> 2018-11-27T03:23:00.8622509Z  * Working directory:
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build'
>
> 2018-11-27T03:23:00.8642956Z  * S:
> '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0'
>
> 2018-11-27T03:23:02.9054285Z  * 
>
> 2018-11-27T03:23:02.9078678Z  * Please include
> /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-build-logs.tar.bz2
> in your bug report.
>
> The cool thing comes now, how do you actually execute those commands to fill up
> a bug report?
>
> Easy, in the job I upload a Docker image with the system exactly as it was after
> the boostrap-prefix.sh command.
>
> So, if anyone wants to debug what is going on, they just need to do (this works
> right now):
>
> docker pull awesomebytes/gentoo_prefix_latest_image
>
> docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash
>
> And you are inside of the box:
>
> > user@b70d59ff9703:~$ ls
>
> > bootstrap-prefix.sh  start_bootstrap_date.txt
>
> > user@b70d59ff9703:~$ cat start_bootstrap_date.txt 
>
> > Tue Nov 27 01:52:20 UTC 2018
>
> > user@b70d59ff9703:~$ ls /tmp/gentoo
>
> > bin  etc  lib  lib64  run  sbin  stage1.log  stage2.log  stage3.log  tmp  usr 
> > var
>
> So you can do those commands suggested by emerge:
>
> /tmp/gentoo/tmp/usr/bin/emerge --info '=sys-devel/gcc-8.2.0-r4::gentoo' -->
> [2]https://pastebin.com/LWS3Bb7S
>
> /tmp/gentoo/tmp/usr/bin/emerge -pqv '=sys-devel/gcc-8.2.0-r4::gentoo' 
> --> [3]https://pastebin.com/gipspmnD
>
> etc.
>
> Or... we could even generate a new bug report automatically from this will all
> the necessary information, including the instructions on how to get into this
> box with exactly that state for anyone to help on fixing it.
>
> I'll try to dig further in with things that I find useful (and I hope you also
> find useful).
>
> Have a nice day!
>
> P.S.: With this I created a bug report easily: [4]https://bugs.gentoo.org/672042
>
> On Tue, Nov 27, 2018 at 2:07 AM Sam Pfeiffer <[5]sammypfeiffer@×××××.com> wrote:
>
> > Little update: The full build log is viewable to anyone with the link, so here
> > you can see the progress of the current job: 
>
> > [6]https://dev.azure.com/12719821/12719821/_build/results?buildId=17&view=logs
>
> > (Or I should say, the log of it, for whenever you open the link).
>
> > On Tue, Nov 27, 2018 at 2:02 AM Sam Pfeiffer <[7]sammypfeiffer@×××××.com>
> > wrote:
>
> >> Hello everyone,
>
> >> I'm very excited to see you are interested in adding continuous integration!
>
> >> I don't know that much about continuous integration, I've only used it (with
> >> systems already setup for me) with in-house Jenkins servers and with the ROS
> >> buildfarm, based on Travis CI on Github. Also a little bit of Gitlab CI in my
> >> lab.
>
> >> I did a bit of research/testing.
>
> >> Given it's quite a hassle to maintain custom machines, I'd try to use some of
> >> the free for opensource CI services. I've checked the conditions of a few to
> >> see which could fit better:
>
> >> * Gitlab CI: 2000 minutes / month
>
> >> * Travis CI: Unlimited minutes / month. But only 50 minutes long per step
> >> (like per script executed).
>
> >> * Azure pipelines: Unlimited minutes / month. 360 minutes long per step (like
> >> per script executed).
>
> >> There are probably many more, but those are the ones I knew about.
>
> >> Given that I wanted to give a try to Azure pipelines. And I did!
>
> >> I created this repo: [8]https://github.com/awesomebytes/gentoo_prefix_ci_test
>
> >> Where I activated Azure pipelines on it. After around 15min of reading the
> >> docs and playing around with the web-gui I got my first pipeline running.
>
> >> As an initial setup I thought I would create a Docker image where I bootstrap
> >> Gentoo Prefix from a Ubuntu 16.04 (as I'm familiar with it with my projects).
>
> >> The repo contains two important things:
>
> >> 1) The Dockerfile where I mainly trigger the
> >> bootstrap: [9]https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
>
> >> 2) The configuration file for Azure pipelines on what to
> >> do: [10]https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml
>
> >> I've implemented here that it tries to build Gentoo Prefix, and whatever the
> >> result, it uploads a Docker image to my DockerHub account with the results.
> >> This implies that:
>
> >> If the bootstrap is successful, one can just [docker pull] and [docker run]
> >> the image to play with Gentoo Prefix.
>
> >> If the bootstrap is unsuccessful, one can just [docker pull] and [docker run]
> >> to find oneself in the exact state of the system after the bootstrap command.
> >> And one can recover the full console log from the Azure pipelines web
> >> interface (even tho it would be nice to find out how to post it publicly
> >> straight away).
>
> >> If all goes well in a few hours anyone will be able to find in my DockerHub
> >> account said image (most probably the failed one), just doing:
>
> >> docker pull awesomebytes/gentoo_prefix_latest_image:latest
>
> >> docker run -it gentoo_prefix_latest_image /bin/bash
>
> >> You'll be inside of a Ubuntu 16.04 box with a user called 'user' and with all
> >> the bootstrapped stuff in /tmp/gentoo.
>
> >> As curiosity, I checked the machines I got served as 'agents' to run my jobs,
> >> and they were of the kind:
>
> >> CPUs: 2x Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
>
> >> RAM: 7GB
>
> >> Disk: 94GB free disk space
>
> >> More than enough to bootstrap Gentoo Prefix!
>
> >> I don't know if this is the way to go. But at least is interesting to have it
> >> in mind.
>
> >> On Tue, Nov 27, 2018 at 12:01 AM Benda Xu <[11]heroxbd@g.o> wrote:
>
> >>> Hi Sam,
>
> >>> Sam Pfeiffer <[12]sammypfeiffer@×××××.com> writes:
>
> >>> > With Azure announcing unlimited minutes on CI/CD for open source
> >>> > projects:
> >>> >
> >>> [13]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
> >>> >
> >>> > Even bootstrapping Gentoo prefix, with pieces of software like gcc
> >>> > taking very long to compile, is possible.
> >>> >
> >>> > The point is: I have been trying to build Gentoo Prefix during the
> >>> > last days after a few months of break since the last time I touched
> >>> > the system. And it's failing. I haven't managed yet to bootstrap it
> >>> > completely. I feel there is no CI/CD setup to catch these issues and
> >>> > be able to offer a working version of Gentoo Prefix at any time.
>
> >>> I completely agree with you.  I hope you can carry on this project to
> >>> setup proper CI for Gentoo Prefix.  I am all in for help, portage/ebuild
> >>> mentoring and coorperation.
>
> >>> A CI for Gentoo Prefix has been on my list for ages.  Thank you for
> >>> triggering this.
>
> >>> Yours,
> >>> Benda
>
> >> --
>
> >> Sammy Pfeiffer
> >> PhD Candidate at The Magic Lab within UTS.
>
> > --
>
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
>
> --
>
> Sammy Pfeiffer
> PhD Candidate at The Magic Lab within UTS.
>
>
>
>  References:
>    1. https://dev.azure.com/12719821/12719821/_build/results?buildId=21&amp;view=logs
>    2. https://pastebin.com/LWS3Bb7S
>    3. https://pastebin.com/gipspmnD
>    4. https://bugs.gentoo.org/672042
>    5. mailto:sammypfeiffer@×××××.com
>    6. https://dev.azure.com/12719821/12719821/_build/results?buildId=17&amp;view=logs
>    7. mailto:sammypfeiffer@×××××.com
>    8. https://github.com/awesomebytes/gentoo_prefix_ci_test
>    9. https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile
>   10. https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml
>   11. mailto:heroxbd@g.o
>   12. mailto:sammypfeiffer@×××××.com
>   13. https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
-- 
Fabian Groffen
Gentoo on a different level