[gentoo-alt] Re: 'Continuous Integration' for Gentoo Prefix?

[Michael Haubenwallner]

Hi Sam,

On 11/26/2018 09:11 AM, Fabian Groffen wrote:
<snip intro>

>> Given my particular annoyance of the 32bit system (and noting that my hosts are
>> 64bits, as is the standard nowadays) I found that I can use either:
>>
>> * [2]https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
>>
>> * [3]https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit Gentoo
>> Docker image
>>
>> And then execute the build step of Docker with:
>>
>> setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .
>>
>> Which will trick any program trying to do uname -m to assume 32bit machine.
>> (Previously I used the variable CHOST=i686-pc-linux-gnu for bootstrapping, but I
>> found some problems, which didn't appear with this method).
>
> Hmmm.

On 64bit machines, one can use 'linux32' to trick the autodetection into 32bit:
 $ uname -m
 x86_64
 $ linux32 uname -m
 i686

>> Once I have all this setup working nicely, I'd like to trigger rebuilds every X
>> time, and on changes on main players (or all dependencies actually) of my setup,
>> like the bootstrapping of Gentoo Prefix (Ideally, on every change of a part of
>> the system just trigger a rebuild from that point on, Docker layers make this
>> possible).
>
> This may be a bit too much, but you could start from every day
> (LATEST_TREE_YES), or we can see to building a tar more frequently from
> rsync0.
>
>> With Azure announcing unlimited minutes on CI/CD for open source projects: 
>>
>> [4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/

Nice, wasn't aware of that one!

>>
>> Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
>> long to compile, is possible.
>
> I'm still missing the infra to administer this, but I have a wrapper
> script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
> an unattended bootstrap.  I want this to run automatically on some boxes
> I have, but some of them will take several days to complete.
> Another thing is for a bootstrapped prefix to periodically emerge --sync
> and emerge -Dua world.  Add on top of that detecting which packages have
> keywords for said prefix and installing them, and we should be able to
> be notice breakage rather sooner than later.

Yes, I do have a buildbot setup within the company I'm working at,
but unfortunately I don't have the time to inspect the results and
fix upcoming problems on a regular base.
Beyond that, I cannot open the buildbot-master to the public, but
I believe I could share build results if we had some public place
to store them.

Anyway - the Prefix buildbot slaves here are doing a clean bootstrap
(with LATEST_TREE_YES=1, noninteractive) once per week, and an emerge
update/deep/world once per day. The list of "KEYWORD (hosts)" here is:
ppc-aix (AIX 6.1/64bit, AIX 7.1/64bit)
x86_64-cygwin (Windows Server 2012)
sparc-solaris (Solaris 10/64bit)
x86-linux (SLES11.1/64bit, SLES12.0/64bit, RHEL7.1/64bit, RHEL6.2/64bit)
amd64-linux (Gentoo/64bit)

Also, I do have need for Prefix Guest only, no RAP here - but there should be
hardware resources left to perform both builds.

>> The point is: I have been trying to build Gentoo Prefix during the last days
>> after a few months of break since the last time I touched the system. And it's
>> failing. I haven't managed yet to bootstrap it completely. I feel there is no
>> CI/CD setup to catch these issues and be able to offer a working version of
>> Gentoo Prefix at any time.
>
> Sorry about that.  I need to make another pass over it.

Indeed - only the 64bit (Cygwin & Linux) ones were succeeding recently,
all the 32bit ones failed.

>> I was going to build it for myself (cause I need it), but instead, I'd like to
>> offer my help to build it for the community. At least offer as an option
>> ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a Docker image,
>> just copy the full folder structure, do the softlink, and you are ready to play
>> with Gentoo. If you mess up anything, just re-deploy.
>>
>> To do this I'd need a bit of help as I'm quite new to Gentoo and I tend to get
>> blocked on little issues that take a while to google or debug (specially with
>> the long compilation times of the bootstrap and some big packages).
>>
>> I've posted a short issue in the Docker repo of Gentoo images about this
>> ([5]https://github.com/gentoo/gentoo-docker-images/issues/62) but I think the
>> maintainers probably don't usually work with Gentoo Prefix.
>
> I'm not very familiar myself with Docker (mostly hiding under a rock
> here), but I suspect the gentoo-docker-images project responds better to
> email to their list (as in Contributing from their README).  I may be
> wrong though.
>
>> Thank you very much for your time, and for your open source efforts.
>
> Thanks for your efforts sofar.  I hope we can pull something together.
> At least in the coming period I hope to be able to fix the bootstrapping
> issues for Solaris/OpenIndiana/Darwin.

I would love to see my buildbot slave's results somewhere in the public,
thank you for triggering this!

/haubi/