[gentoo-user] Re: Re: automated code validation

[Sam Bishop]

In order to catch up a bit since I wasn't subscribed to the
mailing list with this email at the time I found this thread.
If anything sounds odd, read through to the end.
I'm trying to top reply so I'm leaving my 'backstory' till the end.

> Rich Freeman <rich0 <at> gentoo.org> writes:
>
> > James <wireless <at> tampabay.rr.com> writes:
> >
> > I bet our friends at RackSpace will provide all the virtual HorsePower
> > you need, should google not provide the  hundreds/thousands or cores for
> > you to run on.
> >
> My guess is that the hardware to run all this on is the simplest part
> of the problem.  If somebody writes something good and we build the
> processes to support it, then I'm sure we can find donors to provide
> the CPU cycles.  ChromeOS could probably steal whatever we put
> together so there is an obvious synergy there, and I'm sure
> Amazon/Rackspace/etc are always looking for use cases to show off.
>

I agree about 80% with that, well put. The disagreeing 20% is pretty much
all about physical hardware, this is where, to quote a Red Hat employee I know
'the work gets interesting'. For the bulk of work, we can easily use virtual
machines, scavenge bargain basement EC2 spot instance hours, and have lots
of other options, your right that will be easy, the x86/amd64 arch testing
wont be hard to find a home for. Its all the other arch work that wont be easy.
I'm currently in the process of obtaining AMD Opteron A1100 dev kit boards,
and lets just say I'm not expecting our software to 'boot first time'.
Red Hat kindly keep the Beaker project (https://beaker-project.org) moving
forward which will be how I deal with these AMD dev kits. It only really helps
when you have hardware you can put aside for being part of a test pool. But
it is one of the few tools available to easily boot hardware, splat an OS
onto it, connect and perform automated tests on it, get as much info out
as possible even if there are kernel issues and it doesn't boot properly.
Once things progress I'd be amenable to letting my dev boards do ebuild
test runs when I'm not using them to port our software stack.

So while I dont have the 'idle hardware budget' of AWS, Google or Rackspace,
I am however building a cloud platform, with a customised version of CoreOS,
which is based on ChromeOS, which is based on Gentoo.
And the further the company progresses, the more drift I see between
our 'OS' and 'CoreOS'. I could not imagine tackling this if the entire thing
wasnt built on top of the foundation of a Gentoo based OS. So consider me
an ardent supporter of actually getting Gentoo automatic testing.

> Rich Freeman <rich0 <at> gentoo.org> writes:
>
> From past feedback from Diego and such the biggest issue with any kind
> of tinderbox is dealing with the output.  As has been pointed out
> there are folks running Repoman regularly already, and there have been
> past tinderbox efforts.  The real issue is to improve the signal/noise
> ratio.  You'll only get so far with that using code - there has to be
> process change to support it.>
>
> If we were to do something like this long-term I'd envision that this
> would run on every commit or something like that, and the commit
> doesn't go into the rsync tree until it passes.  If the tree goes red
> then people stop doing commits until it is fixed, and somebody gets
> their wrist slapped.  That is my sense of how most mature
> organizations do CI.  The tinderbox is really just doing verification
> - stuff should already be tested BEFORE it goes in the tree.  There
> also shouldn't be any false positives.  There would need to be a
> mechanism to flag ebuilds with known issues so that the tinderbox can
> ignore them, and of course we can monitor that to ensure it isn't
> abused.
>
> Basically doing this sort of thing right requires a big change in
> mindset.  You can't just throw stuff at the tree and wait for the bug
> reports to come in.  You can't just make dealing with the tinderbox
> the problem of the poor guy running the tinderbox.  The tinderbox
> basically has to become the boss and everybody has to make part of
> their job keeping the boss happy.

1st - Mindset change, definitely required. Can't agree more.
2nd - CI on this kind of thing is a multi-headed hydra of a thing. The
processes we wind up with will be quite similar philosophically but not
necessarily similar in implementation, staging or any other area.
For starters most CI pipelines aren't testing an entire distro as complex
as Gentoo ;-)
3rd - Looking at this linearly is less than idea. If an update breaks
5 downstream packages, the entire tree shouldn't go red and the only
person who should stop is the maintainer and/or commiter who submitted
the broken package. It should be more like automated QA 'gates' than a
pass fail build pipeline.
4th - Signal to noise is crucial! I'm going to be actively doing
something here because I have a build process that is building ebuilds
using portage, and when a build can take 30 minutes then an hour to test
the final machine image, I need to fail the build early, just flat out
avoid any broken but installable package versions ever reaching the
machine image testing stage of things.
5th - Ideally I think this will need multiple tools working together.
However as far as being able to build something that can start doing the
job, I feel like buildbot (http://buildbot.net) is probably the best
place to start. Its more of a CI 'framework' than a CI 'tool', so it wont
place too many 'not designed to do that' roadblocks at us. Its just my
first recommendation, I'm sure the matter will need much more discussion.

> James <wireless <at> tampabay.rr.com> writes:
>
> Or some other kind of hook?  Fishing was very difficult till the hook
> was refined. WE need a hook, imho. Easy Gentoo and a project to assist
> in testing code; you've got a killer idea there rich, and I'm glad to
> be on your team!
>

In my mind. Gentoo has the kind of hook you speak of. We just havent
refined it or done a good job showing it off to the world.
The ebuild format is one of the most powerful packaging standards.
It isn't perfect but its an extremely good start. It lets us go anywhere...
About a year ago I surprised a long time OSX user / Linux software developer
twice in the same conversation after they told me how they used to
use Macports before switching to Homebrew. First by mentioning that
netbsd's pkg-src works on OSX, then the second time when I told them
about the Gentoo Prefix project. They wanted 'freshness' in the software,
and felt only Homebrew had it, but that was only because neither pkg-src
or the Gentoo Prefix guys go out of their way to promote themselves as a
tool for that job the way Homebrew and Macports do.

An automatic test battery for Gentoo could possibly catapult Prefix
and similar sub projects forward to much greater adoption, and help
Gentoo reap secondary benefits as a result.

Now all the replying is done, here's the big question, where to organise
the bigger picture things? This mailing list? IRC? A new mailing list?
I want to get involved in this because I'm going to be building some
parts of this already. Why wouldn't I want to give back and help make
Gentoo better.