[gentoo-osx] Package testing -- Automated initiative

[Grobian]

Introduction
============

Recently, once again we were confronted with a package marked as
ppc-macos stable, while it didn't compile at all, let alone run.  It is
believed more of these packages are in portage, and need to be found and
fixed.  Keeping the cause of why they are marked stable up to another
discussion, and out of the scope of this discussion, I will focus on how
to track these packages down and report them to us.

In the secondary line, all 'unstable' packages, marked ~ppc-macos should
be tested as well, since they can be faulty as well.  Since for OSX much
is in ~ppc-macos, many users consider it a normal procedure to switch to
the unstable side of portage, hence some extra need for careful testing
of ~ppc-macos also.


Proposed Global Structure
=========================

Testing should be done on a regular basis, both push and pull based.
This means that the testing machine would start testing packages itself
if it is out of work, and on the other hand starts testing packages as
soon as they are being added/changed in CVS.  It may need no great
imagination to see that the latter 'push-based' activity has priority
over the 'pull-based' work.

Starting over, will for the test machine mean that it starts cleaning
out its world file.  Cleaning this file out to a bare minimum is an
important aspect of getting a test environment that reflects the
situation on new user's machines.  If an ebuild uses a package without
having it in it's DEPEND, this may get noticed only when starting on a
clean machine.  This, however, will add a big delay in testing as many
packages will need to be built prior the right package can be installed.

The testing machine will have a queue file, which it reads packages to
emerge from.  If the queue file is empty, i.e. when there is no push
based work, the machine will generate work by starting to compile
uncompiled packages, or emptying the tree.

Because ~ppc-macos and ppc-macos packages interfere with each other -- a 
~ppc-macos package overwrites a ppc-macos package -- both stable and 
unstable have to be dealt with separately, i.e. they should both have 
their own environment either via two separate machines, or through the 
use of a chroot jail.


Queues
------

In order not to drag in a full DBMS (in the end Portage already is one) 
queues are just simple flat files consisting of absolute package names, 
one per line.  Table wise locking granularity is handled by the OS as 
one process opens the file in write mode.  Consumers -- the testing box 
in this case -- read the first line and delete it, while producers 
simply add one line (or more) to the end of the file.

The queue itself, is more a set than a list.  This means that packages 
that are in the queue, should be unique.  If a package is added that is 
already in the queue, it is dropped such that the original queue 
position of the package is maintained.


CVS Producer
------------

To catch up automatically with changes made to the tree, it is necessary 
to act upon any commit to the tree for an ebuild file.  A possibility to 
do this would be via processing of CVS commit messages, sent out as 
email by the CVS server.  It is a task of the producer to find out 
whether the ebuild found applies to the testing machine (ppc-macos) and 
add the package/ebuild to the queue.


Consumer (testing process)
--------------------------

The test machine reads a line from the queue, and basically executes 
'emerge ${PACKAGE}'.  However, before doing this, first it figures out 
which use flags can be used (emerge -pv) and which dependencies will be 
pulled (emerge -pt).  If portage returns the message all ebuilds that 
could satisfy X have been masked, the emerge is cancelled, the line is 
removed from the queue and an email message will be sent out.

All dependencies are put in the right order and emerged as normal 
packages, that is: all dependencies are pushed at the front of the 
queue, thereby keeping uniqueness of the queue and removing duplicates 
that appear later on in the queue.  After this, the consumer is 
restarted and reads again from the queue.  This should result in usually 
merging only one package at a time, and as such quite isolated cases, 
which should improve the error email notification service.

Compile testing a package is supposed to be a thorough test that tries 
all possible combinations of the package's USE flags.  As this might be 
somewhat endless as some packages are rather big and have zillions of 
USE flags, it may be necessary to have a special "don't do it" file.
Since all dependencies were put at the front of the queue, there should 
normally be no dependencies that the package pulls.
If compilation fails for a certain USE-flag combination this is reported 
by sending out an email, and compilation of the next USE-flag 
combination is attempted.

When everything goes fine, no email notification is being sent out.  A 
convenient log structure would, however, make it possible to see which 
packages and USE-flag combinations successfully passed through. 
Providing this log via a web-page would be a useful thing.  Again 
backing this with a DBMS to allow easy searching, versioning and stuff 
is considered to be overhead, though crafting logs in SQL's "INSERT 
INTO" format might enable another machine to display the output data. 
Perhaps the communication methods needs a section on itself.


Recap and Conclusion
====================

By setting up a testing system, it is possible to greatly improve the 
Quality of Service of the portage tree for an architecture by exhaustive 
testing of both packages already in there, as well as packages added or 
modified.  Automated testing should not release developers from testing 
themselves, but should help in pointing out problems that may arise on 
moving grounds such as portage where packages are constantly updated and 
dependencies might get broken.


ToDo
====

- Not only check dependencies of the respective package, but also 
consider packages that depend on the respective package, thus rebuilding 
all packages that depend on the package to check if anything is broken 
by the update.
- Is there a gleptomaniac in the room?  This would be useful for x86 
also, of course.  In that case it may be necessary to make sure the 
packages are split over multiple machines.
- The message system needs more customisation options, especially 
backing things by a DBMS would allow for many nice bugzilla-like 
preferences for email generation as well as web-based versioned 
info/report pages
- To make the system even bigger, a central DBMS powered server might 
take a leading role and ... {editor note: wait, stop it right now, 
you're going too fast right now}


By The Way
==========

- Kito offers his lil' chico as machine for this automated testing 
initiative.
- Comments are welcome, as well as expressions of worry on my mental state.
- Implementation of described system will need some better specified 
system and needs some coding (the dirty work) in some language...


-- 
Fabian Groffen
eBuild && Porting
Gentoo for Mac OS X

-- 
gentoo-osx@g.o mailing list