[gentoo-qa] Re: [GSoC-status] Collagen - database schema and further changes

[Stanislav Ochotnicky]

My final (GSoC) report on collagen is here.

First, what was going on this past week...I've focused on adding
documentation (docstrings, comments) and a little bit of refactoring.
Then there were quite a few modifications to simplify installation and
now I can say that simple:

 # python setup.py install

plus few configuration steps afterwards will do the whole installation
procedure. Alternatively it's possible to use bundled ebuild instead of
setup.py script. This was actually first time I've created setup.py
script and I have to admit that for simple stuff (like collagen) it's
quite easy to create it from scratch in a few minutes.

So to the main summary of the project, what works, what doesn't really
work but is planned for post-GSoC era etc etc.

We have working end-to-end system for automatic distributed
compilation of packages from portage tree with information stored
in database.  Information being contents of packages or in case
of compilation failure build.log, environment, emerge --info and
application log for good measure. We were able to catch few bugs in
ebuilds already even with limited resources (meaning I was mostly
compiling in a virtualbox on one of my machines). Even if QA like
this was not originally the main goal of collagen, it might later turn
out to be exactly that. Bug hunting monster :-)

As I mentioned in last week's report web interface for this was
done in django and I've implemented 3 basic use-cases for showing
data in database. If you've worked with Django before then you know
it's quite easy to add more functionality.

At least one more important use-case to be implemented is "is this package
colliding with something else?" or even better "show me groups of packages
colliding with each other". Data is there, database schema is able to
support these use cases so this will definitely be implemented
(although presumably after GSoC).

This is where we come to "what now" part. Whole collagen didn't get a
lot of testing yet, and I'd love to try it on more than one virtualbox
machine. There are a lot of places where performance could be improved
(read: where we would not install already installed packages and such).
Another unimplemented idea was using remote binary hosts. However I
really think that this would make much more sense if it was implemented
together with improved binary support for portage. Then there was
authentication of tinderboxes to matchbox for communication between
them. Right now machine able to connect to matchbox server could
probably run python code as user running matchbox. I haven't tried, but
python documentation for pickle module is pretty clear about insecurity
of this approach (I was assured this won't be a problem after auth is
done though). 

All in all collagen is far from being finished, polished piece of
software right now. But at least in my opinion it already showed that it
could be useful and improvements can now be done in quite modular way.
One exception being of course database schema, but even changes there
are nothing to be afraid of (I presonally tried, it's possible without
destroying the data :-) ). Expect few more updates on 
soc.gentooexperimental bugtracking/documents. Then 

Now let me say big thank you to my mentor weaver and the whole
Gentoo team. Few examples:
 * weaver - for changing our meeting time quite a few times because of me,
            and for the patience.
 * zmedico - for being portage king, always ready to help out and even
             explain 'why', not just how
 * robbat2|na - for becoming base unit of problem solving recently.
 * .* - for answering quite a few of my questions and giving me ideas.
         Most of you probably never noticed that you helped me
         (sometimes even I didn't notice at the time :-) ).

I hope I'll work with you to further improve collagen and after
that, other Gentoo projects. I am not going anywhere yet, as long
as you won't chase me away with a stick :-) For me this is not really
over, I'll chill out a little bit of course, but if at all possible I'd
like to continue contributing.

-- 
Stanislav Ochotnicky
Working for Gentoo Linux http://www.gentoo.org
Implementing Tree-wide collision checking and provided files database
http://soc.gentooexperimental.org/projects/show/collision-database
Blog: http://inputvalidation.blogspot.com/search/label/gsoc


jabber: sochotnicky@×××××.com
icq: 74274152
PGP: https://dl.getdropbox.com/u/165616/sochotnicky-key.asc