Gentoo Archives: gentoo-qa

From: Stanislav Ochotnicky <sochotnicky@×××××.com>
To: gentoo-qa@l.g.o
Cc: gentoo-soc@l.g.o
Subject: [gentoo-qa] [GSoC-status] Tree-wide collision checking and files database
Date: Fri, 12 Jun 2009 13:33:39
Message-Id: 20090612133235.GA8898@w0rm
1 Hi everyone,
3 some of you already know that work on GSoC project "Tree-wide collision
4 checking and provided files database" has been started a few weeks ago.
5 For the rest, I will make a short introduction and goals of this
6 project (collagen).
8 Collagen aims to improve quality of ebuilds in portage tree. It does
9 this by compiling as many ebuilds as possible. It specifically takes
10 into account various atoms in DEPEND variable. For example if package
11 ebuild states that it needs =dev-libs/glib-2*, that package should be
12 compilable with every version of glib-2* in portage (taking into account
13 keywords). Therefore collagen will install one version of glib-2*, then
14 ebuild in question, collect information, uninstall ebuild and first
15 glib version. If repeats this process for every glib-2* in the tree.
17 Original idea was to have two sides:
18 * master server (matchbox)
19 * slaves compiling packages (tinderboxes)
21 Master server decides what needs to be compiled (automatically or
22 semi-automatically). Tinderbox asks for job, master provides package
23 name (and optionally version). Tinderbox then goes and tries to compile
24 package with different sets of dependencies reporting results to
25 Matchbox.
27 It seems that whole process could be sped up by hosting binary
28 packages on one central server (Binary host). Obviously various versions
29 of the same package would be created and therefore unique names could be
30 created by using some metadata to create hash part of filename. On a
31 first thought I would use USE flags and DEPEND as metadata to hash.
33 So far two other projects came to light as possible source of
34 inspiration and/or collaboration:
35 * catalyst (mainly tinderbox generating part)
36 * AutotuA (automatic generic job framework)
38 Especially AutotuA seems like good candidate for merging.
40 It doesn't seem possible to compile every project with every version of
41 every dependency, therefore I'd like to ask for your opinion especially
42 about this part. One idea I had was to restrict testing to highest build
43 number for given version. For example we have:
44 glib-2.18.4-r1 and glib-2.18.4-r2, therefore we will only test against
45 glib-2.18.4-r2 and will assume that r1 would be OK too (or users would
46 upgrade since it's a bugfix release)
48 Another approach to optimizing use of resources would be to have a
49 priority list of packages that need most testing. I imagine this could
50 be created by analyzing logs from gentoo mirrors, and figuring out which
51 packages are downloaded most frequently.
53 We would probably need at least one tinderbox per glibc version if I am
54 not mistaken since this cannot be freely up/downgraded.
57 This email was meant just as a teaser, more information (data model, UML
58 diagrams) is available on project website (look for Documents):
61 I'd love to be hear some suggestions, opinions and criticism. You can
62 use this thread, or even various options on
64 --
65 Stanislav Ochotnicky
66 Working for Gentoo Linux
67 Implementing Tree-wide collision checking and provided files database
69 Blog:
72 jabber: sochotnicky@×××××.com
73 icq: 74274152
74 PGP: