1 |
Dude...sounds to me like you need a PBS.... |
2 |
|
3 |
Nirbheek Chauhan wrote: |
4 |
> It's only been a little more than a week since I started working on |
5 |
> the project (due to personal reasons), and my time spent to work ratio |
6 |
> is extremely bad, so I'm sorry but the progress isn't as much as I had |
7 |
> hoped. |
8 |
> |
9 |
> The idea has undergone significant changes in the time passed, and |
10 |
> thanks to Patrick's guidance (and constant cluebats), I now have a far |
11 |
> more clear-cut idea of how the whole thing will come together. I |
12 |
> wonder whether I should describe the project blueprint that we've come |
13 |
> up with, the path that led to it, or what all code I have written. I |
14 |
> suppose the progress of the code written cannot be judged unless one |
15 |
> knows the whole plan, and the path taken to come up with the plan is |
16 |
> largely irrelevant :) |
17 |
> |
18 |
> The general idea has changed somewhat from the abstract: |
19 |
> |
20 |
> As before, there is a master server which acts a storage area, manages |
21 |
> all the slaves and does various bookkeeping. This part will be written |
22 |
> in Django. |
23 |
> |
24 |
> The concept of the slave has changed radically to allow for a less |
25 |
> steep learning curve. The project described "jobs" which consist of |
26 |
> executables stored on the master-server which could be fetched and run |
27 |
> by the slaves. We thought of ways in which we could describe |
28 |
> dependencies between the slaves, and the most obvious answer to me was |
29 |
> an XML format (much to the disgust of Patrick). |
30 |
> |
31 |
> However, there were numerous problems with such an approach (least of |
32 |
> which was the overhead involved with parsing XML and the jing deps on |
33 |
> the Django side for it). The most serious of these was the fact that |
34 |
> learning a new XML format and writing custom executables (scripts or |
35 |
> otherwise) which communicate with the server via the Slave's bindings |
36 |
> has an *extremely* steep learning curve, and will cause chaos. The |
37 |
> project is useless if no one ends up using it, or it gets too |
38 |
> complicated to use. |
39 |
> |
40 |
> The solution came to me in the form of a "Doh!" moment as I was |
41 |
> cycling back to my room. The answer was -- "jobuilds". Bash scripts |
42 |
> are easily adaptable, easy to understand and use (for Gentoo devs), |
43 |
> and their parsing is well-understood. For the second time in my life, |
44 |
> I appreciated the ingenuity of the inventors of the ebuild format. |
45 |
> |
46 |
> |
47 |
> Jobuilds: |
48 |
> ---------- |
49 |
> A jobuild is the smallest possible "quantum" of work. A job consists |
50 |
> of a root jobuild which has dependencies on other jobuilds, and all |
51 |
> these taken together form a job. The format of a jobuild is: |
52 |
> http://pastebin.osuosl.org/8355 |
53 |
> |
54 |
> - The four phases are to be run (by default) in the chroot where the |
55 |
> job will take place. |
56 |
> - SRC_URI are programs: test suites etc which are required by the |
57 |
> jobuild (does not include the deps which will be pulled in by emerge |
58 |
> in the chroot). |
59 |
> - PORTCONF_URI are tarballs which will contain portage config files |
60 |
> (/etc/portage/* /etc/make.conf etc) |
61 |
> - DEPEND are other jobuilds on which this jobuild _hard_ depends, ie |
62 |
> they must be completed in the same chroot (example: Test Amarok |
63 |
> depends on Build KDE which depends on Build X) |
64 |
> - SIDEPEND are SuperImpose Depends, all we need to know is that those |
65 |
> jobuilds completed successfully *somewhere* so that further |
66 |
> distribution of work is possible. (example: testing if all the |
67 |
> packages that import gnome2.eclass still work after some changes to |
68 |
> it) |
69 |
> |
70 |
> SRC_URI will be downloaded before entering the chroot, stored in a |
71 |
> tarballs folder, and hardlinked (if on the same device), or bind |
72 |
> mounted inside the chroot. |
73 |
> |
74 |
> To counter the problem of recursive QA checking, the jobuild format |
75 |
> will be *extremely* simple. That means no EAPI, no eclasses, no SLOTS, |
76 |
> minimal versioning (xxx.yyy), no fancy depends (except perhaps ||). |
77 |
> Built-in functions such as unpack() etc will be provided of course. |
78 |
> |
79 |
> The loss of utility from there not being eclasses will be offset |
80 |
> through the concept of "Template jobuilds" (similar in concept to how |
81 |
> Django handles Templates[1]). However, I am open to including eclasses |
82 |
> in the design (who doesn't love them? :) if enough reasons can be |
83 |
> given. |
84 |
> |
85 |
> NOTE: It will be highly recommended that the autotua work folder be on |
86 |
> the same device. I've assumed this to be true to allow a number of |
87 |
> optimisations, but I will keep (slower) fallbacks in case that is not |
88 |
> true. |
89 |
> |
90 |
> |
91 |
> The Tree: |
92 |
> ----------- |
93 |
> Obviously the jobuilds will be stored in a structured format similar |
94 |
> to the portage tree :^) |
95 |
> And following the tradition of being completely unimaginative, it |
96 |
> shall be called the "Jobtage tree". |
97 |
> The structure is as follows: |
98 |
> |
99 |
> ${user}/ |
100 |
> ${user}.asc |
101 |
> ${jobuild_name}/ |
102 |
> ${jobuild_name}-${ver}.jobuild |
103 |
> Manifest |
104 |
> |
105 |
> The tree will be stored in bzr, with an overlays/ directory in .bzrignore |
106 |
> jobuilds will not be manifested, and will only be signed with the |
107 |
> maintainer's gpg key |
108 |
> SRC_URI and PORTCONF_URI will be Manifested (probably the same way in portage) |
109 |
> |
110 |
> To further offset the problem of QA in this tree (mentioned in |
111 |
> "Jobuild" above), when Jobs are created/committed/uploaded on the |
112 |
> server (the details of that are in the next section), the whole |
113 |
> depgraph is validated, details about that stored as metadata, and the |
114 |
> Job itself is attached to **that specific revision** of the Jobtage |
115 |
> tree. This prevents breakages due to future changes made to the |
116 |
> jobuilds it depends on. If the maintainer wishes to update the |
117 |
> attached revision (for say a bugfix in a depending jobuild), he can |
118 |
> force a re-validation at anytime before a Job is accepted by a Slave. |
119 |
> Whenever a Slave accepts a Job, it syncs with the revision of the tree |
120 |
> it's attached to. |
121 |
> |
122 |
> The other solution to this problem could've been to trigger a reverse |
123 |
> depgraph validation whenever a commit was made to the tree. The |
124 |
> problems with that approach are: |
125 |
> - Load on the server increases exponentially with jobuilds |
126 |
> - Raises the question of what the next action should be -- revert the |
127 |
> (potentially critical) commit or mark (potentially hundreds of) |
128 |
> jobuilds as broken? |
129 |
> - Makes Jobs fragile -- a job might be fine when you upload it, but |
130 |
> horribly broken 4 hours later. |
131 |
> |
132 |
> |
133 |
> Slaves: |
134 |
> -------- |
135 |
> The slave can pull a list of Jobs that it can do from the master |
136 |
> server. A Job will consist of metadata about it: |
137 |
> http://pastebin.osuosl.org/8358 . The actual data is then gathered |
138 |
> from the jobuild(s), the chroot is prepared, etc etc and work begins. |
139 |
> Slave reports back to the master server after every jobuild is |
140 |
> complete with data and receives updates (if any) about the Job |
141 |
> (updates might consist of changing depends due to SIDEPENDs). |
142 |
> |
143 |
> Obviously the Slave has to parse jobuilds. And so the concepts should |
144 |
> be similar to Portage. However, I am drawing inspiration from the |
145 |
> pkgcore[2] codebase, simplifying the extremely versatile code to suit |
146 |
> my needs (which is another reason for my slow progress -- it's not |
147 |
> easy to understand a work of art ;) |
148 |
> |
149 |
> |
150 |
> Actual Progress aka "No more hand-waving": |
151 |
> ------------------------------------------------------ |
152 |
> Now follows my *real* progress w.r.t the code. |
153 |
> |
154 |
> I'm currently working on the slave, and am concentrating on the things |
155 |
> that don't depend on the part of parsing the jobuilds (have a general |
156 |
> idea how it's done, haven't fleshed out the details). Currently I've |
157 |
> implemented an OO interface (in Python of course) to a Job() object |
158 |
> accessed via Jobs(), a Syncer() object (jobtage), a Fetchable object |
159 |
> and a Fetcher (stage3 etc). Total code comes out to 167+70+38+30 = |
160 |
> ~300 lines ;p |
161 |
> |
162 |
> This week I'll start on chroot preparation and iron out the kinks in |
163 |
> that, followed by the Jobuild() object, the jobuild parser |
164 |
> (jobuild.sh), and the bridge connecting them. The #pkgcore guys are |
165 |
> really helpful and nice so I'll have good help for this part :) |
166 |
> |
167 |
> Next week (end of the month) will (hopefully) see a working slave |
168 |
> which accepts Jobs from some magical source and runs them. |
169 |
> |
170 |
> I'll begin work on the Master server the week after that, specifically |
171 |
> the backend work and the details of the communication between the |
172 |
> Master and Slaves. Frontend prettyfication will take place towards the |
173 |
> end. |
174 |
> |
175 |
> |
176 |
> 1. http://www.djangobook.com/en/1.0/chapter04/ -- not the exact |
177 |
> format, only the idea of "Reverse Inheritance" |
178 |
> 2. http://www.pkgcore.org/ |
179 |
> |
180 |
> PS: Another reason why progress is slow is because the Slave portion |
181 |
> has become much more sophisticated than what I had originally |
182 |
> intended. The original idea had (maintainer-made) executables doing |
183 |
> all the work (causing a steep learning curve) with the Slave just |
184 |
> being an API wrapper to talk to the master server. All of that work is |
185 |
> now shifted into the Slave and abstracted for the maintainer to use in |
186 |
> a familiar way. |
187 |
> |
188 |
> |
189 |
|
190 |
-- |
191 |
gentoo-soc@l.g.o mailing list |