1 |
On Mon, 18 Jun 2012 19:42:55 -0700 |
2 |
Brian Harring <ferringb@×××××.com> wrote: |
3 |
|
4 |
> Bleh; wish your attachment had been text/plain for inline |
5 |
> commenting; pardon any mangling... |
6 |
|
7 |
Sorry, forgot to change the type before sending. |
8 |
|
9 |
> > 3. Defining dynamic SLOT groups |
10 |
> > ------------------------------- |
11 |
> > |
12 |
> > The list of supported dynamic SLOT groups should be declared |
13 |
> > in profile `make.defaults` as ``DYNAMIC_SLOT_GROUPS`` variable. That |
14 |
> > variable should contain whitespace-separated list of group names. |
15 |
> > |
16 |
> > For each group, the same file should have a variable named |
17 |
> > ``DYNAMIC_SLOTS_`` followed by the group name. That variable should |
18 |
> > list all possible dynamic SLOTs belonging to that group. |
19 |
> |
20 |
> This is USE_EXPAND machinery; you should clarify why it's not being |
21 |
> reused here, nor treat explicitly as such. |
22 |
|
23 |
I tried to avoid reusing USE flags partially since that could lead to |
24 |
potential problems which I could miss. It's basically fine to use flags |
25 |
here as long as we ensure that only one from a group is enabled for one |
26 |
ebuild, and they are reused like the rest of the text states. |
27 |
|
28 |
> Also, this gets fugly when one starts talking about individual pkgs |
29 |
> that have their own slotting, rather than global patterns like |
30 |
> python/multilib. |
31 |
> |
32 |
> Should consider that and address it... |
33 |
|
34 |
Could you provide an example use case? I guess we could assume there |
35 |
could be dyn. SLOTs/flags not mentioned in the global list, and that's |
36 |
where developer has to explicitly express their relations. |
37 |
|
38 |
> > 4. Defining supported SLOTs in an ebuild |
39 |
> > ---------------------------------------- |
40 |
> > |
41 |
> > An ebuild supporting building for multiple dynamic SLOTs has to |
42 |
> > declare the supported slots using ``DYNAMIC_SLOTS`` variable. The |
43 |
> > variable can be inherited from eclasses. |
44 |
> > |
45 |
> > For example, an ebuild supporting building for multiple Python ABIs |
46 |
> > would declare (either explicitly or implicitly through an eclass):: |
47 |
> > |
48 |
> > DYNAMIC_SLOTS='py2.6 py2.7 py3.1 py3.2' |
49 |
> > |
50 |
> > which would mean that when the package is built, one of the |
51 |
> > ``pyX.Y`` SLOTs must be used. As all of the listed SLOTs belong to |
52 |
> > the same group, only of them may be used at a time. |
53 |
> > |
54 |
> > An ebuild may also declare dynamic SLOTs from multiple groups:: |
55 |
> > |
56 |
> > DYNAMIC_SLOTS='py2.6 py2.7 lib32 lib64' |
57 |
> > |
58 |
> > In this case, both one of ``pyX.Y`` and ``libX`` SLOTs need to be |
59 |
> > declared. |
60 |
> |
61 |
> The words "I'll implement that when hell freezes over" come to |
62 |
> mind. That's a mess requiring the PM to know all potential values |
63 |
> and do interpolation on the fly; you're expecting the PM to be far, |
64 |
> far more intelligent then it can be, and the results won't be |
65 |
> pleasant (not counting the joys of writing such a resolver mind you). |
66 |
|
67 |
Doesn't the previous section exactly state where all possible values |
68 |
are to be listed? |
69 |
|
70 |
> Additionally, your notion breaks down if py3.3 supports lib64, but |
71 |
> not lib32. |
72 |
|
73 |
Doesn't sound like a likely case. |
74 |
|
75 |
> If the groupings are treated as USE_EXPAND (even if a segregated |
76 |
> group of it), you can abuse the same REQUIRED_USE machinery to |
77 |
> specify the allowed pigeon holes/slots, including arbitrary group |
78 |
> combinations. |
79 |
> |
80 |
> Still will be a bit harsh for the resolver I expect, but that's at |
81 |
> least descriptive enough to handle the py3.3/lib64 scenario I |
82 |
> mentioned, while being explicit data the PM can operate on (local to |
83 |
> that ebuild) without having to do nastyness. |
84 |
> |
85 |
> Note also that if one just dropped your notion of reinventing the |
86 |
> wheel, and reused REQUIRED_USE logic for slots... well, strikes me |
87 |
> that gives the flexibility you desire while folding it into existing |
88 |
> slot machinery. Haven't prototyped it, so may be cracktastic, but |
89 |
> seems a bit more integrated than what you're proposing. |
90 |
|
91 |
Agreed. |
92 |
|
93 |
> > 5. Building the ebuild against chosen SLOTs |
94 |
> > ------------------------------------------- |
95 |
> > |
96 |
> > In ``pkg_*`` and ``src_*`` phases, the build environment is provided |
97 |
> > with currently enabled dynamic SLOTs via variables named |
98 |
> > ``DYNAMIC_SLOT_`` followed by dynamic SLOT group name. The ebuild |
99 |
> > must use this variable to adjust the build process accordingly |
100 |
> > which can be done either directly or via an eclass. |
101 |
> > |
102 |
> > For example, in an ebuild using dynamic SLOTs for Python ABIs, the |
103 |
> > check may look like:: |
104 |
> > |
105 |
> > case ${DYNAMIC_SLOT_PYTHON} in |
106 |
> > py2.6) |
107 |
> > # ... |
108 |
> > ;; |
109 |
> > py2.7) |
110 |
> > # ... |
111 |
> > ;; |
112 |
> > # ... |
113 |
> > esac |
114 |
> |
115 |
> For ebuilds where one can't reuse the same source (moreso, can't do |
116 |
> the build of two different targets w/in that source, meaning you need |
117 |
> two work trees), this breaks down- which I expect is common. That |
118 |
> workflow needs to be non sucky for ebuild devs- so... sort that angle |
119 |
> please. |
120 |
|
121 |
Yes, argparse is a good example. I was still thinking how to handle |
122 |
that. Probably will require conditionals in SRC_URI and so on, so |
123 |
probably one more argument for USE_EXPAND. |
124 |
|
125 |
> > 6. Relevance to binary and installed packages |
126 |
> > --------------------------------------------- |
127 |
> > |
128 |
> > It is necessary to store the dynamic SLOTs for which a package was |
129 |
> > built in the binary package and installed package metadata. The |
130 |
> > exact semantics are left to be implementation-specific. The |
131 |
> > implementation must ensure, however, that a multiple dynamic SLOT |
132 |
> > variants of the same package can be installed at the same time. |
133 |
> > |
134 |
> > For example, dynamic SLOT could be appended after package version:: |
135 |
> > |
136 |
> > dev-python/foo-1.2.3+py2.6 |
137 |
> > dev-libs/bar-1.0+lib32 |
138 |
> |
139 |
> As ciaran stated, this is predicated upon the vdb being opaque to |
140 |
> ebuilds from this point forward. That's a large assumption you need |
141 |
> to validate before pushing this forward. |
142 |
|
143 |
Right now, I'm mostly trying to check whether we can work out a good, |
144 |
working solution for the issue. |
145 |
|
146 |
> Once you have ensured that ebuilds treat the VDB as opaque... this |
147 |
> section should be gutted. Ebuild format shouldn't be dictating the |
148 |
> VDB implementation (especially since the moment we *can*, I gurantee |
149 |
> the PM authors will be redesigning the VDB to be non shitty, and the |
150 |
> decision of how to handle the above is implementation specific). |
151 |
|
152 |
The implementation is not dictated. It is just an example to simplify |
153 |
understanding the text. |
154 |
|
155 |
> > 7. Installed file collisions |
156 |
> > ---------------------------- |
157 |
> > |
158 |
> > Due to the specifics of dynamic SLOT implementation, it is possible |
159 |
> > that files installed by various dynamic SLOT package variants |
160 |
> > collide. For example, each dynamic SLOT variant of a Python module |
161 |
> > may install a README file in the same docdir. |
162 |
> > |
163 |
> > The ebuilds must ensure that the colliding files are equivalent |
164 |
> > between various dynamic SLOT variants of the package. The package |
165 |
> > manager should overwrite those files whenever a new dynamic SLOT |
166 |
> > variant is installed, and must ensure that they will be removed |
167 |
> > when last dynamic SLOT variant of the package is uninstalled. |
168 |
> > |
169 |
> > The installed package metadata must ensure that the colliding files |
170 |
> > belong to all dynamic SLOT variants of the package, and their |
171 |
> > metadata is kept up-to-date. |
172 |
> |
173 |
> Collisions should be whitelisted imo via a specific var; via that, it |
174 |
> can be handled/tracked (easier on the PM, and easier on the ebuild |
175 |
> dev since they have an explicit QA control to utilize). |
176 |
|
177 |
Possibly. |
178 |
|
179 |
> > 8. Inter-package dependencies |
180 |
> > ----------------------------- |
181 |
> > |
182 |
> > If an ebuild supports at least one dynamic SLOT from a particular |
183 |
> > dynamic SLOT group and its dependency does so, the package manager |
184 |
> > must implicitly depend on matching dynamic SLOT variant of the |
185 |
> > dependant package. |
186 |
> > |
187 |
> > If the dependant ebuild does not support a matching dynamic SLOT, |
188 |
> > the dependency must not be satisfied by that package. |
189 |
> > |
190 |
> > In other words, if ``dev-python/foo`` depends on ``dev-python/bar`` |
191 |
> > and both of those ebuilds support dynamic SLOTs for Python ABIs, |
192 |
> > then the package manager must ensure that building |
193 |
> > ``dev-python/foo`` with ``py2.7`` dynamic SLOT will pull in |
194 |
> > ``dev-python/bar`` with ``py2.7`` SLOT. |
195 |
> |
196 |
> You're not giving any justification as to why this must be explicit, |
197 |
> versus builtin via a new slot dep operator... |
198 |
|
199 |
The reason is very simple: if we require a new slot dep operator, we |
200 |
require devs to check all the ebuilds in the tree and add that operator |
201 |
everywhere as necessary. This also includes checking all the reverse |
202 |
dependencies whenever dependent package starts supporting dynSLOTs. |
203 |
|
204 |
In other words, I'll certainly lose Ruby support on this one. And |
205 |
probably most of other devs will not support this too. |
206 |
|
207 |
> > 9. User interface |
208 |
> > ----------------- |
209 |
> > |
210 |
> > The choice of dynamic SLOTs for ebuilds whenever not directly caused |
211 |
> > by implicit dependencies is to be implementation-defined. |
212 |
> > |
213 |
> > The package manager should provide an ability to control the default |
214 |
> > dynamic SLOT choice to external tools like eselect. For example, |
215 |
> > the default Python SLOT would be controlled by eselect-python. |
216 |
> > |
217 |
> > The package manager must provide user with an ability to explicitly |
218 |
> > choose dynamic SLOT when merging a package and to store such a SLOT |
219 |
> > specification in the world file. |
220 |
> |
221 |
> This sounds strongly like you're requiring the manager to go |
222 |
> interactive. |
223 |
|
224 |
As in s/choose/specify/? |
225 |
|
226 |
> Note that the "store such a slot specification" bit is |
227 |
> inaccurate also- you've thus far basically treated slotting as a |
228 |
> separate beast from dynamic slotting. |
229 |
> |
230 |
> On that subject, what *occurs* when an ebuild is already slotted, and |
231 |
> has dynamic_slot crap in place? Gcc for example, is slotted, and |
232 |
> dynamic slotting is abusable for generating cross compilation |
233 |
> toolchains- you skipped that entire interaction (a very, very non |
234 |
> simple interaction I suspect). |
235 |
|
236 |
I don't think one does collide with the other at any point. The main |
237 |
reasoning for dynamic SLOTs is to allow creating multiple pseudo-SLOTs |
238 |
with one package version. They shouldn't interact at all with regular |
239 |
SLOTs which are bound to versions. |
240 |
|
241 |
> > The package manager may support specifying more than a single |
242 |
> > exclusive dynamic SLOT variants. In this case, the package manager |
243 |
> > should split the request into merging multiple dynamic SLOT |
244 |
> > variants of the package. |
245 |
> > |
246 |
> > For example, the following call:: |
247 |
> > |
248 |
> > emerge dev-python/foo+py2.6+py2.7 |
249 |
> > |
250 |
> > may cause the package manager to either reject the request (because |
251 |
> > of colliding dynamic SLOTs) or build two dynamic SLOT variants |
252 |
> > of the package. |
253 |
> |
254 |
> Re: "colliding dynamic slots", your proposals text, in current form, |
255 |
> doesn't allow for any collisions. To actually do what you describe, |
256 |
> you'd need ^^ ( py2.6 py2.7 ) |
257 |
|
258 |
The groups imply implicit collisions as the 'definitions' part states. |
259 |
I didn't see a reason to repeat that in every ebuild. |
260 |
|
261 |
> > 10. Unsolved problems |
262 |
> > --------------------- |
263 |
> > |
264 |
> > a) Large common parts of packages supporting dyn. SLOTs |
265 |
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
266 |
> > |
267 |
> > Often packages supporting dynamic SLOTs have large, common parts. |
268 |
> > In most common case, it is API documentation which gets generated |
269 |
> > at build time. Supporting multiple dynamic SLOTs would mean that |
270 |
> > documentation is going to be regenerated for every SLOT (or at least |
271 |
> > SLOTs supporting documentation generation). |
272 |
> > |
273 |
> > This get worse when various versions of the SLOT introduce tiny, |
274 |
> > irrelevant differences in the generated files. As the spec does |
275 |
> > disallow installing colliding files with different contents, this |
276 |
> > has to be handled some other way. |
277 |
> > |
278 |
> > Possible solutions: |
279 |
> > |
280 |
> > 1. Let each dynamic SLOT install own copy of files (i.e. separate |
281 |
> > docs for each version), |
282 |
> > 2. Move the common files into separate ebuild, |
283 |
> > 3. Introduce a special dynamic SLOT which would install just the |
284 |
> > common files, and depend on it (requires adding support for |
285 |
> > explicit dynamic SLOT dependencies). |
286 |
> |
287 |
> 3 is sounding painful. |
288 |
> |
289 |
> > b) Packages installing multiple kinds of dynamic SLOTs |
290 |
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
291 |
> > |
292 |
> > Example: packages installing bindings for multiple languages. The |
293 |
> > usual way of defining dynamic SLOTs, i.e.:: |
294 |
> > |
295 |
> > DYNAMIC_SLOTS='py2.6 py2.7 ruby1.8 ruby1.9' |
296 |
> > |
297 |
> > would mean that each time both a Python version, and a Ruby version |
298 |
> > has to be chosen. This would have to be backed up at least with USE |
299 |
> > flags and it gets more ugly... |
300 |
> > |
301 |
> > An interesting solution would be to assume that only one of the |
302 |
> > listed SLOTs is to be used at a time. This would mean that each |
303 |
> > build involves building just one of the bindings for a particular |
304 |
> > language of interest. Sadly, this would require making the spec |
305 |
> > more complex. |
306 |
> > |
307 |
> > A straightforward solution is to simply split the package. |
308 |
> |
309 |
> /me thinks you need to take a hard look at subversion bindings; |
310 |
> that's not "go compile the lib and install it", that's generally |
311 |
> "compile the lib once with variants in the binding" which won't match |
312 |
> into your scheem all that well best I can tell. |
313 |
> |
314 |
> |
315 |
> > c) Packages depending on utilities which don't need matching SLOTs |
316 |
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
317 |
> > |
318 |
> > The implicit dependency part assumes that all dependencies expect |
319 |
> > matching SLOTs. However, this is not always true. |
320 |
> > |
321 |
> > For example, a package building Python module may depend on another |
322 |
> > Python package yet call only a utility provided in that package. |
323 |
> > In that case, it is more likely that the utility would actually need |
324 |
> > to be pulled in with the currently selected system Python ABI rather |
325 |
> > than one used for the requested package. |
326 |
> > |
327 |
> > This issue could be probably solved through introducing explicit |
328 |
> > dynamic SLOT dependency atoms. |
329 |
> |
330 |
> If you want to do implicit... frankly, you need to create a working |
331 |
> prototype demonstrating it. That's a high bar requirement I realize, |
332 |
> but I'm stating it since this is the sort of nasty resolution problem |
333 |
> where actually *doing* it and seeing how it flies, flexes, etc, is |
334 |
> critical for proving it's a viable solution for people to rely upon. |
335 |
> |
336 |
> Most likely in creating such a beast, you'll wind up simplifying the |
337 |
> proposal due to sharp edges encountered; that simplification should |
338 |
> occur w/in the proposal itself rather than having it get passed, then |
339 |
> people start going "well fuck me sideways, this isn't actually sane |
340 |
> nor particularly implementable". |
341 |
> |
342 |
> Just saying; prototype is required there, frankly for most of this |
343 |
> proposal in general. |
344 |
|
345 |
Honestly, I'm not giving it much hope. And certainly I'd like to avoid |
346 |
hacking around if it's a no-go anyway. |
347 |
|
348 |
In pseudo-code, I thought about a logic like the following. Feel free |
349 |
to correct me if this couldn't work. |
350 |
|
351 |
(where 'slots' here are short for dynamic SLOTs) |
352 |
|
353 |
current_slots = ('py2.7', 'lib32') |
354 |
|
355 |
for d in dependencies: |
356 |
for p in matching_packages(d): |
357 |
try: |
358 |
sub_slots = () |
359 |
|
360 |
for s in current_slots: |
361 |
sg = slot_group_for(s) |
362 |
if p.supports_any_slot(sg.possible_slots): |
363 |
if p.supports_slot(s): |
364 |
sub_slots += (s,) |
365 |
else: |
366 |
raise DynSlotNotSupported() |
367 |
|
368 |
parse_package_deps(p, sub_slots) |
369 |
break |
370 |
except DynSlotNotSupported: |
371 |
continue # try next package |
372 |
else: |
373 |
raise NoMatchingPackages() |
374 |
|
375 |
-- |
376 |
Best regards, |
377 |
Michał Górny |